configuring hci with the netapp deployment engine · 2019-10-30 · • vsphere vcenter server...

37
Configuring HCI with the NetApp Deployment Engine September 2019 | SL10518 Version 0.0.9

Upload: others

Post on 03-Feb-2020

33 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetAppDeployment Engine

September 2019 | SL10518 Version 0.0.9

Page 2: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine2 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

TABLE OF CONTENTS

1 Introduction...................................................................................................................................... 3

2 Prerequisites.................................................................................................................................... 4

3 Lab Environment............................................................................................................................. 5

4 Lab Activities................................................................................................................................... 8

4.1 Deploy HCI with the NetApp Deployment Engine..................................................................8

4.2 Review Resulting HCI Configuration.....................................................................................27

5 Version History.............................................................................................................................. 36

Page 3: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine3 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

1 IntroductionThe NetApp Deployment Engine (NDE) greatly simplifies the configuration of NetApp HCI. By presenting a fewconsolidated web forms to fill out, the NDE quickly gathers the needed information and automates the deploymentof the following HCI components:

• Element storage cluster.• vSphere ESX hypervisors.• vSphere datastores hosted on the Element cluster.• vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance).• NetApp Management Node (for managing the Element cluster, including the vCenter plugin).

1 Lab ObjectivesThis lab starts with two unconfigured HCI compute nodes and four unconfigured storage nodes. The stepsprovided will utilize the NDE to fully configure the HCI system. The NDE workflow is as follows:

• Confirm Prerequisites.• Accept license agreements for NetApp and VMware.• Name the new vCenter instance that NDE will deploy.• Enter the administrator credentials to be used across all HCI components.• Select the network topology (6-wire or 2-wire configuration options).• Select from available compute and storage nodes for the configuration.• Specify a hostname prefix and define three IP ranges to use for network interfaces.• Review settings and start the HCI configuration deployment.

Page 4: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine4 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

2 PrerequisitesTo get the most out of this lab, you should have a basic understanding of the following topics and concepts:

• NetApp HCI• VMware vSphere ESX and vCenter

Page 5: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine5 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

3 Lab EnvironmentThis virtualized environment is designed to be the functional equivalent of a NetApp HCI system as shipped, butprior to configuration.

The following illustration depicts the smallest NetApp HCI configuration that fits into a mere 4RU of rack space!

Figure 3-1: NetApp HCI Photo and Minimum Configuration Layout

Figure 3-2:

Page 6: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine6 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

Figure 3-3: Lab Environment Diagram

Figure 3-4:

3 Table of Systems

Host Name Operating System Role/Function IP Address

jumphost Windows Server 2012 R2 primary desktop entrypoint for lab

192.168.0.5 / 192.168.1.5

dc1 Windows Server 2012 R2 Active Directory / DNS 192.168.0.253

NetAppHCI-stg-01 NetApp Element 11.3 Element node 1management / Storage

initially DHCP-suppliedaddresses

NetAppHCI-stg-02 NetApp Element 11.3 Element node 2management / Storage

initially DHCP-suppliedaddresses

NetAppHCI-stg-03 NetApp Element 11.3 Element node 3management / Storage

initially DHCP-suppliedaddresses

NetAppHCI-stg-04 NetApp Element 11.3 Element node 4management / Storage

initially DHCP-suppliedaddresses

NetAppHCI-cpu-01 vSphere ESX 6.7 NetApp HCI hypervisor initially DHCP-suppliedaddresses

NetAppHCI-cpu-02 vSphere ESX 6.7 NetApp HCI hypervisor initially DHCP-suppliedaddresses

Page 7: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine7 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

3 User IDs and Passwords

Host Name User ID Password Comments

Jumphost DEMO\Administrator Netapp1! Domain Administrator

Page 8: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine8 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

4 Lab ActivitiesThis lab contains the following activities and tasks:

• Deploy HCI with the NetApp Deployment Engine on page 8• Review Resulting HCI Configuration on page 27

4.1 Deploy HCI with the NetApp Deployment EngineThe NetApp Deployment Engine starts with just a few consolidated web forms, and then automates the remainderof the configuration for you.

The following steps will initiate the configuration of this (currently unconfigured) HCI system:

1. Open FireFox. The default web page is set to point to one of the four unconfigured storage nodes.

Note: The unconfigured HCI nodes use DHCP to get their initial IP addresses; however, the NDEwill assign static IP addresses during the configuration automation.

2. Click Get Started.

2

1

Figure 4-1:

3. To accept the self-assigned certificates, click Advanced....4. Click Accept the Risk and Continue.

Page 9: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine9 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

4

3

Figure 4-2:

5. Check the three prerequisite boxes.

Note: The referenced deployment guide lays out the networking requirements for NetApp HCI.These requirements need to be in place prior to running NDE.

6. Click Continue.

Page 10: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine10 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

65

Figure 4-3:

7. Scroll down through the NetApp End User License Agreement and check the box for I accept.

Page 11: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine11 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

7

Figure 4-4:

8. Scroll down through the VMware End User License Agreement and check the box for I accept.9. Click Continue.

Page 12: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine12 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

9

8

Figure 4-5:

10. HCI can either be integrated into an existing vSphere vCenter Server instance, or it can create a newone. For this demonstration, select the radio button for Configure a new vSphere deployment.

11. Select the radio button for Configure Using vSphere Version 6.7 Update 1.12. Select the radio button for Configure Using a Fully Qualified Domain Name and enter the following:

• vCenter Server Fully Qualified Domain Name: vc1.demo.netapp.com• DNS Server IP Address: 192.168.0.253

13. Click Continue.

Page 13: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine13 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

13

12

1110

Figure 4-6:

14. The credentials that you set on this page are used for the Element cluster, vCenter server, and theManagement Node. Enter the credentials as shown:

• User Name: admin• Password: NetApp123!

15. Click Continue.

Page 14: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine14 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

15

14

Figure 4-7:

16. In this lab environment, the networking has been configured to enable the 6 Cable Option. Select the 6Cable Option, and check the box for Use vSphere Distributed Switch?.

17. Click Continue.

Page 15: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine15 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

17

16

Figure 4-8:

18. This next step is where you need to select at least 2 compute nodes and 4 storage nodes for theminimum HCI configuration to continue. Check the box to select both storage nodes.

Note: This lab has been configured with exactly the minumum number of nodes, so you will beselecting all compute and all storage nodes in this step. If more nodes were available when theNDE is run, you would have the option here whether to include them in the initial configuration,or not.

19. Check the box to select all four storage nodes.20. Click Continue.

Page 16: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine16 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

20

19

18

Figure 4-9:

21. The Network Settings page can be filled out manually to allow granular control of each component'sIP address; however, it is recommended to use the easy form to pre-populate these values using IPranges. Before using the form, change the NTP Server Address 1 to be 192.168.0.1

22. Click the button to use the easy form.

Page 17: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine17 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

22

21

Figure 4-10:

23. Enter the following values into the form:

• Naming Prefix: NetAppHCI• Will you assign VLAN IDs? (click the NO radio button)• Management Network Subnet: 192.168.0.0/24• Management Network Default Gateway: 192.168.0.1• Management Network Starting IP: 192.168.0.100• vMotion Network Subnet: 192.168.2.0/24• vMotion Network Starting IP: 192.168.2.100• ISCSI Network Subnet: 192.168.1.0/24• ISCSI Network Starting IP: 192.168.1.100

Note: Be careful to use the matching subnet for each Subnet/Starting IP pair. Forexample: 192.168.X.0/24 should be paired up with the starting IP of 192.168.X.100.

24. Click Apply to Network Settings.

Page 18: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine18 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

24

23

Figure 4-11:

25. Click Yes to confirm.

Page 19: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine19 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

25

Figure 4-12:

26. This next page has an integrated network validation tool. Let the IP address validation complete beforecontinuing...

Note: The “Continue” button in the next step will not appear until either the validation completeswithout error, or the button for disabling validation is used.

Page 20: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine20 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

26

Figure 4-13:

27. Notice the specific IP addresses that have been applied. You can optionally modify these beforecontinuing, but for now, click Continue.

Page 21: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine21 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

27

Figure 4-14:

28. You can expand the review sections by clicking on any of the + icons. When ready, click StartDeployment.

Page 22: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine22 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

28

Figure 4-15:

29. Click Copy to Clipboard to copy the NDE monitoring URL to the clipboard. This will only be needed ifthere is a disconnect, or the browser is accidentally closed during the deployment.

30. click OK.

Page 23: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine23 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

30

29

Figure 4-16:

31. Wait while the deployment begins. Continue to the next step when the browser is redirected to a newpage as shown in the step below.

Page 24: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine24 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

31

Figure 4-17:

32. This status page will show where the NDE automation is working during the deployment process.

Note: The process, both on hardware and in this virtual lab, takes ~45 minutes to complete.

The NDE will now automate the following tasks:

• Create a four-node Element cluster.• Create two Element volumes for use as vSphere datastores.• Create two ESX hypervisor hosts.• Add the ESX iscsi initiators to the Element configuration.• Create an Element Volume Access Group and configure it to include both datastore volumes

and both ESX initiators.• Format the attached volumes (iscsi LUNs) with VMFS.• Configure vSphere portgroups and uplinks for managing VM traffic, including vCenter and the

Management Node.• Deploy the vSphere vCenter Server Appliance (VCSA) onto an ESX host and use an

Element-supported datastore for the VCSA storage.• Register the two ESX hosts with the vCenter instance.• Deploy the NetApp Management Node and register it with vCenter.

Matt Miller
Because of the time it takes for the automated install procedure to complete, don’t wait for it to finish. Instead, review the remainder of this guide to see what the deployed components look like from the viewpoint of the vSphere Web Client. �
Page 25: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine25 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

32

Figure 4-18:

33. When you started the NDE process, the URL was pointing to 192.168.0.250. This was a staticallyassigned address for the first storage node. All other interfaces were configured with DHCP. Once theNDE configures the nodes with the new addressing from the Network Topology configuration above,the browser will redirect to the new IP address so that the process can continue to be monitored. Whenthis refresh happens, you will be prompted about the self-assigned SSL certificates for the new URL.

Note: APIPA (Automatic Private IP Addressing) can also be used for the initial HCI nodeaddresses.

To accept the self-assigned certificates, click Advanced....34. Click Accept the Risk and Continue.

Page 26: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine26 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

33

34

Figure 4-19:

35. At this point, you will return to the progress monitoring page at the new IP address. When theconfiguration is complete, you will see the following summary page. There are two buttons available:

• Update Services: This button causes the Management Node to be upgraded to the latestrelease. Updating the Management Node is useful in upgrading the Element cluster, as wellas preparing HCI for Cloud Data Services like the NetApp Kubernetes Service.

Note: There is an issue with the current Management Node in this lab environmentthat is preventing this update from succeeding. This will be addressed as soon as aresolution is available.

• Launch vSphere: This button opens a browser tab to the new vSphere instance created bythe NDE.

Page 27: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine27 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

35

Figure 4-20:

The NetApp HCI configuration is now complete. Continue to the next section to discover the configuredcomponents of this NetApp HCI system.

4.2 Review Resulting HCI ConfigurationNow that the NDE process is complete, the system is ready to run virtualized workloads. The NetApp ElementVirtual Console Plugin is installed by the NDE so that the system can now be maintained through the familiarvCenter web UI.

The following steps show a quick discovery of the configuration accomplished by the NDE.

1. From the final page of the NDE process, click the Launch vSphere button.

Page 28: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine28 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

1

Figure 4-21:

2. To accept the self-assigned certificates, click Advanced....3. Click Accept the Risk and Continue.

Page 29: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine29 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

3

2

Figure 4-22:

4. Click the X to close this dialogue box.

Page 30: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine30 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

4

Figure 4-23:

5. Enter the credentials below:

• User name: [email protected]• Password: NetApp123!

6. Click Login.

Page 31: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine31 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

6

5

Figure 4-24:

7. The warning at the top of the vSphere UI is there because the system has been installed with temporarylicenses. In a real environment, you would use the normal vSphere process to add and apply the neededlicenses to the vCenter and ESX assets. Click the drop-down menu and select NetApp ElementConfiguration.

Page 32: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine32 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

7

Figure 4-25:

8. Click on the Clusters tab, and notice the Element cluster that was created has already been configuredin the Virtual Console Plugin.

Page 33: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine33 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

8

Figure 4-26:

9. Click on NetApp Element Management.10. Click on the Reporting tab.11. Here you can see the primary metrics for the Element cluster, as well as the tabs for the other Element

cluster management links.

Page 34: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine34 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

11

10

9

Figure 4-27:

12. Click on the Management tab.13. Here you can see the two pre-configured vSphere datastores that are hosted on the Element cluster.

Page 35: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine35 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

13

12

Figure 4-28:

You have now configured NetApp HCI with the NetApp Deployment Engine. The NDE is also used foradding independent compute and storage resources to the HCI system when needed; however, this isbeyond the scope of this demonstration.

Page 36: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Configuring HCI with the NetApp Deployment Engine36 © 2019 NetApp, Inc. All rights reserved. NetApp Proprietary

5 Version History

Version Date Document Version History

0.0.8 September 2019 INITIAL DRAFT

0.0.9 September 2019 Updated new URL and removed oldbookmarks

Page 37: Configuring HCI with the NetApp Deployment Engine · 2019-10-30 · • vSphere vCenter server instance (or, optionally, it utilizes an existing vCenter instance). • NetApp Management

Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exactproduct and feature versions described in this document are supported for your specific environment.The NetApp IMT defines the product components and versions that can be used to constructconfigurations that are supported by NetApp. Specific results depend on each customer's installation inaccordance with published specifications.

NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of anyinformation or recommendations provided in this publication, or with respect to any results that may be obtainedby the use of the information or observance of any recommendations provided herein. The information in thisdocument is distributed AS IS, and the use of this information or the implementation of any recommendations ortechniques herein is a customer’s responsibility and depends on the customer’s ability to evaluate and integratethem into the customer’s operational environment. This document and the information contained herein may beused solely in connection with the NetApp products discussed in this document.

Go further, faster®

© 2019NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consentof NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Data ONTAP®,ONTAP®, OnCommand®, SANtricity®, FlexPod®, SnapCenter®, and SolidFire® are trademarks or registeredtrademarks of NetApp, Inc. in the United States and/or other countries. All other brands or products are trademarks orregistered trademarks of their respective holders and should be treated as such.