lee evans - ldc commodity hosting project portfolio 2005 - 2012

Prepared for LDC Internal Operations by LeeWare Development Consulting

LeeWare Development Consulting Engineering success through people, organizations, and technology

Management Information Systems (MIS) Commodity Hosting Project Portfolio

2005 - 2012 Version 1.0

1


1.0 Legal Notice

1. This project portfolio contains a composite of representative projects completed for internal operations and customers who have engaged the services of LeeWare Development Consulting.

2. LeeWare Development Consulting maintains exclusive supplier agreements to conduct work for

stakeholders. Most agreements are for independent contracting, other work is done on a retained basis.

3. LeeWare Development Consulting has a legal obligation to protect the identity of its customers,

as well as the legal obligation to not release to anyone information that is proprietary to customer operations. For the purpose of this presentation, the context, scenarios, and solutions are real, but the companies and downstream customers have been fictionalized to protect the integrity and confidentiality of stakeholder operations. The major aim of this work is to present a high-level overview of the work product.

2.0 What is Management Information Systems (MIS)?

1. Management Information System (MIS) is the study of people, technology, organizations, and the relationships among them. 1

2. MIS professionals help organizations of all sizes realize maximum benefit from investments in

personnel, equipment, and business processes. MIS is people-oriented, with an emphasis on service. Although today it is increasingly built on computer hardware, software and networks, it does not necessarily have to be computer-based.

3. In practice, I work with stakeholders to get a clear picture of where the business wants to be,

outline a plan for how to get there, and provide the necessary governance and tactical support to reach and maintain the desired business state. I have the capacity to do technical work as demonstrated by my portfolio whereby 75% of my work is strategic while only 25% is practical.

3.0 Why do Stakeholders commission my services?

1. Resource Mentorship/Augmentation. LeeWare Development Consulting vCIO can act as an “as-needed” resource to mentor existing internal IT resources. The vCIO can also provide assistance/guidance for areas/times where existing resources have either skills or availability gaps.

1 https://en.wikipedia.org/wiki/Management_information_system

2


2. Strategic Guidance/Planning. LeeWare Development Consulting vCIO will provide a neutral resource to assist in strategic planning and decision making.

3. Project Ownership. LeeWare Development vCIO will provide management of IT related projects, focusing on project deliverables and success by “owning” the project and providing a single point of contact and accountability.

4. On-going Account Reviews. LeeWare Development Consulting prepares and delivers a report on each deliverable, outlining all services performed, current customer business initiatives/projects, timelines/status for existing projects, and recommendations for moving forward.

Industries Served: ■ Technology startups ■ Managed Service Providers ■ Central Station Alarm Security ■ Casino Gaming Industry ■ Publishing ■ Digital Advertising ■ Property Management, ■ Virtual Organizations ■ ISPs ■ Accounting ■ Medical Specialties: ■ IT Strategy & Architecture Design ■ Multi-Site IT Operations ■ IT PCI-DSS, UL and FM/Regulatory Compliance ■ IT Governance / ITIL ■ Risk & Business Impact Assessment ■ Business Continuity & Recovery ■ Information Security Operations ■ Virtualization (VMware) ■ IT Project Management ■ Startups/Turnarounds ■ Vendor Management ■ Training / Mentoring

4.0 What does the work look like in practice?

1. My engagements with stakeholders are both strategic and tactical. From a strategic standpoint, I get a big picture understanding of what they do and where they want to be. From a tactical standpoint, I work with them to execute actions to deliver the desired business outcomes.

2. Each project starts with the same basic methodology: Discovery/Planning, Change/Do, Delivery/Check, and Act/Analysis.

3


5.0 Engagement Scope and Examples: The work samples in this portfolio are taken from an engagement with the following broad objectives.

1. Internal operations for commodity and commercial infrastructure hosting engagements.

2. End-to-end systems design, planning and implementation. (Commercial design information not included.)

6.0 Background: From 2005 - 2012 LeeWare Development owned and operated a commodity hosting business. These assets were run out of publically available and private IDCs. 6.1. The service started with 8 gateway 2000 machines running on bus-topology connected to the internet using 1.1/1.1 Synchronous DSL connection with static IP addresses with a company called NorthPoint Communications. 2

Diagram 1.1 System Design:

Table 1.1 System Components.

Item Task Description

1 DNS01 - Domain Name System

2 DNS02 - Domain Name System

3 DC01 - NT-PDC-FS01

4 DC02 - NT-BDC-FS02

2 https://en.wikipedia.org/wiki/NorthPoint_Communications

4


5 WEB01 - Primary - FTP

6 WEB02 - Secondary - FTP

7 DEV01 - Content Distribution

8 ADM01 - Administration

9 DSL Point-to-Point 1.1Mbps

10 Public Internet

MP Photo 1.1 Equipment

The network basically consisted of gateway 2000 machines which were 486DX machines with 256 or 512MB of ram. The machines had internal SCSI drives and DDS-1 tape drives. They were arranged in a bus topology using Coax on a 10Mbps network. Eventually, these PCs were phased out for emachines that allowed for the installation of 120GB HDD and 2GB of RAM to be installed on board. I was an early virtualization adapter using VMware workstation for Linux and Xen. 6.2 Before opening and operating my own hosting facilities in Chicago, I rented assets at Internet Data Centers around the US. Chicago, Texas, and Florida. I had multiple modes at each of the locations. Most photos are from more than 10 years ago. Here are some computer pod photos: MP = Marketing Photos.

MP Photo 1.2 LDC EV1 500 Node JBOB Cluster Retail Texas

MP Photo 1.3 LDC 280 Node JBOB Cluster The planet Texas

5


MP Photo 1.4 LDC 65 Node JBOB Serverpronto FL

2000 - 2005. The bottom was falling out of the dedicated hosting business. The cost of a dedicated server dropped from around $300/month to around $100/month. You could get better equipment for a lower price. By 2003 Virtualization was the hot new thing. I played with VMware Workstation for Linux and QEMU in the lab to develop a product offering. (QEMU is a processor emulator, it allowed me to run Windows guests on Linux Machines.) This is basically what you can do with ESXi today. VMware was started in 1998. By 2001, I used VMWare GSX for some commercial hosting. Between 2005 and 2006 desktop class hardware supported hardware assisted virtualization. This development coupled with what I had learned to do with XEN on linux, using pssh, bash shell scripts, bridging QEMU, tar, and disk images. Created a serious technical advantage for LDC hosting. To appreciate the significance of this technical advantage you only need to know that most of what is taken for granted today in terms of functionality, such as: auto deploy, vmotion, VDS cloning etc didn’t exist or were crudely implemented at the time. 2005-2006 I started searching for a Chicago home for my hosting operations. Ameen and Chris didn’t move out of downtown to start GigeNET in Arlington Heights until 2006 but the stuff they were doing with DDoS protection was pretty spectacular at the time. There are a lot of established providers today that got their start around the same time. Some are peers I have consulted or mentored. The difference between them and me is they had outside or group funding for their operations whereas, I was a self-funded independent operator leveraging my credit and service revenues to fund the shape and direction of my service offerings. I never met the Owner of FDCServers Petr Kral outside of official communication. But I did meet the Chicago business partner who gave me a tour of the facility. They were running a very shaky operation, but I appreciated the honesty that for the price, they were basically operating a best-effort service. While this allowed me to avoid financial risks, it significantly undermined consumer confidence in terms of service reliability. I dropped all of my IDC rented assets, virtualized them into large linux based clusters and consolidated to this provider. They had lots of serious power and network related outages that costs me countless customers. At the time, FDC had a single location, they were Cogent’s biggest Chicago customer. Their facility is at 141 Jackson in the CBOT building on the 11th floor. They were hosting about 5,000 servers at the time. When I put my assets there, I was the fifth largest customer to the data center and the largest colocation customer.

MP Photo 1.5 FDCServers Data Center Tour: This is a photo of the original NOC. The desk with the two computers on it is still the NOC today, it has been expanded for two people and it is still located in the same position but has been walled off into a separate space. The 42 U cabinets were eventually moved to a separate location and the facility became jam packed with equipment over time.

6


MP Photo 1.6 FDCServers Data Center Tour: This is a photo of the ANSUL systems, CRAC, PDU, Core Routing Gear lower right and server storage racks. The next several photos show how the space pictured in the top two photo’s changed as more equipment was added to the space. Reference: Photos 1.7, 1.8, 1.9 1.10

MP Photo 1.7 FDCServers Data Center Tour:



7



MP Photo 1.11 FDCServers Data Center Tour: They started to expand into additional space by annexing the surrounding offices on their floor into their primary hosting space. I know their network engineer and the lead technician. I consulted with them on optimizing space utilization through pod design. They could optimize the efficiency of their space, cooling and power by controlling the form factors of the cases. You can see this improvement in the roll out of their Denver location: Reference Photos: 1.12, 1.13 and 1.14.

Most of this was implemented after I moved out of the Chicago facility and after I consulted with the guys on kick starting a Virtualization business for which I took 10% profit. MP Photo 1.12 FDCServers Data Center Tour:

8




6.3 LDC Commodity and Shadow IT IaaS:

1. LDC evolved into two hosting operations. The commodity site of the business basically involved providing best effort IaaS to the general public. Today you might by services from an MSP, a Private Cloud provider, Amazon EC2 and Microsoft Azure. This business accounted for ⅓ of the service revenues.

2. Shadow IT IaaS: This part of business represented contracts for providing IaaS IT services to

commercial interests such as: VARs, Financial firms, universities, telecom providers, research institutions, etc. ) Basically I put up the capital to build, maintain and operate infrastructure and then rent it out on a cost reimbursable contract where my return is anything in excess of my original investment. The downstream customer pays the ongoing cost of the service. This is an extraordinary risky business model as the potential for getting smoked is high. But for the risks, the up site is really good. Think about the current cloud computing model. Take a startup that needs to spend $200,000 to setup and operate infrastructure. As a provider, you provide said infrastructure for $10,000 per month. 20 months you break even. If the customer procures your services for 5 years or 60 months you gain $400,000. This number will be reduced by inflation, taxes, maintenance and other expenses. The more of your infrastructure you can reuse and share the more potential. Think economy of scale. The biggest risk is diminishing returns due to increased competition and that someone will come along and do the same things better, cheaper and more effectively. This business represented ⅔ of the service revenues.

9


3. At peak, the service had just over 1,000+ servers distributed across eight geographic locations. The commodity service was a combination of linux based dedicated and virtual machines and accounted for 300-400 machines. The rest of the machines 600-700 were all part of the Shadow IT service. I’m a big FAN of AT&T OPT-E-MAN and metro ethernet services in general, I have used it and VPLS on various projects.

Diagram 1.2 LDC Network Solution Architecture using AT&T Opt-E-MAN

When I put a 300+ machine at my office location in Schiller Park in 2007. Cogent added the Point-of-Presence to the official Cogent Network Diagram: I used 600Mbps to connect LOFs 1-4 and back hauled the internet connection over a 14-mile fiber loop to the CRG west building which is one of Cogent’s Chicago POP. Diagram 1.3 LDC Node on Cogent Global Map

10


6.4 LDC Computing PODs:

1. I used to buy pallets of machines retail. The MSRP was about $560 per box. But due to volume purchasing I could get them for around $300 per box. To purchase a 128 compute node costs $36,000 + tax. 256GB of RAM. To upgrade the RAM to 512GB it would cost $25,600+ tax. Retail would be double.

2. I would stage all of this stuff at my house, inventory it, upgrade the RAM, install linux, rent a uhaul

and install the stuff in the data centers. I was putting so much equipment in FDC that they made floor space for me so that all of my equipment to stay together. I would do everything in batches of 20. Eventually, i would register as a OEM and build my own PCs.

MP Photo 1.15 FDCServers Space Allocation: For a LDC 128 Node installation.

11


MP Photo 1.16 LDC Staging 128 Node Installation Commodity IaaS

MP Photo 1.17 LDC Staging 128 Node Installation Commodity IaaS

12


MP Photo 1.18 LDC FDC DC01 20 Node Capacity Expansion: Commodity IaaS


13




14


MP Photo 1.22 LDC Commodity Hosting 160 Node LOFX: Commodity IaaS

MP Photo 1.23 LDC Commodity 100 Node LOFX: Commodity IaaS

15


MP Photo 1.24 LDC Commodity Hosting 96 Node LOFX: Commodity IaaS

MP Photo 1.25 LDC IDF Cross Connect Front: Commercial IaaS

16


MP Photo 1.26 LDC IDF Cross Connect Back: Commercial IaaS

MP Photo 1.27 LDC OPT-E-MAN MPOE- Commercial IaaS

17


MP Photo 1.28 LDC Commercial hosting 600 Nodes- Commercial IaaS

MP Photo 1.29 LDC Commercial Hosting New Delivery Commercial IaaS

18


MP Photo 1.30 Commodity Hosting 100 node Capacity Expansion POD9 LOFX:

MP Photo 1.31 Commercial Hosting EOR NetPOP LOFX

19


MP Photo 1.32 Commodity Hosting 44 Node Capacity Expansion POD7 LOFX



20




21


MP Photo 1.37 Hosting 64 Node Capacity Expansion POD3 LOFX

MP Photo 1.38 LDC Commodity Hosting Routing Gear Metro Ethernet LOFX

22


MP Photo 1.39 LDC Commodity Hosting Metro Ethernet 6 x Point-to-Point LOFX and Fiber Loop to Cogent-POP

23


MP Photo 1.40 LDC Commercial Hosting Server Staging and Preparation.

MP Photo 1.41 LDC MIS Unsold Equipment. NetApp SAN, 150 PCs Dual Core AMD 2.2Ghz AMD-V 4GB RAM 250GB HDD. Boxes of new AMD-mATX MB. Boxes of Quad Core AMD CPUs, Boxes of 2GB RAM modules. Network Gear.

24


The hosting service consumed 15 /24 IP Blocks. or 3,500 IP addresses.

Item LDC IDC Network Blocks

1 66.90.121.0/24

2 66.90.74.0/24

3 67.159.29.0/24

4 69.90.126.0/24

5 67.159.31.0/24

6 208.53.129.0/24

7 67.159.36.0/24

8 66.90.112.0/24

9 67.159.30.0/24

10 67.159.3.0/24

11 67.159.4.0/24

25


Item LDC Metro Ethernet IP Blocks

1 38.102.48.0/24

2 38.106.195.0/24

3 38.100.101.0/24

4 38.106.199.0/24

6.5 Miscellaneous Screen Shots from Management Systems: I had a network accounting system that could provide deep insights into how nodes were using network resource. The tool was helpful for investigating abuse issues and determining resource use.

26


I used Citrix Xen Server for a bit. The following image shows the management console:

27


Finally, I used a portion of my large linux cluster to provide server slots for Popular Valve games. I ran the largest privately held L4D and L4D2 Game Cluster in the midwest. I used it to display Ads to thousands of gamers. The following screenshot shows what things look like when monitoring resources from my management console.

28

lee evans - ldc commodity hosting project portfolio 2005 - 2012

Documents