devops and cloud at ni

34
National Instruments

Upload: ernest-mueller

Post on 15-Jan-2015

1.421 views

Category:

Technology


4 download

DESCRIPTION

My presentation at the October Agile Austin DevOps SIG about how we implemented DevOps on my team at National Instruments - techniques we used and lessons learned.

TRANSCRIPT

Page 1: DevOps and Cloud at NI

National Instruments

Page 2: DevOps and Cloud at NI

DevOps and Cloud at NI

Ernest MuellerCloud Systems Architect, LabVIEW R&D

[email protected] @ernestmueller http://theagileadmin.com

Page 3: DevOps and Cloud at NI

3

The Short Form

We built a DevOps team to rapidly deliver new SaaS products and product functionality using cloud hosting and services (IaaS, PaaS, SaaS) as the

platform and operations, using model driven automation, as a key differentiating element.

With this approach we have delivered multiple major products to market quickly with a very small staffing

and financial outlay.

Page 4: DevOps and Cloud at NI

4

National Instruments (aka “NI”)

• 30 years old; 5000+ employees around the world, half in Austin, mostly engineers; $873M in 2010

• Hardware and software for data acquisition, embedded design, instrument control, and test

• LabVIEW is our graphical dataflow programming language used by scientists and engineers in many fields

Page 5: DevOps and Cloud at NI

5

From toys to black holes

Page 6: DevOps and Cloud at NI

6

Genesis

• Our hardware and software product strategy started to spawn software-as-a-service ideas – some from customer demand, some from internal drivers

• There were existing product to Web integration points but these were uncoordinated and poorly maintained

• LabVIEW R&D greenfielded an internal group in 2009 to serve as an internal ISV for hosted services, the dotCom team, seeded by a couple key folks from IT

Page 7: DevOps and Cloud at NI

7

Challenges• Lack of close R&D/IT relationship• Traditional siloed IT department (programmers split by

business unit, infrastructure split by technology)• Low organizational agility – 6 weeks to get a server• Uptime problems from complexity and silos• On premise data centers at power/cooling capacity• R&D primarily experienced in desktop software and

specialized, dedicated hardware, not server/Web scale• Consensus driven environment

Page 8: DevOps and Cloud at NI

8

More Challenges• IT software releases painful – quarterly for ERP, monthly for

Web – Web often having 80+ release items, many manual.• Production issues (availability, performance, security) would

persist for months on average awaiting resolution• Waterfall system development process created friction

against dev teams uptaking agile methods• Software often not designed with any service management

goals in mind• Dev teams frustrated with low throughput of

changes/requests from systems team

Page 9: DevOps and Cloud at NI

9

Now, Discover Your Strengths

• Strong base of “best and brightest,” motivated employees• Culture of innovation and “do it yourself”• Large Web presence (ni.com) with extensive in house

programming and operational experience• Entrepreneurial internal environment• Significant reinvention/retooling effort going on in R&D• Increasing focus on system sales and quality (performance,

reliability, security) over yet-more-features

Page 10: DevOps and Cloud at NI

10

DevOps Ideation• We had developed a Systems Development

Framework we used with dev teams, but it was an inherently waterfall model

• Velocity conference began to spread Ops best practices and catalyze innovation in 2008

• OpsCamp Austin (Jan 2010) introduced us to the Visible Ops book and the “DevOps” term. “Hey, so that’s what we’ve been trying to do!”

• Able to move forward quickly with more confidence

Page 11: DevOps and Cloud at NI

11

Sidebar: What Is DevOps?

• When I say DevOps, I mean: Developers and operations staff both included as part of

the project team – NOT developers throwing over a wall to QA who throws over a wall to operations

Developers and operations staff collaborating on service design, development, testing, release, and management – availability, capacity, security, etc. all need work at design time and have obvious operational requirements

Page 12: DevOps and Cloud at NI

12

Starting Fresh - Blessing and Curse

Everything was new, so we had to simultaneously develop:• Products• Team• Process• Systems• Code• Operations• System Automation

Page 13: DevOps and Cloud at NI

13

The Products (“Hosted Services”)

• Customer facing: LabVIEW Web UI Builder (2009) LabVIEW FPGA Compile Cloud (2010) Technical Data Cloud (2011) More in progress!

• Internal facing (NI product teams are the customer): LabVIEW.com Cloud Framework Cloud Hosting Operations

Page 14: DevOps and Cloud at NI

14

LabVIEW Web UI Builder

• Write a LabVIEW(ish) app, save it to the cloud and run it there. • Build and deploy it to an embedded target and hook it up to

Web services to give it a sweet UI• Also, an experimental testbed for LabVIEW changes• Freemium model – use it for free, packaging and deploying

your app to a target requires a license (compiles run in the cloud) – try it at ni.com/uibuilder

• Silverlight RIA, back end on Amazon Web Services – EC2, S3, SimpleDB; Java/Linux/Apache/Tomcat and .NET/IIS/Windows

Page 15: DevOps and Cloud at NI

15

Page 16: DevOps and Cloud at NI

16

Page 17: DevOps and Cloud at NI

17

LabVIEW Web UI Builder Cloudlet

BrowserWebLV

Services

WebLVCompiler

LDAP

InternalServices

DNS LoadBalancer

InstallServices

MgmtServer Gateway File

Server

Routing

Web

Ser

ver Security

Services ProjectData

Auth Db

License Db

PIE

Page 18: DevOps and Cloud at NI

18

LabVIEW FPGA Compile Cloud

• LabVIEW FPGA compiles take hours and consume extensive system resources; compilers are getting larger and more complex

• Implemented on Amazon - EC2, Java/Linux,C#/.NET/Windows, and LabVIEW FPGA

• Also an on premise product, the “Compile Farm”

Page 19: DevOps and Cloud at NI

19

LabVIEW FPGA Compile Cloud

NI Hosted Compile Service

User Login& Rights management

Links to user account& support

Page 20: DevOps and Cloud at NI

21

Technical Data Cloud

• “I just want to upload my sensor data directly to the cloud, man.”

• REST and LabVIEW API that lets you upload and retrieve discrete and waveform data

• Welcome to the Internet of Things• Being built on Microsoft Azure – specific bits TBD,

all .NET and LabVIEW

Page 21: DevOps and Cloud at NI

22

LabVIEW.com Cloud Framework• Platform that makes the magic happen by providing base

plumbing for developers of SaaS apps• Core Services - reusable Web services and facilities • ILLS (internal login & licensing services) – distributed user

repo and licensing, complete with feed from Oracle and self service user portal; Java/Tomcat, OpenDS LDAP, mySQL

• PIE (Programmable Infrastructure Environment) – sets up systems for you, autoscales, deploys code; uses an XML model and runtime registry; Java – more on this later!

• Building out a core platform? Didn’t that slow velocity? No.

Page 22: DevOps and Cloud at NI

24

The Team• DevOps!

Application architect Systems architect (me) 2 developers 1 system automation developer Operations lead 2 follow-the-sun operations staff in Malaysia

• Work with other R&D product development teams Different orgs (LabVIEW, non-LV software, hardware) Geographically distributed (Austin, Aachen, Bangalore, Singapore)

Page 23: DevOps and Cloud at NI

25

The Process• Agile!• All systems work used the “developer” tools and systems as

part of DevOps collaboration philosophy Revision control (Perforce) Bug tracking (HP Quality Center) Specs and reviews (Atlassian Confluence wiki) Task tracking and burndown (JIRA/Greenhopper -> TFS)

• All members collaborate on all aspects of the product• Test driven development – well, ideally

Page 24: DevOps and Cloud at NI

26

The Systems• Cloud!• After a quick cost assessment and experimentation, decided

on Amazon EC2 as our initial hosting platform, added Azure later due to business relationship

• Needed control and agility we wouldn’t be able to get internally – dynamic requirements, fast scaling

• Needed Linux and Windows both for software support• Using multiple point SaaS providers for functionality (If it’s not

core, outsource it!)• Agility and time to market far outweighed cost efficiency

Page 25: DevOps and Cloud at NI

27

Code

• REST!• All REST-based Web services• Multiple tech stacks - cloud and systems mgmt code mostly

in Java, product code mostly in C#/.NET• Key cloud app architecture concerns – multitenant, parallel,

asynchronous, loosely coupled, APIed, instrumented, resilient in dynamic/ephemeral environment

• Developers deliver tests, monitoring, system model with their service

Page 26: DevOps and Cloud at NI

28

Operations

• The “secret sauce”!• Not just ticket handling or “keep the lights on.” Focus on

delivering value to the customer and developer.• Provide performance management, availability, systems

management, incident handling, security, log management, monitoring, rapid deployment

• Inspirations: O’Reilly “Secret Sauce” paper, Velocity conference, Visible Ops book, Transparent Uptime blog

Page 27: DevOps and Cloud at NI

29

Sidebar: What Is Operations?

• Operations is a domain of expertise within IT systems management that involves the planning, design, deployment, operation, security, maintenance, monitoring, tuning, and repair of applications and systems.

• Includes: systems engineers, systems administrators, release managers, support techs

• It is a set of knowledge used across lifecycle phases

Page 28: DevOps and Cloud at NI

30

System Automation

• PIE!• The “Programmable Infrastructure Environment”• XML system model defines systems, services, code installs,

runtime interaction• Runtime registry for systems info and eventing • PIE autobuilds the runtime system from the model –

provisioning, software installs, monitoring integration• Perform orchestration and control on many instances of

dynamic environments

Page 29: DevOps and Cloud at NI

31

Page 30: DevOps and Cloud at NI

32

Results

• Win!• A continuous pipeline of products delivered quickly• LabVIEW Web UI Builder went beta in 2009, 1.0 in

2010• FPGA Compile Cloud went beta in 2010, 1.0 in 2011• Technical Data Cloud going beta in 2011• Unqualified happiness with cloud, DevOps approach• Not innovation vs. reliability – new approach gets both!

Page 31: DevOps and Cloud at NI

33

Bottom Line

• Operations is no longer a basic facilities-like “gate” – many modern applications (our cloud applications, for sure) require extensive systems engineering for functionality, customer satisfaction, and profitability

Page 32: DevOps and Cloud at NI

34

Residual Challenges/Lessons Learned• Culture change – building collaboration, mutual respect, and

trust among globally distributed dev teams, ops, and others (QA, security, etc.)

• Educating developers on operational issues and vice versa• General agile challenges (large teams, global teams)• New agile infrastructure challenges (what does unit testing

mean for my environment build, and how do I do that?)• Maintaining vision through rapid change• Cloud-compatible tooling still emerging• Selling the products; product manager collaboration

Page 33: DevOps and Cloud at NI

35

Where do we go from here?

• Continue to evolve PIE to add better orchestration, treat data as a first class citizen

• Move to full continuous integration and deploy-on-demand, necessitating intense investment in testing

• Uptake of Microsoft Azure (mostly complete) as an additional cloud provider

• Private cloud integration; OpenStack/virtualization• More SaaS products!

Page 34: DevOps and Cloud at NI

36

Discussion!