turning software projects into production solutions dan fraser, phd production coordinator open...
TRANSCRIPT
Turning Software Projects Turning Software Projects into into
Production SolutionsProduction Solutions
Dan Fraser, PhDDan Fraser, PhD
Production CoordinatorProduction Coordinator
Open Science GridOpen Science Grid
OU Supercomputing SymposiumOU Supercomputing Symposium
October 2009October 2009
Four QuestionsFour Questions
-What gives some software -What gives some software projects “staying power” projects “staying power” while others stagnate ?while others stagnate ?
-What is the most important -What is the most important thing for a sustainable thing for a sustainable software project ?software project ?
-What is the difference -What is the difference between a software project between a software project and a software solution ?and a software solution ?
-What is needed to create a -What is needed to create a “production” quality “production” quality solution ?solution ?
My backgroundMy background LANL, Air Force Supercomputing, Thinking LANL, Air Force Supercomputing, Thinking
Machines, NEC, Sun, Paremus Ltd., Globus, Machines, NEC, Sun, Paremus Ltd., Globus, Open Science GridOpen Science Grid
Software development, software support, user Software development, software support, user and application support, software scalability, and application support, software scalability, production system management…production system management…
I have never fully appreciated the answers to I have never fully appreciated the answers to these questions until I worked on Productionthese questions until I worked on Production
And I’m still learning …
Software withSoftware with “Staying Power” “Staying Power”
Microsoft WindowsMicrosoft WindowsHow many of you How many of you run linux as the run linux as the primary os on your primary os on your laptop ?laptop ?
Mac does not countMac does not count
It solves the same It solves the same “problem” as windows“problem” as windows
GridFTP, GRAM2, GridFTP, GRAM2, MyProxy, Nagios, MyProxy, Nagios, Ganglia, Rocks, Ganglia, Rocks, CFEngine, SSH …CFEngine, SSH …
Question 1Question 1What gives some software projects “staying What gives some software projects “staying power” while others stagnate ?power” while others stagnate ?
Question 1Question 1What gives some software projects “staying What gives some software projects “staying power” while others stagnate ?power” while others stagnate ?
The answer is NOT:The answer is NOT: FundingFunding
Microsoft did not get big because it had lots of cashMicrosoft did not get big because it had lots of cash
MyProxy has not been funded for over 5 years.MyProxy has not been funded for over 5 years. But the team still fixes bugs on weekends…But the team still fixes bugs on weekends…
Some answers include:Some answers include: It solves a problem that users need solvedIt solves a problem that users need solved It “works” in the user environmentIt “works” in the user environment The community supports itThe community supports it
Solutions that “work”Solutions that “work”
When software When software solutions “work”, solutions “work”, even though they even though they are not perfect, it is are not perfect, it is hard to get users to hard to get users to migrate away from migrate away from themthem
Anyone remember Anyone remember plot10 ?plot10 ?
Question 2Question 2
What is the most important thing for a What is the most important thing for a sustainable software project ?sustainable software project ?
Question 2Question 2What is the most important thing for a sustainable software project ?What is the most important thing for a sustainable software project ?
Answer: Users.Answer: Users.
Production Ξ UsersProduction Ξ Users
The “users” are not always who we thinkThe “users” are not always who we think
GridFTPGridFTPA well engineered production codeA well engineered production code
Used in all the High Energy Physics Used in all the High Energy Physics experiments + many other placesexperiments + many other places
Easy to install (make install gridftp)Easy to install (make install gridftp)
We did not always understand end user We did not always understand end user difficulties.difficulties. ““Works great one day, try again one week later and Works great one day, try again one week later and
it fails” it fails” Problem with CRLs (security issues)Problem with CRLs (security issues) GSI is the software “environment”GSI is the software “environment”
GridFTP is most successfulGridFTP is most successful
In places where the security environment is well In places where the security environment is well maintainedmaintained
Where end users don’t necessarily see GridFTP Where end users don’t necessarily see GridFTP directly.directly. TeraGrid developed TGCP, a friendly user interface to TeraGrid developed TGCP, a friendly user interface to
GridFTPGridFTP UberFTP and other interfaces widely usedUberFTP and other interfaces widely used GridFTP is an GridFTP is an embedded embedded component at all the HEP sitecomponent at all the HEP site OSG packages GridFTP as part of a complete solution OSG packages GridFTP as part of a complete solution
(a Storage Element) including security(a Storage Element) including security
The main “user” is often the “integrator”
Integration is CriticalIntegration is Critical
Software integration Software integration is not a dirty word, is not a dirty word, but it can be dirty but it can be dirty work.work.
Integration is a Integration is a critical component critical component in production in production solution solution developmentdevelopment
Find an Integrator to work with !
Question 3Question 3
What is the difference between a software project What is the difference between a software project and a software solution?and a software solution?
A solution is something greater than the sum of its A solution is something greater than the sum of its software componentssoftware components
Examples:Examples:GridFTP is a software project GridFTP is a software project
UI + GridFTP + Authz framework + Credential mgmt = a solutionUI + GridFTP + Authz framework + Credential mgmt = a solution
GRAM2 is a software projectGRAM2 is a software project
Condor-G + Gram2 + GridMon = a solutionCondor-G + Gram2 + GridMon = a solution
Ganglia is a software project that does monitoringGanglia is a software project that does monitoring
Ganglia + something that acts on the data = a solutionGanglia + something that acts on the data = a solution
Solutions create new problemsSolutions create new problems
Multiple components must work togetherMultiple components must work together Which components do I need?Which components do I need? Which versions work together?Which versions work together? I need a new feature. Can I upgrade one I need a new feature. Can I upgrade one
component, or will it break everything ?component, or will it break everything ? When a solution breaks, which component is When a solution breaks, which component is
to blame ?to blame ? How do I get resolution when component How do I get resolution when component
providers are blaming each other ? providers are blaming each other ?
Question 4Question 4
What is needed to What is needed to create a “production” create a “production” quality solution ?quality solution ?
Answer: A good process Answer: A good process Putting components together Putting components together that work and solve a problem that work and solve a problem for a communityfor a community
Processes can be Processes can be reliable even with weak reliable even with weak components !components !
What is needed for a Production process?What is needed for a Production process?
-Coordination with -Coordination with software providers and software providers and communitescommunites
-Integration-Integration
-Testing-Testing
-Monitoring-Monitoring
-Solution Releases -Solution Releases
-Operations-Operations
-…-…
OSG – A good process exampleOSG – A good process example
-Projects in the OSG -Projects in the OSG software stack have software stack have good “staying power”good “staying power”
-Projects get into OSG -Projects get into OSG by being demanded by by being demanded by the user community.the user community.
-OSG provides the -OSG provides the “production” process“production” process
OSG Enabled Production SolutionsOSG Enabled Production Solutions
HEP AnalysisHEP Analysis: HEP, CMS, D0, CDF, …: HEP, CMS, D0, CDF, …
Gravitational Wave: Gravitational Wave: #2 in the world#2 in the world
Protein structure prediction: Toyota InstituteProtein structure prediction: Toyota Institute
Weather Research Forecasting: Weather Research Forecasting: UNC, U NebraskaUNC, U Nebraska
Structural Biology Predictions: Structural Biology Predictions: Harvard MedicalHarvard Medical
NanotechnologyNanotechnology Simulation and Modelling: Purdue, Simulation and Modelling: Purdue, Nanohub collaboration.Nanohub collaboration.
Molecular Dynamics: Molecular Dynamics: U of Buffalo, ArgentinaU of Buffalo, Argentina
Theoretical Nuclear PhysicsTheoretical Nuclear Physics: Duke University..: Duke University..
Text Mining: Text Mining: U. North CarolinaU. North Carolina
Exploring Mathematical algorithmsExploring Mathematical algorithms: U. Colorado: U. Colorado
On creating production solutionsOn creating production solutionsUnderstand what the problems areUnderstand what the problems are Understand what the users really want to doUnderstand what the users really want to do Make sure there are enough likely usersMake sure there are enough likely users Understand the barriers to adoptionUnderstand the barriers to adoption
Understand the Understand the environment environment where the where the solution will livesolution will live
Work with integrators (or be one)Work with integrators (or be one)
Know and nurture your user communityKnow and nurture your user community