S3IT: Service and Support for Science IT
OpenStack for academicuse-cases ?Sergio MaffiolettiS3IT: Service and Support for ScienceIT,University of Zurich
22.03.2016
What is S3IT and who am I
S3IT - Service and Support for ScienceIT:http://www.s3it.uzh.ch.
Enabling research through Science IT competences.
S3IT is an academic solution provider (end-to-end).
Sergio MaffiolettiSenior Infrastructure and Application specialist atUniversity of Zurich.
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Overall message
Build and adaptinfrastructure to use-cases
and not the other way round.
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Let’s start with someexamples
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Research needs to do more than just dataanalysis.
Personalized server
Large-scale data analysis and simulations
Integrate solution
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Research needs to do more than just dataanalysis.
Personalized server→ RStudio/R Shiny→ Windows image analysis
Large-scale data analysis and simulations
Integrate solution
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Research needs to do more than just dataanalysis.
Personalized server
Large-scale data analysis and simulations
→ Hadoop/Spark→ Own SLURM cluster→ Matlab and toolboxes
Integrate solution
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Research needs to do more than just dataanalysis.
Personalized server
Large-scale data analysis and simulations
Integrate solution
→ from instrument to store, to processing to makedata accessible.
UZH/S3IT OpenStack for academic use-cases 22.03.2016
What is OpenStack ?
UZH/S3IT OpenStack for academic use-cases 22.03.2016
What is OpenStack and how we use it
→ Self-provisioning of Virtual Instances, Storage andNetwork→ Elastic allocation of resources on demand→ Multi-tenancy
UZH/S3IT OpenStack for academic use-cases 22.03.2016
OpenStack project
OpenSource projectask our sysadmin why this is important. . .
Foundation supported by more than 200 companies.https://www.openstack.org/foundation/companies/
Network APIs→ Amazon compute and storage cloud compliant→ Bindings for many languages
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Let’s implement some of theexamples
UZH/S3IT OpenStack for academic use-cases 22.03.2016
What approach ?
→ Provision, Configure and Orchestrate basecomponents: network, storage, compute.→ Build on top of baseline infrastructure.→ Customize and tailor to use-cases needs.
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Personalized server: R Shiny
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Personalized server: R Shiny
1. Create own instance
2. Automated customization (e.g. ansible)
3. Application-specific settings
4. Save base image for further use
UZH/S3IT OpenStack for academic use-cases 22.03.2016
R Shiny: Create own instance
Instances are virtual machines that run inside the cloud.
UZH/S3IT OpenStack for academic use-cases 22.03.2016
R Shiny: Automated customization (e.g.ansible)
Ansible is a free-software configuration managementplatform for configuring and managing computers.
UZH/S3IT OpenStack for academic use-cases 22.03.2016
R Shiny: Application-specific settings
Security groups are sets of IP filter rules that are applied toan instance’s networking.
UZH/S3IT OpenStack for academic use-cases 22.03.2016
R Shiny: Save base image for further use
The OpenStack snapshot mechanism allows to create newimages from running instance.
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Create your own batch cluster: SLURM
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Create your own batch cluster: SLURM
1. Provision cluster instances (1 frontend + Xcompute)
2. Create own data volume and attach it to frontend
3. Install and configure frontend and compute
4. Export via NFS data volume to all compute nodes
5. Start NFS and slurm services
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Create your own batch cluster: SLURM
. . . or use elasticluster: http://gc3-uzh-ch.github.io/elasticluster/
https://youtu.be/gkXkcHDd588
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Why flexibility: from our perspective
→ Reduce total time to get actual results→ Allow to run on multiple infrastructures(portability)→ Automate (more research, less development forend-user)
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Why flexibility: from user perspective
Self-provisioning and Elasticity of resources
Customization and control of the environment
Network API
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Why flexibility: from user perspective
Self-provisioning and Elasticity of resources→ End-users can allocate and release resourceswhen needed.
Customization and control of the environment
Network API
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Why flexibility: from user perspective
Self-provisioning and Elasticity of resources
Customization and control of the environment→ End-users can tailor the research infrastructureto their specific needs.
Network API
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Why flexibility: from user perspective
Self-provisioning and Elasticity of resources
Customization and control of the environment
Network API→ To programmatically create and control an ownresearch infrastructure.→ Build services on top
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Downsides
→ Resource allocation: policy is basically done bysupport team.→ Efficient use of resources: users need to takeresponsibility.→ Need to have strong support team to complementthe offer to end-users.→ Ops costs comparable as for regular batch clusterbut know-how has to be developed.→ Storage solution has to be planned separately andmostly depends on use-cases.
UZH/S3IT OpenStack for academic use-cases 22.03.2016
Conclusions
→ Cloud is flexible: build and adapt infrastructure touse-cases and not the other way round.→ Users have more possibilities but also moreresponsibilities.→ Easier for end-users to think and plan theirresearch infrastructure.
UZH/S3IT OpenStack for academic use-cases 22.03.2016