Trystack.org: The Free OpenStack Community Sandbox

Will Foster - Red Hat Kambiz Aghaiepour – Red Hat

2

TryStack?What is

3

✓ Free, Public OpenStack Sandbox

✓ Incepted in 2011, revamped/moved in 2015

✓ Sponsored by OpenStack Foundation

✓ Hardware/Resources Donated by Corporate Sponsors

✓ Volunteer/Community Operated

What

4

✓ Try OpenStack without any commitment

✓ Remove complexity of deployment

✓ Increase usage/adoption of OpenStack

✓ Showcase latest OpenStack features

✓ Vanilla OpenStack, all distribution images

Why

5

✓ Complete revamp/re-deployment in new DC

✓ Kilo → Liberty Newton→

✓ Triple original hardware capacity

✓ Doubled the public address space

2016-2017 Revamp

6

✓ Hosted in East Coast U.S. Datacenter✓ /23 Public IP address space✓ 7 x Dell C6105 Blades✓ 6 x Dell FX2 (FC430) Nodes✓ 8 x Dell r620 1U Servers (new)✓ Netapp FAS2552 (C-Mode)✓ Juniper EX4300 Switches✓ Dell Force10 10GbE Switches (new)

Where and How

7 Trystack Datacenter

8

I think this one fixed Neutron once

9

‣ Devstack

‣ Cloud App Testing

‣ RDO CI ‣ Manila Upstream Development (Netapp) ‣ Substantial Tire Kicking

TryStack Use Cases

10

Academia/Education

➢ University of Ljubljana● (Slovenia)

➢ Cork Institute of Technology● (Cork, Ireland)

TryStack Use Cases (continued)

11

Upstream Manila Development Using TryStack Manila Mitaka adds:

● Experimental share replication! (no GUI yet)● Improvements to share migration

Manila Newton adds:● Support for driver-assisted share migration

○ Possibility to optimize migrations between same vendor backends

● Official support for Red Hat OpenStack (tripleo)● Support for Manila in Mirantis OpenStack (fuel)● UI Support for share migration, replication

TryStack Use Cases (continued)

12

Tota

l Use

rs

AVG DAILY INSTANCES AVG MONTHLY INSTANCES Current Active Networks

300 7,500 800

26.5kAll time

4,759

Curr

ent A

ctiv

e U

sers

Usage and MetricsAs of 2016-10-22 @ 15:00 IST

13

14

✔ Puppet and Ansible (config mgmt, deployment)

✔ Graphite / Grafana / Collectd(graphing/trending)

✔ ELK: Elasticsearch, Logstash, Kibana(log aggregation)

✔ Nagios(monitoring)

✔ Browbeat(perf/scale testing)

Common Ops Tooling

15

Custom Nagios Monitoring

16

✓ Floating IPs ● Purged every 12hrs

✓ Network Gateways

● Cleared every 24hrs ✓ Cinder Volumes

● Purged every 48hrs

✓ Instances● Deleted every 24hrs

House Cleaning

17

18

19

✓ Trystack.org Website Content ● Managed via upstream CI & Gerrit processes● Automatically merged once reviewed

✓ Service Alerting● Tied into IRC bots for notifications● Email alerts, optional SMS

Further Automation

20

Demand and GrowthDozens of people sign up every day, small volunteer effort

Security Free, public cloud can lead to some 'interesting' usage

Rationing of ResourcesWe never seem to have enough resources to keep up with demand, e.g. floating IP space, so culling is needed

Record Keeping and AuditCustom tooling sometimes needed to record things like when a tenant sets a gateway, or tracking history of floating IP addresses

Challenges Cloud be hard, pal

21

Neutron Paper TrailWe needed a way to track/record history of all floating IP allocation and map this to tenant ownership

Network Sprawl We needed to create a way to track when router gateways were set

And Justice for AllOSPurge is great for removing tenant/user objects but is a “Judge Dredd” approach.

We need ways to overcome lack of network audit information not present in the database to ensure equal/fair usage for all tenants.

Custom Tooling: Resource and Audit nothing can save you

22

✓ Expand Hardware Footprint

✓ Newton and Ocata Deployment

✓ Proper Staging Environment

✓ OpenIDC / OpenStackID auth

✓ Phase out Facebook Auth

2016/17 Updates

23

✓ +9 x Dell R620 1U Servers∙ 24 vCPUs per node∙ 64 G Memory per node

✓ Upgraded to 10GbE Switches

New Donations!

24

✓ Newton-based deployment available sometime after Summit● Already in place, undergoing testing, openidc development/integration

✓ Ocata to follow in a few months

✓ Continue to be tested and burned in via Browbeat

✓ Cutover to 10GbE backend networking

✓ Netapp FAS2552 (C-Mode) integration for backend storage

New TryStack Deployment

25

Move to OpenID Connect supported Federation via Apache (mod_auth_idc)

Use FreeIPA 4.x self service as primary auth until openstackid is ready to consume alongside upstream infra

Deprecate Facebook Oauth2 provider federation✓ Use mod_auth_idc as Oauth 2.0 resource for Facebook until we can fully

move to openstackid

Federation and OpenIDC

Trystack Keystone

26

✓ Liberty Deployment (active)

∙ 7 x Nova Compute∙ 1 x Neutron∙ 1 x Controller

✓ Newton Deployment (staging)∙ 6 x Nova Compute∙ 1 x Controller+Neutron

✓ Consolidated Newton (final)

∙ 15 x Nova Compute∙ 1 x Neutron+Controller

Deployment Details

27

✓ Browbeat orchestrates Rally workloads and other tools

Burn-in with Browbeat

✓ Browbeat identifies tuning recommendations and CVEs

28

Rally via Browbeat

29

Grafana during Rally Runs

30

Founders/Alumnus‣ Dan Radez @radez‣ Nachi Ueno @nati‣ Jay Pipes @jaypipes‣ Todd Deshane @deshantm‣ Anne Gentle @annegentle

Current DevOps Team ‣ Kambiz Aghaiepour @kaghaiepour‣ Will Foster @sadsfae / https://hobo.house‣ Ben Swartzlander (Manila PTL/dev) @bswartzlander Moderators - Thanks to:Matthias Runge and Rain Leander

TryStack Devs/Operators

Resources https://github.com/trystack/trystack-utils

https://github.com/trystack/trystack-collectdhttps://github.com/openstack/ospurgehttps://github.com/openstack/browbeat

IRC: #trystack on irc.freenode.netWEB: trystack.org

Will Foster - @sadsfae – http://hobo.house - wfoster on IRCKambiz Aghaiepour - @kaghaiepour - kambiz on IRC