moving forward under the weight of all that state
TRANSCRIPT
![Page 1: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/1.jpg)
Upgrading under the weight of all that
stateQuinton Anderson
![Page 2: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/2.jpg)
Context
![Page 3: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/3.jpg)
Canonical Model
![Page 4: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/4.jpg)
Source
Source
Source
Source
Raw Data
Business Data
Access Layer
Access Layer
Access Layer
Access Layer
![Page 5: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/5.jpg)
Load Balance
r
//TODO
Function
Cntrl-V
Scaling
![Page 6: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/6.jpg)
![Page 7: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/7.jpg)
Downstream systems• Specialised
management systems
• Reporting Systems• Product management
Channel & product systems
Master Data Management
Hadoop• Leverage all data & reduce
integration costs• Comprehensive dataset –
internal & external, realtime & batch, structured & unstructured
• Advanced analytics / machine learning
Group Data Warehouse• Understand our business• Accurate, conformed, and
reconciled data• Access layer to support BI &
reporting
BI/Reporting• User facing tools• Regulatory reporting• Dishoarding• Self service BI for the
masses
Customer record &insights
All data
Price, conversation,credit dec. etc.
Financial Data
Subset ofdata
Useraccess
‘ Reconciled
data
Information for people
Core Financial Systems and functions• P&L• Recon• General Ledger• Etc…
Closed loop,automated ‘decisions’
Decisioning• Personalise/optimise decisions,
maximise customer value• E.g. price, credit decision, next
conversation, experience
Core information repositories
Analytics applications
Other systems
![Page 8: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/8.jpg)
Channels
Had
oop
Rules
Serving and decisioning
AnalyticRecords
Systems Of Record
Core Banking Payments
Event Processor
Raw Data
Derived Data
Feature Store
Event Store
Scoring
Machine Learning
www
Event Streams
Customer Information data loaded
Data analysed
& processed
Insights & events captured
Integration API/Service Discovery
![Page 9: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/9.jpg)
> 4000 Daily Batch Jobs
![Page 10: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/10.jpg)
> 6 PB of State and growing
![Page 11: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/11.jpg)
Hbase, Cassandra,
HDFS, Influx,
Elastic Search, Kafka,Etcd,
ZookeeperOpenStack Swift
![Page 12: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/12.jpg)
Oracle,MySQL,
Postgres
![Page 13: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/13.jpg)
Hundreds of services
![Page 14: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/14.jpg)
MR1,MR2,
Spark,Akka
![Page 15: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/15.jpg)
Dev,Test,
Staging,Prod 1,Prod 2,Etc…
![Page 16: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/16.jpg)
== Complexity
![Page 17: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/17.jpg)
Imperative:
![Page 18: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/18.jpg)
![Page 19: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/19.jpg)
![Page 20: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/20.jpg)
![Page 21: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/21.jpg)
Culture
![Page 22: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/22.jpg)
![Page 23: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/23.jpg)
![Page 24: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/24.jpg)
Architecture
![Page 25: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/25.jpg)
Immutable
![Page 26: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/26.jpg)
Someone else’s computer
![Page 27: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/27.jpg)
State Locality
![Page 28: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/28.jpg)
Workload non-locality
![Page 29: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/29.jpg)
Flexible over optimal
![Page 30: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/30.jpg)
Practically, it is a closed system
![Page 31: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/31.jpg)
State management is my problem
![Page 32: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/32.jpg)
All abstractions are leaky
![Page 33: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/33.jpg)
Repo(s) CI/CD Apps
Docker CalicoMesos Yarn
Spark, MR, Impala, etcMarathon, Chronos, Cassandra, etc
CI/CD
CI/CD
Repo(s)
Repo(s)
Open Stac
k
Nova
Nova/Ironic
OSKVM
OSFirmware + Hardware + Tags
![Page 34: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/34.jpg)
Strategies
![Page 35: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/35.jpg)
Outsource the problem, and tool away the resulting issues
![Page 36: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/36.jpg)
Delete it, tool away the resulting issues
![Page 37: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/37.jpg)
Be stateless, tool away the resulting issues
![Page 38: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/38.jpg)
Implement some patterns, incrementally optimise. Tool away the resulting issues
![Page 39: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/39.jpg)
Excess Capacity
![Page 40: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/40.jpg)
Patterns
![Page 41: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/41.jpg)
Consumer
Router
DB
Old Old
Web
App
DB
Web
App
![Page 42: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/42.jpg)
Consumer
Router
DB
Old Old
Web
App
DB
Web
App
![Page 43: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/43.jpg)
L4
HAProxy
Old Old Old Old
![Page 44: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/44.jpg)
L4
HAProxy
Old Old Old Old New
![Page 45: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/45.jpg)
L4
HAProxy
Old Old Old Old New
![Page 46: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/46.jpg)
L4
HAProxy
Old Old Old Old New
![Page 47: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/47.jpg)
L4
HAProxy
Old Old Old New New
![Page 48: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/48.jpg)
L4
HAProxy
Old Old New New New
![Page 49: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/49.jpg)
L4
HAProxy
Old New New New New
![Page 50: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/50.jpg)
L4
HAProxy
New New New New New
![Page 51: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/51.jpg)
== Incrementally accept risk
![Page 52: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/52.jpg)
In place upgrade
![Page 53: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/53.jpg)
Stateful
![Page 54: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/54.jpg)
![Page 55: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/55.jpg)
CAP, PACELC
![Page 56: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/56.jpg)
Data models
![Page 57: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/57.jpg)
Atomicity
![Page 58: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/58.jpg)
Access patterns
![Page 59: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/59.jpg)
Implementation approaches = ??
![Page 60: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/60.jpg)
Upgrade Duration O(N)
![Page 61: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/61.jpg)
for node in nodes: if info[node]['instance']:
if Status(node).run().wait() == AVAILBLE_FOR_MAINTENANCE:
MaintenanceMode(node).run().wait()Upgrade(node).run().wait()Health = HealthTests(node).run.wait()UpdateStatus(node, health).run.wait()
![Page 62: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/62.jpg)
all_good = Truehost = self.cdh.get_host(self.host_map[self.node_name])if host.healthSummary != 'GOOD': all_good = False
# Look up the host by its rolesfor c in self.cdh.get_all_clusters(): for s in c.get_all_services(): for r in s.get_all_roles(): h = r.hostRef if h.hostId == self.host_map[self.node_name]:
if r.healthSummary != 'GOOD': all_good = False
return all_good
![Page 63: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/63.jpg)
O(log N)
![Page 64: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/64.jpg)
![Page 65: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/65.jpg)
nodeComputation = for {_ <- Status(node)_ <-
MaintenanceMode(_,node)_ <- Upgrade(node)nodeResult <- HealthTests(node)
} yield nodeResult
upgrade = for {node <- groupcomp <- nodeComputation(node)
} yield comp.exec
groups.map(upgrade)
![Page 66: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/66.jpg)
Repo(s) CI/CD Apps
Docker CalicoMesos Yarn
Spark, MR, Impala, etcMarathon, Chronos, Cassandra, etc
CI/CD
CI/CD
Repo(s)
Repo(s)
Open Stac
k
Nova
Nova/Ironic
OSKVM
OSFirmware + Hardware + Tags
![Page 67: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/67.jpg)
Workflow
![Page 68: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/68.jpg)
Jenkins
Environment
Branch PR
Merge
Dev
Deploy
Master
Deploy
TestChange Plan
![Page 69: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/69.jpg)
clusters: green-cluster: dns: nameservers: - x.x.x.x data_domain: *.*.* etcd: token: green-cluster masters: able: provision_id: 1 lan: - mac: 0c:c4:7a:c1:2e:92 ip: 1.1.11.151/24 vlan: 11 gateway: 1.1.1.1 ironic_id: a7af76ad-6583-4209-ba5f-cf1477b6405e flavor: ramish-baremetal-flavor2 image: *mesos-master-green theta: provision_id: 2 lan: - mac: 0c:c4:7a:a9:04:0c ip: 1.1.11.53/24 vlan: 11 gateway: 1.1.1.1 ironic_id: 8ff1fd1c-4893-11e6-a447-2f366077ca0e flavor: ramish-baremetal-flavor2 image: *mesos-master-green tobias: provision_id: 3 lan: - mac: 0c:c4:7a:a8:f6:ac ip: 1.11.11.52/24 vlan: 11 gateway: 1.1.1.1 ironic_id: c89fdd08-232c-40fe-b965-49fc3e4dcba7 flavor: ramish-baremetal-flavor2 image: *mesos-master-green
![Page 70: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/70.jpg)
![Page 71: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/71.jpg)
Recommendations
![Page 72: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/72.jpg)
Instrument as much of deployment and provisioning as you can
![Page 73: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/73.jpg)
Optimise incrementally, learn the right hard lessons
![Page 74: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/74.jpg)
Allow for manual intervention, but attack it aggressively
![Page 75: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/75.jpg)
Encourage your people to intervene
![Page 76: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/76.jpg)
Prevent Pets
![Page 77: Moving forward under the weight of all that state](https://reader035.vdocuments.net/reader035/viewer/2022070600/58ec8ed61a28ab75158b4671/html5/thumbnails/77.jpg)
Spend more time on testing