an iterative approach to service oriented architecture
Post on 27-Aug-2014
776 Views
Preview:
DESCRIPTION
TRANSCRIPT
Iterative Service Oriented
Architecture
Eric Saxby@sax @ecdysone @sax
From thisFrom this
To thisTo this
Why should you listen to me?
Proprietary and Confidential
Why should you listen to me?
Proprietary and Confidential
■ I have many leather bound books
Why should you listen to me?
Proprietary and Confidential
■ I have many leather bound books
■ My apartment smells of rich mahogany
Why should you listen to me?
Proprietary and Confidential
■ Application developer Many various technologies. Rails for ~6 years."
■ Recent focus has been operational"
■ Such acronyms, so buzzwordTDD, Agile, DevOps, SOA, Cloud, IaaS
■ Broken a surprising amount of things in a short time"
■ Work at Wanelo
Wanelo? Whaaa?
Wanelo? Whaaa?
■ Social network for shoppingIf you buy things on the Internet, you are our demographic
Wanelo? Whaaa?
■ Social network for shoppingIf you buy things on the Internet, you are our demographic
■ Many millions of users
Wanelo? Whaaa?
■ Social network for shoppingIf you buy things on the Internet, you are our demographic
■ Many millions of users
■ Databases with billions of records
More importantly
More importantly
■ Iterated from one large Rails codebase to services
More importantly
■ Iterated from one large Rails codebase to services
■ Open source as much as we can
More importantly
■ Iterated from one large Rails codebase to services
■ Open source as much as we can
■ Scaled Ruby where people say Ruby can’t be scaled
More importantly
■ Iterated from one large Rails codebase to services
■ Open source as much as we can
■ Scaled Ruby where people say Ruby can’t be scaled
■ Very small team
But are you really here for success?
“Good judgment comes from experience. Experience comes from bad judgment.” — Proverb"
— Theo Schlossnagle"
http://www.infoq.com/presentations/Scalable-Internet-Architectures
“The Anti-Pattern Goldmine”
■ That I may or may not have worked for"
■ One giant, tangled Rails 2 codebase our vendored code had vendored code!
■ 30 engineers racing to commit code as fast as possible"
■ Success == large project launched on specific date
A completely hypothetical company
Releases a mess
■ Code branches for months at a time"
■ Monthly releases turn to weekly releases"
■ Change list distributed and explained to executives"
■ Multi-hour deploys"
■ Annual code freeze means 3 month code bonanza
Over time, things get worse…
Over time, things get worse…
■ Rough estimates == Firm deadlines
Over time, things get worse…
■ Rough estimates == Firm deadlines
■ Shortcuts only way to meet deadlines
Over time, things get worse…
■ Rough estimates == Firm deadlines
■ Shortcuts only way to meet deadlines
■ As soon as projects finished, teams split to meet new deadlines
Over time, things get worse…
■ Rough estimates == Firm deadlines
■ Shortcuts only way to meet deadlines
■ As soon as projects finished, teams split to meet new deadlines
■ Estimates based on worst-case timelines… never worst-case enough
Technical DebtTechnical Debt
Technical DebtTechnical Debt
What did I think I learned?
■ When programming is not fun, THERE IS A PROBLEM"
■ SOA IS SOLUTION TO ALL OUR PROBLEMS"
■ DEVOPS IS THE ANSWER
Now for a role change
■ I move into the operations team renamed to “not the operations team”BECAUSE DEVOPS"
■ Working on systems automation"
■ Engineering: SOA is the only way forward"
■ Product: SOA is that thing getting in the way of cranking out features
If at first you can’t succeed
If at first you can’t succeed
■ Take a firm stance
If at first you can’t succeed
■ Take a firm stance
■ Take some deep breaths
If at first you can’t succeed
■ Take a firm stance
■ Take some deep breaths
■ Cry like a baby
New features would be services, or not built
■ New login feature as separate application"
■ Data service to manage Enterprise software home-grown Rails app
“This will be hard and we’ll screw up, but we have no other option”
Okay, so what is SOA again?
One site composed of multiple applications
Internal External
UIAPI only
HTTP
Synchronous
TCP
Asynchronous
And why would I want this?
■ Scale organization"
■ Each team manages their own code"
■ Each team deploys their own code"
■ Outsource some code without giving keys to the kingdom
And why would I want this?
■ Scale codebase"
■ Performance tightly coupled to workload"
■ Hide complexity behind clean interfaces"
■ Use tools best suited to task at hand
And why would I want this?
■ Scale tests"
■ Small codebase == small test suite"
■ but… costly learning curve to test service boundaries
At some point, the cost of not doing services outweighs the cost of doing
services
Ok… back to our protagonists
■ Data service 9 months "
■ Complex state transitions between applications"
■ No major data problems"
■ Dependent applications ambiguously successful
EngineeringEngineering
We didn’t break anything!We didn’t break anything!
ProductProduct
9 months??????9 months??????
Ok, ok, ok, but what about?…
■ 3 months 2-4 engineers + the “DevOps” team"
■ Staging servers available in 2-3 weeks 2 months later… no one had deployed to them!
■ Released with feature flagExtra few weeks to break it in production
But then…
■ Worked exactly as intended"
■ Very resilient to failures!
■ User metrics generally successful
EngineeringEngineering
Success!!!1!!Success!!!1!!
ProductProduct
3 months??????????3 months??????????
How many engineers does it take to Service Oriented Architecture?
What is success?
■ Engineering: Success"
■ Product: Failure!
■ Who wins?
Nobody
But let’s go deeper…
■ Why did we need SOA?"
■ Engineering did not trust product"
■ Engineering did not trust that we could fix the shortcuts required to do our job
And even deeper…
■ Product did not trust Engineering to deliver on promises"
■ Product accountable for features, not for quality
Trust
So what did we learn?
So what did we learn?
Fuck it, SOA FROM THE BEGINNING next time
So what did we learn?
Fuck it, SOA FROM THE BEGINNING next timeX
Agile is more than just sprinting
■ Iterate, using data to guide next iteration"
■ Deploy soon, to start collecting data"
■ Red — Green — Refactor"
■ SOA will not solve organizational problems
SOA does not “fix” your code quality
SOA is a tool to help with our technological pain points
Iteration is how we create and deploy changes
Iteration
■ Small changes deployed quickly"
■ Feature flags"
■ Each step prioritized and completed when necessary"
■ But when SOA?
Performance
Code complexity will be less than not doing SOA
New code completely unrelated to old code
With the caveat…
■ May go into old codebase short-term"
■ Deploy today is better than deploy next week"
■ Requires TRUST that this will be fixed later
So… Performance
■ Site getting slower"
■ DBs becoming I/O bound on disk"
■ User growth/activity is crazy, and increasing
■ One table completely outstripping others"
■ Reads from table killing caches for others"
■ We’re in the cloud, so max resource limits
■ Site will stop working, soon"
■ Don’t want to stop feature development"
■ 10 engineers
Step 1: Isolate the data
Remove ActiveRecord relations
class Product < ActiveRecord::Base has_many :saves end
Remove ActiveRecord relations
class Product < ActiveRecord::Base def saves Save.where(product_id: self.id) end end
Pretend it’s in a different database
class Save < ActiveRecord::Base establish_connection "saves_#{Rails.env}" end
production: database: production_db host: 10.0.0.100 saves_production: database: production_db host: 10.0.0.100
■ Each change deployable"
■ Tests + staging servers will show where things break"
■ Watch out for DB connection count!
Switch to using different database
■ Create read-only replica"
■ Site maintenance mode"
■ Push new database.yml pointing to replica"
■ Promote replica to be master"
■ Restart unicorns"
■ Disable maintenance
■ Performance may now be fine"
■ Next step can come later, whenever necessary or convenient
Step 2: Isolate the interface
■ How many ways do you access table?"
■ What is long term growth?"
■ What are long term goals?
How would you shard if you could shard?
■ Wanelo == Saves"
■ Saves are viewed by"
■ Product"
■ User
Refactor to create an API
class Product < ActiveRecord::Base def saves Save.where(product_id: self.id) end end
Refactor to create an API
class Product < ActiveRecord::Base def saves Save.by_product(self.id) end end
Refactor to reduce the API
■ Remove redundancy"
■ Add tests"
■ You will break things very soon!
Step 3: Extract DB adapter
Move finders into a module or base class
class Save include SavesClient end
module SavesClient def self.included(other) other.send(:attr_accessor, :id, :product_id) other.extend ClientClassMethods end !
module ClientClassMethods def by_product(*args) adapter.new(self).by_product(*args) end !
def adapter @adapter ||= SavesClient::DbAdapter end end end
■ Finders call an adapter"
■ Adapter wraps database calls
Why an adapter?
■ Adapter is your feature flag"
■ Deployable as in-place refactor"
■ Can be replaced later with different adapters
module SavesClient class DbAdapter def self.close_connections # Check in the database connection, # since we're shutting down this thread SavesService::Save.clear_active_connections! end !
def by_product(*args) relation :by_product, *args end !
def relation(method, *args) SavesClient::AdapterRelation.new(self, SavesService::Save.send(method, *args)) end end end
■ Because THREADS"
■ State changes should always return a new instance"
■ #all, #first, #etc are called on Relation instance"
■ Relation will delegate to adapter
def relation(method, *args) SavesClient::AdapterRelation.new(self, SavesService::Save.send(method, *args)) end
Adapter methods return a Relation
■ ActiveRecord model, which moves into the client"
■ Insert your favorite DB adapter here"
■ Important part is that this is hidden from the including class
def relation(method, *args) SavesClient::AdapterRelation.new(self, SavesService::Save.send(method, *args)) end
Scope of query is called on…
Save.by_product
DBAdapter
AdapterRelation.new
relation.all
adapter.all
ActiveRecord::Save Postgres
Instantiate client instance(s)
Critical to deploy at this step
■ 100% guaranteed to make mistakes"
■ 100% guaranteed to have missed something"
■ Cost of fixing this is low at this step
Step 4: Launch service app
Insert your framework of choice
■ HTTP? JSON? MessagePack?"
■ What is easiest to develop?"
■ Transport mechanism maps to an adapter in the client"
■ Can be replaced later
brief interlude
Q: Why should you have deployed by now?
A: Because the client is much more complex than the server
So………..
Your bugs are in the client
Server code will be dictated by the client
■ By service launch time, whiteboard diagram will have changed"
■ Understanding will have changed"
■ Don’t write your server before you understand the needs of the client
So, back to the server
■ We used Sinatra + Oj"
■ Sinatra is awesome!
Step 5: Use the service
Adapter is your feature flag
switch between DB and service"
=="
switch between DB and HTTP adapters
Save.by_product
HTTPAdapter
AdapterRelation.new
relation.all
adapter.all
HTTPClient JSON to Hash
Instantiate client instance(s)
Retrospective
■ Isolate data"
■ Isolate interface"
■ Extract DB adapter into gem"
■ Launch service"
■ Switch to service adapter
Tests?
Integration vs Unit
■ Spin up server, a la Sunspot?"
■ “Fake” adapter"
■ In-memory store (Array)"
■ Need unit + integration"
■ Can delete redundant tests later
Development
■ Need to actually run service"
■ Failures are in interactions"
■ Foreman + Subcontractor"
■ Each app needs isolated Bundler env"
■ Don’t mix dependencies
But what about a new app?
How can we iterate on something that doesn’t exist?
■ Deploy as early as possible"
■ Chef/Puppet/Salt/Capistrano/CFEngine"
■ Focus on being able to change quickly"
■ Isolate and understand differences
Feature flags are more than on/off
■ Off"
■ Asynchronously write"
■ On
Integrate deep before wide
■ Code path from beginning to end"
■ Edge cases/Errors will bubble up"
■ Avoid complexity or cleverness in design"
■ It will come on its own
Takeaways
■ Barriers to SOA are barriers to all development"
■ Figure out how to iterate"
■ Hexagonal architecture can bubble up from organization’s needs"
■ Priorities/needs change. Facilitate this.
Thanks!
sax ecdysone sax
especially to everyone involved in this work, of which I was one small part
top related