automatic discovery of service metadata for systems at scale
TRANSCRIPT
![Page 1: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/1.jpg)
Automatic Discovery of Service Metadata
Martina Iglesias
September 2016
![Page 2: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/2.jpg)
Spotify scale
![Page 3: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/3.jpg)
Spotify Users
100M Active Users
40MPaid Subscribers 59 Countries
![Page 4: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/4.jpg)
Spotify tech
● +800 tech employees
● 120 teams
● Microservices architecture (scale and work
independently)
![Page 5: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/5.jpg)
![Page 6: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/6.jpg)
Artist page
Number of streams
![Page 7: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/7.jpg)
Artist page
Artist Discography
![Page 8: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/8.jpg)
Artist page
Artist Playlists
![Page 9: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/9.jpg)
Artist page
Merchandise
![Page 10: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/10.jpg)
Example: aggregating service
Artist
Discography Playcount Playlists Merch
![Page 11: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/11.jpg)
Spotify infrastructure
● +1000 services
● The number of services grows as we add new
features
![Page 12: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/12.jpg)
Spotify officesStockholm
BostonNew York
San Francisco
Gothenburg
![Page 13: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/13.jpg)
Previous situation
● Each team had doc. in different places
● README.mk
● Markdown files in doc/
● Wiki
● Link to a document somewhere else
![Page 14: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/14.jpg)
![Page 15: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/15.jpg)
System Z
![Page 16: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/16.jpg)
System Z
● Web application
● Internal tool
● Catalogue of all our systems and its parts
● Very well integrated with our apollo services
● Easy to discover and access
![Page 17: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/17.jpg)
![Page 18: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/18.jpg)
![Page 19: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/19.jpg)
artist
playcount
discography
merch
playlist
loadbalancer
![Page 20: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/20.jpg)
System Map
● Generated from runtime and declared
dependencies
● Uses graphviz
![Page 21: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/21.jpg)
DNS
System Z overview
Register artist serviceARTIST SERVICE:
DNS: _artist._http
OWNER: martina
SYSTEM Z(backend)
artist-1.spotify.com
Lookup machines
HTTP req to
/meta/0/
![Page 22: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/22.jpg)
Apollo and apollo-meta
![Page 23: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/23.jpg)
Apollo
● Java libraries for writing microservices
● Open Source
● https://github.com/spotify/apollo
● In production since 2014
![Page 24: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/24.jpg)
APOLLO SERVICE
MODULES
OKHTTP-CLIENTJETTY-HTTP-SERVER
APOLLO-CORE(manages lifecycle)APOLLO-API APOLLO-META
SERVICE LOGIC
interacts
uses
Apollo service overview
![Page 25: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/25.jpg)
Apollo-meta
● Metadata module
● Open Source
● https://github.com/spotify/apollo-meta
● Exposes endpoints with metadata of the service
● Runtime generated - source of truth
![Page 26: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/26.jpg)
Example: Creating a route
Route.async("GET", "/v1/artist/<id>",
request-> getArtist(request))
.withDocString("Get the artist page for a specific id.")
)
![Page 27: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/27.jpg)
Endpoints exposed by apollo meta
1. Instance information2. Configuration3. Endpoints4. Call Information
![Page 28: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/28.jpg)
1. Apollo-meta: instance info
● Collects information about the service: build
version and uptime
● Useful to get the full picture when making rolling
upgrades.
![Page 29: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/29.jpg)
1. Apollo-meta: instance info
curl http://artist-a1/_meta/0/info =>
{ "result": { "buildVersion": "artist 2.1-SNAPSHOT", "containerVersion": "apollo-standalone 1.1.0", "systemVersion": "java 1.8.0_60", "serviceUptime": 303577.347 }}
![Page 30: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/30.jpg)
2. Apollo-meta: configuration
● The current loaded config of the service, possibly
filtered
![Page 31: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/31.jpg)
2. Apollo-meta: config
curl http://artist-a1/_meta/0/config =>{ result: { http: { server: { port: 8080 } } }}
![Page 32: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/32.jpg)
3. Apollo-meta: endpoints
● Lists the endpoints of the service
![Page 33: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/33.jpg)
3. Apollo-meta: endpoints
curl http://artist-a1/_meta/0/endpoints =>{ methodName: "/v1/artist/<id>[GET]", uri: "/v1/artist/<id>" method: [ "GET" ], docstring: "Get the artist page for a specific id.", queryParameters: []
},...
![Page 34: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/34.jpg)
4. Apollo-meta: call info
● Lists services that make incoming requests
● Lists all other services we make requests to
![Page 35: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/35.jpg)
4. Apollo-meta: call info
curl http://artist-a1/_meta/0/calls =>
incoming: { loadbalancer: { endpoints: [ uri: "/v1/artist/<id>", method: ["GET"], queryParameters: [catalogue, locale] ], }, },..
![Page 36: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/36.jpg)
4. Apollo-meta: call info
... outgoing: { discography: [], playcount: [], playlist: [], merch: [] } }}
![Page 37: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/37.jpg)
Apollo meta <-> System Z
● System Z calls these endpoints
● Displays a merged version of all the data
![Page 38: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/38.jpg)
Conclusions
![Page 39: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/39.jpg)
● Quicker access to relevant information
● Know immediately where to go when solving an
incident
● Less interruptions
● Less boring work
Situation now
![Page 40: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/40.jpg)
● Think about growth and scaling
● Automate all boring tasks that you can
● Put all the information related in one, easy to
access, place
● All related links in one place
Learnings
![Page 41: Automatic Discovery of Service Metadata for Systems at Scale](https://reader035.vdocuments.net/reader035/viewer/2022081515/5880380f1a28abfd0a8b4869/html5/thumbnails/41.jpg)
● apollo
● swagger.io
● raml.org (jax-rs)
Documentation generators