cs 430p/530: internet, web, & cloud systems
TRANSCRIPT
APIs/abstractions
Previously
Abstractions for infrastructure to ease operations (Ops) Manage complexity in deploying software systems
Examples covered… VMs, containers
Infrastructure as a Service
Platform as a Service
Functions as a Service
Infrastructure as Code
Portland State University CS 430P/530 Internet, Web & Cloud Systems
But, what about the apps themselves?
Emerging software abstractions for building functionality
Example: Your homework (e.g. Flask) Common web application abstractions as a framework
No need to program sockets, HTTP, cookies, sessions, routing, HTML
generation, database communication, etc.
Imagine writing it in C (from scratch!)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Historically
From Unix to Windows in 90s/00s
Portland State University CS 430P/530 Internet, Web & Cloud Systems
To this, 20 years later
What happened?
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Currently this…on the front-end
Mobile apps (Android/iOS SDKs)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Web apps (client-side JS frameworks)
Or both Progressive web apps
Seamless off-line/on-line experience
Native/hybrid applications
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Meanwhile…on the backend
Server-side web and API frameworks
Data and event management
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Now this…
Software as a service via APIs
Comprehensive lists at https://rapidapi.com/
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Even traditional businesses are building them…
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Others…
Payments (PayPal, Venmo)
Social media (Twitter, Facebook, Instagram, Pinterest, Spotify)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Multi-service instant
messaging and
communications (Twilio,
Nexmo) Communicate with
customers using whatever
they have!
Portland State University CS 430P/530 Internet, Web & Cloud Systems
IoT (LiFX, GCP IOT Core, AWS IOT Core)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Food and nutrition APIs (Nutritionix)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Spoonacular
Portland State University CS 430P/530 Internet, Web & Cloud Systems
On GCP…
Maps API, Search API, Knowledge Graph API, Analytics, Ads, Voice
transcription, Image analysis, Video analysis, Translation, etc.
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Accessing APIs
#1: Python packages
(if they exist)
PyPI
Python Package Index Packages installable via pip (as we have been doing all quarter)
Most popular web APIs come with supported Python packages
Many are auto-generated via Swagger
Note Code examples are written for simplicity
NEVER specify API keys within source code
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Example: Giphy
pip install giphy_client
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import giphy_client
# Create app and API_KEY at https://developers.giphy.com# Note insecure pattern: Should supply from environment variablesAPI_KEY = os.environ.get('YOUR_GIPHY_API_KEY')
api_instance = giphy_client.DefaultApi()api_response = api_instance.gifs_search_get(API_KEY,
'clapback', limit=1, rating='g', lang='en', fmt='json')if len(api_response.data) > 0:
print(api_response.data[0].embed_url)
Example: Spotipy (Spotify)
pip install spotipy
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import spotipyfrom spotipy.oauth2 import SpotifyClientCredentials
# Obtain credentials via https://developer.spotify.com# Pass to client via environment variables that are not committed to gitCLIENT_ID=os.environ.get('SPOTIFY_CLIENT_ID')CLIENT_SECRET=os.environ.get('SPOTIFY_CLIENT_SECRET')
# Return a spotify playlist given querycredentials = SpotifyClientCredentials(client_id=CLIENT_ID,
client_secret=CLIENT_SECRET)spotify = spotipy.Spotify(client_credentials_manager=credentials)results = spotify.search('Foo Fighters', limit=1, type='playlist')if len(results['playlists']['items']) > 0:print(results['playlists']['items'][0]['external_urls']['spotify'])
#2: API discovery
OpenAPI specifications
OpenAPI initiative (Linux Foundation) Formerly known as Swagger (swagger.io)
"Standard, language-agnostic interface to REST APIs which allows
humans and computers to discover and understand the capabilities fo the
service without access to source code, documentation, or through
network traffic inspection"
Specify REST API and its semantics in standard way Code parses specification to automatically build calls that use it
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Google's API Discovery format
API discovery format Supports both REST APIs with OpenAPI specifications as well as gRPC's
generated API clients Recall gRPC (open-source RPC framework) supporting zero-copy, binary data
transfer with specifications written in Protocol Buffers language
More info at https://googleapis.github.io/
OpenAPI/Swagger tools here
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Example
In-platform access of storage API via apiclient.discovery Avoid use of key management for accesses done completely within GCP
pip install google-api-python-client google-auth
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import apiclient.discoveryimport google.auth
# Obtain credentials and instantiate APIcredentials, project_id = google.auth.default()gcs_api = apiclient.discovery.build('storage', 'v1', credentials=credentials)
# Obtain credentials and instantiate APIgcs_api.objects().insert(bucket='wuchang-bucket',
name='myfile.txt',media_body='./myfile.txt').execute()
YouTube API via apiclient.discovery Access from anywhere, but requires API key
Portland State University CS 430P/530 Internet, Web & Cloud Systems
# https://developers.google.com/youtube/v3/docs/search/listimport apiclient.discoveryAPI_KEY = os.environ.get("API_KEY")
# Build the request for YouTube search contentyoutube = apiclient.discovery.build("youtube",
"v3",developerKey=API_KEY)
search_response = youtube.search().list( q = "Grumpy Cat",part = "id, snippet",maxResults = 1, ).execute()
for search_result in search_response.get("items", []):if search_result["id"]["kind"] == "youtube#video":print("%s" % search_result["snippet"]["title"])print("%s" % search_result["id"]["videoId"])
Knowledge Graph API via apiclient.discovery
Translation API via apiclient.discovery
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import apiclient.discovery, os
API_KEY = os.environ.get('API_KEY')
kgsearch = apiclient.discovery.build('kgsearch', 'v1',developerKey=API_KEY)
request = kgsearch.entities().search(query='Portland State University',limit=1)
# Returns a dict of search result inforesult = request.execute()
import apiclient.discovery, os
translate = apiclient.discovery.build('translate', 'v2',developerKey=os.environ.get("API_KEY"))
request = translate.translations().list(source = 'en',target = 'fr',q=['flower','car'] )
result = request.execute()
#3: HTTP requests to REST API
Language support for HTTP
Most languages with native support for sending HTTP requests and
parsing HTTP responses Typically, GETs and POSTs with JSON being returned
Packages are wrappers for the underlying REST API call over HTTP
Python Requests (My) preferred package for HTTP
Most examples will use this
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Example: Google Places API (REST via Python)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import requests
def post(self):""" Sends search box information to Google Places API and returns response """api_key = os.environ.get('API_KEY')keyword = request.form['search']
# Base URL for Google Places APIurl = 'https://maps.googleapis.com/maps/api/place/nearbysearch/json?'
# Lat/Long of Portland, ORlocation = 'location=45.5122,-122.6587&radius=1500'
# Places type variableplace = 'restaurant'
# API responseresponse = requests.get(url + location + '&type=' + place +
'&keyword=' + keyword + '&key=' + api_key)r = response.json()return render_template('home.html', response=r['results'])
Example: Knowledge Graph API (REST via Python)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import requests
# Grab Knowledge Graph API Key from APIs & Services:Credentials and# set environment variableapi_key = os.environ.get('API_KEY')
term = 'Portland State University'
# Use term and api_key string to formulate a URL for hitting the# Knowledge Graph API endpoint. Note f-strings are Python 3.6url_string = f'https://kgsearch.googleapis.com/v1/entities:search?query={term}&key={api_key}&limit=1&indent=True'
# Take given response, parse it as a json object, and return a dictresponse = requests.get(url_string)
result_dict = response.json()
Example: Yelp API
Portland State University CS 430P/530 Internet, Web & Cloud Systems
import requestsdef yelpSearch(self, title):
# Craft Yelp's API endpoint URL, set location to Portland and limit resultsurl = 'https://api.yelp.com/v3/businesses/search?term=' + title +
'&location=portland&limit=3'
# Declare the api header with api key (Fill in your own key)headers={'Authorization': "Bearer YOUR_API_KEY"}
# Query the API endpoint, parse JSON response and return dictresponse = requests.get(url, headers=headers)return response.json()
Side-by-side (Google Maps)
pip install googlemaps
pip install requests
Portland State University CS 430P/530 Internet, Web & Cloud Systems
# Requires enabling geocoding API and generation of an API key in Google Cloudimport googlemapsimport os
API_KEY = os.environ.get('API_KEY')gmaps = googlemaps.Client(key=API_KEY)boba_query = 'Mo Cha Tea House, Portland, OR'boba_shops = gmaps.geocode(boba_query)for shop in boba_shops:print(shop['formatted_address'])
import requestsimport jsonimport os
API_KEY = os.environ.get('API_KEY')boba_query = 'Mo Cha Tea House, Portland, OR'resp = requests.get( f'''https://maps.googleapis.com/maps/api/geocode/json?address={boba_query}&key={API_KEY}''')boba_shops = json.loads(resp.text)for shop in boba_shops['results']:
print(shop['formatted_address'])
Side-by-side (Nutritionix)
pip install nutritionix nutritionix-client
pip install requests
Portland State University CS 430P/530 Internet, Web & Cloud Systems
from nutritionix import NutritionixClientrecipe = 'Pasta Salad'
# Get the credentials from environment variables (more secure than in code)APP_ID=os.environ.get('NIX_APP_ID')API_KEY=os.environ.get('NIX_API_KEY')
nut_info = NutritionixClient(application_id=APP_ID, api_key=API_KEY)result = nut_info.search(q=recipe, limit=3, offset=0)
def nutritionix(self, ingredient):url = 'https://trackapi.nutritionix.com/v2/natural/nutrients'# Declare the api header with api key (Fill in your own App ID and Key)headers = {"Content-Type":"application/json", "x-app-id":"YOUR_APP_ID",
"x-app-key":"YOUR_APP_KEY"}
# Declare POST query as a dictionary, set query field to ingredientbody = {"query":ingredient,"timezone": "US/Eastern"}
# Make request to API endpoint, parse JSON response and return dictresponse = requests.post(url, headers = headers, json = body)return response.json()
AWS API Gateway
Building and deploying APIs
Types of Web APIs
RESTful APIs (request-response) Online transactions
Weather application
Blogs, social media
Via HTTP methods PUT, GET, POST, DELETE, etc.
Take in and output HTML, XML, or JSON
WebSocket APIs (bi-directional) Chat applications
Real-time alerts and notifications
Portland State University CS 430P/530 Internet, Web & Cloud Systems
API Gateway
Make it easy to scale and manage versions of an API Fully managed, serverless API platform
REST and WebSocket support
Lifecycle management support for APIs
Portland State University CS 430P/530 Internet, Web & Cloud Systems
API Gateway overview
https://www.alexdebrie.com/posts/api-gateway-elements/
Portland State University CS 430P/530 Internet, Web & Cloud Systems
1. Authorization
and access
control
Routing to
endpoint,
access control
Packaging
request to API
logic in
integration
Transform response, format
HTTP response headers
2. Request flow
3. Response flow
1. Authorization and access control
User access management to API IAM-based, resource-based policies
AWS Cognito
Custom (e.g. Authorizer built in a Lambda)
OAuth/SAML
User API key generation and management
Usage-plans for billing and throttling Per-API key metering
Usage per second or daily quota
Configuration of external site access via Cross-Origin Resource
Sharing (CORS) Restrict sites that can access API via cross-domain requests
TLS certificate management for endpoint
Portland State University CS 430P/530 Internet, Web & Cloud Systems
2. Request Flow
Endpoint configuration, request routing based on HTTP method
Method request Validation of request parameters
Validation of request bodies Done via pre-defined models (data structure for specifying request and response
payloads in a JSON Schema format)
Integration request Transformation to format method request into backend integration
request (via request mapping templates)
Integration invocation to perform API logic
Integration Implementation of API logic (EC2, ECS, Lambda, etc)
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Example model and mapping templates
Cost calculator request model that contains a price, size, and unit API Gateway ensures requests supply all parts of the model
Mapping templates Transform from one model to another
For requests, translate method request model into integration request
model for use by backend integration
Portland State University CS 430P/530 Internet, Web & Cloud Systems
3. Response Flow
Transform integration responses and return values back into HTTP
response Done similarly to request flow via response mapping templates and
method response models
Portland State University CS 430P/530 Internet, Web & Cloud Systems
API Gateway miscellany
Caching support (via CloudFront) Endpoint types
Edge (Latency optimized via deployment in CloudFront) Regional (For requests and responses residing in the same AWS region) Private (Exposed only to services and resources inside your VPC)
Stages Snapshots or versions of the API available for clients to call Create an API deployment and associate it with a stage (stage settings include
caching enabled, request throttling, logging config) SDK generation
Automatically generate language SDKs (Java, JavaScript, Objective-C, Swift, and Ruby)
CloudWatch monitoring built-in (metrics collected per minute) Count Integration latency CacheHitCount CacheMissCount HTTP 400 and 500 errors
Portland State University CS 430P/530 Internet, Web & Cloud Systems
Labs
Leverage APIs to build applications (Slack/Knowledge Graph)
Manually deploy APIs (Guestbook) A lot of work!
Next week IAC + API deployment
ML APIs
Backend as a Service APIs
Portland State University CS 430P/530 Internet, Web & Cloud Systems