what's cooking in the cloud?
TRANSCRIPT
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
What’s Cooking in the Cloud? Expanding the Menu for App Developers
Kasper Løvborg Jensen
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Chief Wizard, Founder Leafcastle Labs Associate Professor Aarhus University School of Engineering Organizer Google Developer Group Aarhus
@kasperlj [email protected]
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
GCP NEXT ‘16 Conference
• Machine Learning for all! • Cloud Vision for all! • Cloud Speech API for all (soon)! • StackDriver (works with AWS)! • …lots more!
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Outline
• Why cloud? • Mobile backends – Experiences – Patterns
• New powerful Tools/APIs – Machine Learning – Speech API – Cloud Vision
• Demo: TA-800
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Why I am Excited about Cloud / GCP
• Backends for mobile/wearable/web apps – Low/No Maintenance – Scalability – Reliability – Security
• APIs / Services for applications – Integrate/synch with existing Services – Add powerful features/functionality through APIs
• Great for startups – Business models and focus on creating value – Cost scales with growth – Speed of iteration and experiments
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
App Backend Needs
• Storage • Computing • Network / serving • Cloud Messaging • Analytics • Tools • …
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Google Cloud Platform
Source:GCPNEXT2016keynote
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Mobile Backends?
Source:Google,Android,Apple,Chrome
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Mobile Backends “Building a backend service for a mobile app is similar to building a web-based service, with some additional requirements:
• Limit on-device data storage • Synchronize data across multiple devices • Handle the offline case gracefully • Send notifications and messages • Minimize battery drain”
hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
App Engine w. EndPoints
Modifiedfrom:hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Pure Firebase
hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Firebase + App Engine
hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Firebase + MVMs | Containers
hAps://cloud.google.com/soluEons/mobile/mobile-app-backend-services
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Machine Learning
• Machine Learning • Vision API • Speech API • Translate API
Source:Google
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Speech API
• Solves really hard problems! – Natural language recognition – Open vocabulary – Noisy environments – 80+ Different languages
hAps://cloud.google.com/speech/
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Speech Horror Story ca. 2005 • Distributed speech recognition in
app as part of my PhD – On PDA running Windows Mobile
with feature extraction through custom DLL (c/c++)
– Live stream features to Sphinx IV DSR on Tomcat server that could only handle one connection/request at a time due to memory requirements
– Months of building/training speech, grammar and acoustic models
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Live or Batch?
• Batch – Upload audio sample(s) and get
results when recognition complete
• Live – Stream audio to server and
receive partial results as they become available
hAps://cloud.google.com/speech/
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Cloud Vision API • Solves really hard problems! – Extract and label objects in arbitrary image
without context – Can be anything – Find details like facial features, tilt and
“emotions“ – Lighting conditions – Blurry pictures – Vast amount of cameras and settings – …
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Case Study
• Problem: Students falling asleep during lectures
• Solution: Need person some sort of contraption to detect students falling asleep and then wake them up
“Never send a man to do a machine’s job”
-Gandalf
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
TA-800
• IF we wanted to build a Terminator as a teaching assistant – Running Android – Powered by Google Cloud / Skynet
• Detect and “terminate” inactive students during lectures – High precision Nerf gatling gun
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
TA-800 Hardware
• Nexus 6 or 6p • ATLAS prototype • Nerf minigun
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
TA-800 Behavior Protocol
While(true){ – Detect situation
• Identify features and classify place/situation
– Acquire potential targets • Detect if sleeping
– TERMINATE Wake up targets – Report to master
}
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
TA-800 Hack/Demo • Image processing based on Cloud Vision API
example app https://github.com/GoogleCloudPlatform/
cloud-vision/tree/master/android/CloudVision • Firebase integration based on Firebase chat
example https://github.com/firebase/AndroidChat
• Total dev time:~6 hours (50%+ time on UI)
• Will be on GitHub:
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Cloud Vision API
hAps://cloud.google.com/vision/
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Using Cloud Vision API
• Specify request (JSON) – Image
• Content/source – Features
• Type • Max results
• Handle result (JSON) • Android: use CloudVision API – com.google.apis:google-api-services-vision:v1-rev2-1.21.0 !
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Cloud Vision Detection Types DETECTION_TYPES = [ ! 'TYPE_UNSPECIFIED', ! 'FACE_DETECTION', ! 'LANDMARK_DETECTION', ! 'LOGO_DETECTION', ! 'LABEL_DETECTION', ! 'TEXT_DETECTION', ! 'SAFE_SEARCH_DETECTION', !] !
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Rich Response Data as JSON • Labels
– Things/situations in the image • Faces
– Faces, location in picture – Happy, angry, sad, surprised – Face “landmarks”: eyes, ears, … – Tilt, …
• Logos – Known brands and logos
• Landmarks – “Eiffel Tower”, “AROS museum”, … – Also including e.g. location metadata
• Texts – OCR, finding text in the image
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
Your turn!
GCP: https://cloud.google.com/
List of (most) products: https://cloud.google.com/products/
Playground: https://cloud-playground.appspot.com/playground/
Launcher: https://cloud.google.com/launcher/
Firebase: https://www.firebase.com/
Kasper Løvborg Jensen Leafcastle Labs | Aarhus University
THANK YOU! Questions?
@kasperlj