The Glass Class: Mirror API

July 7th – 11th 2014

Mark Billinghurst, Gun Lee

HIT Lab NZ

University of Canterbury

An Introduction to

Glassware Development

Mirror API using Java Servlet

Gun Lee

* Images in the slides are from variety of sources,

including http://developer.android.com and http://developers.google.com/glass

Glassware Development

Mirror API

Server programming, online/web application

Static cards / timeline management

GDK

Android programming, Java (+ C/C++)

Live cards & Immersions

https://developers.google.com/glass/

Mirror API

REST API

Java servlet, PHP, Go,

Python, Ruby, .NET

Timeline based apps

Static cards

- Text, HTML, media attachment (image & video)

- Standard and custom menu items

Manage timeline

- Subscribe to timeline notifications

- Sharing with contacts

- Location based services

https://developers.google.com/glass/develop/mirror/index

Mirror API based Web App

3. Insert a static card

User sees the card

Glassware Web app

https://developers.google.com/glass/develop/mirror/stories

Develop with Mirror API

Prepare a web server

https callback required for OAuth 2.0

Minimum storage for credentials

Create a Google APIs Console project

Enable Mirror API and get Client ID & secret

Create a web application

Java servlet, PHP, Go, Python, Ruby, .NET

Implement OAuth 2.0 authorization

Use Mirror API to make REST API calls

- Wrapper classes/methods provided

Google App Engine

Free web server space!

You can skip this step if you have another server

Upgrade to paid service based on storage and traffic

needed

https://developers.google.com/appengine/

Python, Java, PHP, Go

Cloud Storage

Easy integration with Google APIs

Google APIs Console

http://console.developers.google.com

Login with your own Google account

Create a New Project

The Project ID becomes your URL

- http://your-project-id.appspot.com

Google APIs Console

In the project’s APIs & auth section,

enable Google Mirror API

Google APIs Console

APIs & auth > Credentials > OAuth: Create

New Client ID

Choose "Web Application"

Type in your website and OAuth callback

address https://your-app-id.appspot.com

https://your-app-id.appspot.com/oauth2callback

Google APIs Console

Note both Client ID and Client secret

Google APIs Console

APIs & auth > Consent screen

Fill in the Product Name and Homepage URL

This information is shown when users

authorizing your web app

Live Demo

- Setup Google App Engine

- Setup Google APIs Console

Dev. Env. Setup

ADT Bundle (Eclipse + Andriod SDK)

http://developer.android.com/sdk/index.html

GAE plugin for Eclipse

http://dl.google.com/eclipse/plugin/4.2

- Google App Engine Tools for Android

- Google Plugin for Eclipse

- SDK > Google App Engine Java SDK

Sign in on Eclipse GAE plugin

Use the Google account that will host your

web app on GAE

Create a Web App for GAE

File > New > (Other > Google > )

Web Application Project

Fill in the Project name and Java package

namespace to use

Check on option Use Google App Engine

Uncheck option Use Google Web Toolkit

Select option Use App Id then click Browse

button to choose your app id

Check on Generate project sample code

Click Finish

Anatomy of a Web App Project

Web App Project

src (source folder)

- Java classes organized in namespaces

war (web application archive)

- WEB-INF (settings and JAR libraries)

• appengine-web.xml

• web.xml

• lib

- index.html

- favicon.ico

Deploying to GAE

Right click on the Project

Google > Deploy to App Engine

Uncheck option Launch app in browser ...

Click Deploy

Open your GAE site on a browser

http://your-app-id.appspot.com

Live Demo

- Creating a new Web App project

- Anatomy of a Web App project

- Deploy to GAE

Adding Mirror API library

Right click on the Project

Google > Add Google APIs

Choose Google Mirror API

Click Finish

Authorization with OAuth 2.0 Authorizing your web app to access timeline on user’s

Google account

Programming Google Glass - The Mirror API by Eric Redmond

http://pragprog.com/book/erpgg/programming-google-glass

Authorization with OAuth 2.0

Enable HTTP sessions on GAE

Force https access only

Filter out inappropriate access

Implement OAuth 2.0 callback servlets

Code from Quick Start sample project

- https://developers.google.com/glass/develop/mirror

/quickstart/index

Add a new servlet to logout

https://developers.google.com/glass/develop/mirror/authorization

Enable sessions on GAE

war/WEB-INF/appengine-web.xml

<appengine-web-app ...>

...

<sessions-enabled>true</sessions-enabled>

</appengine-web-app>

Force https access only

war/WEB-INF/web.xml

<web-app ...>

...

<!-- force https only -->

<security-constraint>

<web-resource-collection>

<web-resource-name>Protected Area</web-resource-name>

<url-pattern>/*</url-pattern>

</web-resource-collection>

<user-data-constraint>

<transport-guarantee>CONFIDENTIAL</transport-guarantee>

</user-data-constraint>

</security-constraint>

...

</web-app>

Define Auth Filters war/WEB-INF/web.xml

<web-app ...>

...

<!-- filters -->

<filter>

<filter-name>authFilter</filter-name>

<filter-class>com.google.glassware.AuthFilter</filter-class>

</filter>

<filter-mapping>

<filter-name>authFilter</filter-name>

<url-pattern>*</url-pattern>

</filter-mapping>

<filter>

<filter-name>reauthFilter</filter-name>

<filter-class>com.google.glassware.ReauthFilter</filter-class>

</filter>

<filter-mapping>

<filter-name>reauthFilter</filter-name>

<url-pattern>*</url-pattern>

</filter-mapping>

</web-app>

Define Auth Servlet

war/WEB-INF/web.xml <webapp ...>

…

<servlet>

<servlet-name>oauth2callback</servlet-name>

<servlet-class>com.google.glassware.AuthServlet</servlet-class>

</servlet>

<servlet-mapping>

<servlet-name>oauth2callback</servlet-name>

<url-pattern>/oauth2callback</url-pattern>

</servlet-mapping>

...

</webapp>

Add Java classes for Auth

Create com.google.glassware package in

the src folder

Right click on src folder > New > Package

Copy Java classes for Auth

Drag and drop the following .java files into the

created package

AuthFilter.java, AuthServlet.java,

AuthSettings.java, AuthUtil.java,

ReauthFilter.java, WebUtil.java

Add Java classes for Auth

In AuthSettings.java

Fill in your CLIENT_ID and CLIENT_SECRET

strings you’ve noted from Google APIs

Console’s credential section

public static String CLIENT_ID =

"567615950876-14k9b9sggrtpulhu9s72le4vsejjscak.apps.googleusercontent.com";

public static String CLIENT_SECRET = "lo9hajhpQFneXP5K8YR0gEVK";

Add a New Servlet for Logout

Add LogoutServlet class

Right click on your app package in src folder

New > Class

Name: LogoutServlet

Superclass: javax.servlet.http.HttpServlet

Add a New Servlet for Logout

Implement LogoutServlet class

Open LogoutServlet.java

Source > Override/Implement Methods ...

Check on doGet(…) the click OK

Call AuthUtil.clearUserId(req) in the method

// log out

AuthUtil.clearUserId(req);

// print out results on the web browser

resp.setContentType("text/html; charset=utf-8");

resp.getWriter().println(

"<html><head><meta http-equiv=\"refresh\" content=\"3;url=/index.html\"></head> " +

"<body>Good bye!<br></body></html>" );

Add a New Servlet for Logout

Add servelt tag to war/WEB-INF/web.xml <webapp ...>

…

<servlet>

<servlet-name>Logout</servlet-name>

<servlet-class>org.hitlabnz.hw.LogoutServlet</servlet-class>

</servlet>

<servlet-mapping>

<servlet-name>Logout</servlet-name>

<url-pattern>/logout</url-pattern>

</servlet-mapping>

...

</webapp>

Add a New Servlet for Logout

Add a link to the LogoutServlet in

index.html

<a href="logout">Logout</a>

Live Demo

- Enable HTTP sessions on GAE

- Filter out inappropriate access

- Implement OAuth 2.0 callback

- Add logout servlet

Get Access to Mirror API

In HelloWorldServlet class

Add method getMirror(req)

private Mirror getMirror(HttpServletRequest req) throws IOException {

// get credential

Credential credential = AuthUtil.getCredential(req);

// build access to Mirror API

return new Mirror.Builder( new UrlFetchTransport(),

new JacksonFactory(), credential)

.setApplicationName("Hello Glass!")

.build();

}

Insert a Card to Timeline

In HelloWorldServlet class

Add code in doGet(...) method // get access to Mirror API

Mirror mirror = getMirror(req);

// get access to the timeline

Timeline timeline = mirror.timeline();

// create a timeline item (card)

TimelineItem timelineItem = new TimelineItem()

.setText( "Hello World!" )

.setDisplayTime(new DateTime(new Date()))

.setNotification(new NotificationConfig().setLevel("DEFAULT"));

// insert the card into the timeline

timeline.insert(timelineItem).execute();

Insert a Card to Timeline

In HelloWorldServlet class

Modify code in doGet(...) method for richer

response on the web browser

// print out results on the web browser

resp.setContentType("text/html; charset=utf-8");

resp.getWriter().println(

"<html><head>“ +

"<meta http-equiv=\"refresh\"content=\"3;url=/index.html\">" +

"</head> " +

"<body>A card is inserted to your timeline.<br></body></html>" );

Live Demo

- Access to Mirror API

- Insert a Card into Timeline

HTML in a Card

Follows CSS for the Glass UI https://mirror-api-playground.appspot.com/assets/css/base_style.css

String html = "<article><section><p class=\"text-auto-size\">" +

"<em class=\"yellow\">Hello World!</em><br>" +

"Welcome to the <strong class=\"blue\">Glass Class</strong> at HIT Lab NZ." +

"</p></section></article>";

timelineItem.setHtml(html);

Google Mirror API Playground

https://developers.google.com/glass/tools-

downloads/playground

To try it with your Glass

Go to Google API console

- https://console.developers.google.com

- Login and open your project

Create a new Client ID with the following URL

- https://mirror-api-playground.appspot.com

Use the generated Client ID to authorize

access on the Playground site

Multiple Pages in a Card

Auto paginate

Manually paginate

Multiple articles

<article class="auto-paginate">

Very long article ...

</article>

<article class="cover-only">

<section>

<p class="text-auto-size">This is the cover

page.</p>

</section>

</article>

<article>

<section>

This is second page.

</section>

</article>

<article>

<section>

Third page.

</section></article>

https://developers.google.com/glass/develop/mirror/timeline

Grouping Multiple Cards

Bundle – a set of cards grouped under a

cover card (or the latest card)

timelineItem.setBundleId( “UniqueBundleID” )

timelineItem.setIsBundleCover( true );

https://developers.google.com/glass/develop/mirror/timeline

Built-in Actions with Menu READ_ALOUD, DELETE, OPEN_URI, PLAY_VIDEO,

TOGGLE_PINNED, VOICE_CALL, NAVIGATE, REPLY,

SHARE

List<MenuItem> menuItemList = new ArrayList<MenuItem>();

menuItemList.add(new MenuItem().setAction("READ_ALOUD"));

timelineItem.setSpeakableText("Hello World!");

menuItemList.add(new MenuItem().setAction("TOGGLE_PINNED"));

menuItemList.add(new MenuItem().setAction("DELETE"));

timelineItem.setMenuItems(menuItemList);

https://developers.google.com/glass/develop/mirror/static-cards#creating_menu_items

Live Demo

- Cards with HTML

- Add Menu Items

Say hello to me

Add web form in index.html

Override doPost(...) method and get

parameter from the request

<form action="helloworld" method="post">

Hello <input type="text" name="custom_name" value="World"> !

<input type="submit" value="Submit">

</form>

String custom_name = req.getParameter("custom_name");

String message = "Hello " + custom_name = "!";

Image attachment

Adding static resources to the server

Edit war/WEB-INF/web.xml

Add a folder named static under the war

folder and add image files in it

<web-app ...>

...

<static-files>

<include path="/static/*" />

</static-files>

...

</web-app>

Image attachment

Attached image will be used as the

background of the card

In low-level HTTP Multipart attachment

// get an image to use with the card

URL url = new URL("http://hello-world-mirror.appspot.com/static/hitlabnz.jpg");

InputStreamContent attachment =

new InputStreamContent("image/jpeg", url.openStream());

// insert the card into the timeline

timeline.insert(timelineItem, attachment).execute();

https://developers.google.com/glass/develop/mirror/static-cards#inserting_a_timeline_item_with_media

Live Demo

- Input from web forms

- Image attachment

Managing Timeline Items

List

Delete

https://developers.google.com/glass/v1/reference/timeline/list

// request for list of the timeline items

Mirror.Timeline.List listRequest = timeline.list();

listRequest.setMaxResults(5L);

TimelineListResponse listResponse = listRequest.execute();

// retrieve results

List<TimelineItem> timelineItemList = listResponse.getItems();

mirror.timeline().delete(deleteItemId).execute();

Update Pinned Timeline Item

Simulating Live Cards with Mirror API

Ask the user to pin the item

Update the item regularly TimelineItem timelineItem = getSavedTimelineItem();

// If a saved item exists, isn't deleted, and is pinned

if (timelineItem != null

&& !(timelineItem.getIsDeleted() != null && timelineItem.getIsDeleted()) &&

(timelineItem.getIsPinned() != null && timelineItem.getIsPinned()))

{

// update the pinned item

timelineItem.setHtml( new_html );

timeline.update( timelineItem.getId(), timelineItem ).execute();

} else {

// create a new one and save the id for latter use

}

https://developers.google.com/glass/v1/reference/timeline/update#examples

Live Demo

- List Timeline Items

- Delete Timeline Items

Location

Get latest known location

Draw maps in your card

Location location = mirror.locations().get("latest").execute();

double latitude = location.getLatitude();

double longitude = location.getLongitude();

https://developers.google.com/glass/develop/mirror/location

<article>

<figure>

<img src="glass://map?w=240&h=360&marker=0;42.369590,-71.107132"

height="360" width="240">

</figure>

<section>

<div class="text-auto-size">

<p>Map shown on the left</p>

</div>

</section>

</article>

Navigate Menu Item

Set Location to the timeline item and add the

NAVIGATE menu item ...

timelineItem.setLocation(

new Location()

.setLatitude( -43.522087 )

.setLongitude( 172.582823 )

.setAddress("University of Canterbury, Ilam, Christchurch")

.setDisplayName("HIT Lab NZ") );

// add menu items with built-in actions

List<MenuItem> menuItemList = new ArrayList<MenuItem>();

menuItemList.add(new MenuItem().setAction("NAVIGATE"));

timelineItem.setMenuItems(menuItemList);

// insert the card to the timeline

timeline.insert(timelineItem).execute();

Live Demo

- Locate My Glass

- Add a Navigation Card

Subscriptions

Mirror API (server) calls back your web app with

notification POST

Your web app can subscribe to

Location updates

Timeline updates

- Share, reply, delete, custom menu item, launch with voice

Your web app must respond with a 200 OK

HTTP status code if no error occurred.

If not, Mirror API might resend notifications.

https://developers.google.com/glass/develop/mirror/static-cards#subscriptions

Subscribe

Add subscription to the mirror api

Better check if already subscribing

Subscription subscription = new Subscription();

subscription.setCollection("locations"); // or "timeline"

subscription.setCallbackUrl("https://your_web_app/notification");

subscription.setUserToken( AuthUtil.getUserId(req) );

mirror.subscriptions().insert(subscription).execute();

Unsubscribe

Add subscription to the mirror api // find subscription to delete

SubscriptionsListResponse subslist_resp =

mirror.subscriptions().list().execute();

List<Subscription> subsclist = subslist_resp.getItems();

String subscription_id_to_delete = null;

for (Subscription subsc : subsclist) {

if (subsc.getId().equals("locations")) { // or “timeline"

subscription_id_to_delete = subsc.getId();

break;

}

}

// delete the subscription

mirror.subscriptions().delete(subscription_id_to_delete).execute();

Handle Notification Callback

Create a callback Servlet that

handles notification POST

responds with a 200 OK HTTP status code

Make sure your callback Servlet URL is

not filtered in AuthFilter.java

Use the user ID provided by

notification.getUserToken() to call Mirror

APIs subsequently

Handle Notification Callback protected void doPost(HttpServletRequest req, HttpServletResponse resp) {

// Respond with OK and status 200 to prevent redelivery

resp.setContentType("text/html");

Writer writer = resp.getWriter();

writer.append("OK");

writer.close();

// extract notification object

JsonFactory jsonFactory = new JacksonFactory();

Notification notification =

jsonFactory.fromInputStream(req.getInputStream(), Notification.class);

// Figure out the impacted user and get their credentials for API calls

String userId = notification.getUserToken();

Credential credential = AuthUtil.getCredential(userId);

Mirror mirror = new Mirror.Builder(new UrlFetchTransport(),

new JacksonFactory(), credential)

.setApplicationName("Hello World")

.build();

if (notification.getCollection().equals("locations")) { // or “timeline“

...

Handle Notification Callback

Handling location updates

Handling timeline updates

User action type

- REPLY, DELETE, CUSTOM, LAUNCH, SHARE

// get notified location

Location location = mirror.locations().get(notification.getItemId()).execute();

location.getLatitude();

location.getLongitude();

...

// Get the timeline item which triggered notification

TimelineItem notifiedItem = mirror.timeline().get(notification.getItemId()).execute();

if(notification.getUserActions().contains(new UserAction().setType("REPLY")))

{

String message = "Hello " + notifiedItem.getText() + "!";

...

https://developers.google.com/glass/develop/mirror/subscriptions

Live Demo

- Subscribe to Location Updates

- Subscribe to Timeline Updates:

REPLY

Contact

LAUNCH and SHARE user actions need

contact of your web app

Contact helloWorldContact = new Contact();

helloWorldContact.setId("org.hitlabnz.helloworld");

helloWorldContact.setDisplayName("Hello World");

ArrayList<String> imageUrlList = new ArrayList<String>();

imageUrlList.add("http://hello-world-mirror.appspot.com/static/hitlabnz.jpg");

helloWorldContact.setImageUrls(imageUrlList);

ArrayList<Command> commandList = new ArrayList<Command>();

commandList.add(new Command().setType("POST_AN_UPDATE")); // TAKE_A_NOTE

helloWorldContact.setAcceptCommands(commandList);

https://developers.google.com/glass/develop/mirror/contacts

https://developers.google.com/glass/develop/mirror/subscriptions

Live Demo

- Subscribe to LAUNCH

DIY

- CUSTOM menu item

- Subscribe to SHARE

https://developers.google.com/glass/develop/mirror/contacts

https://developers.google.com/glass/develop/mirror/subscriptions

More Tips

Templates

JSP

http://freemarker.org

Scheduled jobs on server

CRON settings on GAE

Programming Google Glass - The Mirror API

By Eric Redmond

http://pragprog.com/book/erpgg/programming-google-glass

Quick Start Samples https://developers.google.com/glass/develop/mirror/quickstart/in

dex

Sample Code Online

Mirror-QuickStartSample-Java-GAE

https://glass-mirror-java-sample.appspot.com

Mirror0-HelloGlass

https://hello-glass-mirror.appspot.com

Mirror1-HelloWorld

https://hello-world-mirror.appspot.com

More Information

Website

https://developers.google.com/glass

http://arforglass.org

http://www.hitlabnz.org

Gun Lee

gun.lee@hitlabnz.org

Mark Billinghurst

mark.billinghurst@hitlabnz.org