Android 2012: Integrating Sensors Into Android Hardware Rian Sanderson

Sensor Platforms Inc.

PresenterPresentation NotesNotes: * Rian Sanderson * share some of what Ive learned over the past year working with Android

* presentation is not about using sensors in an android app * not how transformative sensors are

* You have a board running android, you have senors on some daughter card * Theoretically there is open source code to bridge the gap between them * This is the presentation for you

Next Step: * Thats where I was a year ago

integrating android sensor

hardware

the source code is out there but not much of the big picture

the problem

google this

PresenterPresentation NotesNotes: * it wasnt great three months ago, it was worse a year ago, is now getting better * adding in a board name: beagle, snoball helps * just not much in the way of guides out there for people developing their own hardware * One of the best links: tells you how to build it, tells you to read a header file, but thats it * Another is a great guide from TI, I have a link to it on a further reading section

Next Step: * like many things in this business * if know what to look for more a matter of cranking it out

this talk should enable you to

integrate new sensors into Android debug each of the layers in the sensor stack

download these slides and check out the speakers notes if youd like to read all my bullet points

PresenterPresentation NotesNotes: * I'll show you what to look for and where * Introduce you to some tools to make sure you're getting it right * Illustrate the low level sensor architecture in Android * Hands on demo of tools for debugging at each layer * Based on Gingerbread release, but not substantially different in Ice Cream Sandwich

Next Step: * before diving in, take a step back

creating sensors systems is an art

Software System Integration Physical Conventions Sensor Fusion Hardware System Engineering Processor Resources

todays talk

the software integration is a craft that can be picked up quickly

PresenterPresentation NotesNotes:Sensor system design is more akin to RF design than traditional digital design

Focusing on Software integration today * only talking about 1/3 the software stack! * have a good diagram about this on Big Picture slide

Physical conventions * This is the stuff that rockets blow up over * acceleration in m/s^2, magnetometer in uT, accelerometer sees gravity as a force pointing UP * all the sensors aligned correctly on the board

Sensor Fusion * serious math * kalman filters * signal processing

Hardware System Engineering * Kevin Shaw speaking in Sensors Theatre on Thursday * garbage in garbage out * sensor noise * PCB flex corrupting accelerometer * magnetometer next to battery

Processor Resources * break this out because its so important * after cost, cost, and cost * responsiveness and battery life are arguably the next most important attribute of a great product * CPU choice, RAM, storage drive costs * goal is to keep that processor in power save * every instruction executed counts, especially at 100Hz

Next Step: * I want to introduce you the platform for todays demos

demo: BeagleTab

you too can integrate all the sensors you want into Android

PresenterPresentation NotesNotes: * Something only an engineer could love * resistive touch display, buttons * beagle board,batteries * sensor board: ** (Accel + Mag + Gyro) x3 ** breakout board + GPS + pressure + temp ** i2c + SPI through EXT connector * Sensor Fusion is done with FreeMotion library

Next Step: * This is doable, and if you have android up and running already, can get sensors in there very quickly

big picture

insert your code here

PresenterPresentation NotesNotes:sensors at the bottom, phone/tablet applications at the top

* Jim Steele is talking about how to develop apps that talk to sensors * Karim Youngers new book talks about android internals * android specifies a sensors hardware abstraction * libsensor aka sensors.so and is specific to android systems * you must implement sensors.so * huge amount of linux infrastructure to utilize in completing that task

Next Step:Will bounce around this diagram, in the manner of how I approach debugging a new Android board

data flow

PresenterPresentation NotesNotes: * drivers are loaded at boot time * Android Sensor Service starts and reads what sensors are available * App starts, requests sensor data from SensorManager * Sensor Service asks libsensor to go start up a sensor if its not already started * libsensor flips the switch in the driver, sensor starts up, starts spewing data * libsensor reads that data, reformats it for android, puts it in a queue * sensor service comes along and gets that data at a regular rate * delivers to each app subscribed to sensor manager

Next Step: * lets see one of those apps

app level debugging

AndroSensor displays sensor info exactly as an app

developer will see it

PresenterPresentation NotesNotes: * demonstrates the power and joy of android * I didn't have to spend time writing this * I know this works and is tested * shows exactly the information an app sees when querying Android.SensorManager

Next Step:skipping over the middle of the big picture, diving into the bottom

sensor drivers

Tools for debug

adb shell lsmod | grep

dmesg

dont write sensors drivers keep them platform agnostic

PresenterPresentation NotesNotes: * first thing with a new board, I want to see if drivers are working * Linux drivers are available for most any sensor out there * dmesg: is it working? * lsmod | grep : is it loaded * adb shell with a pipe -> utilize text processing of your dev machine

Next Step: * Youre probably compiling lots of drivers, so glossing over this

linux infrastructure

leverage the most

appropriate infrastructure

PresenterPresentation NotesNotes: * in software we talk about toolboxes, I think of linux as a warehouse * hard to find the most appropriate tools * best to build on what other people have done

Next Step: * highlighted input event framework because its so prevalent

Input Event Framework presents each sensor as name and a stream of data events

PresenterPresentation NotesNotes: * Historically for connecting mice and keyboards to X server * helps unify disparate driver interfaces * find right drivers, can process data from a OneWire thermo like an SPI accel * standard methodology, standard handling, standard tools, easy to debug

Next Step: * it unifies driver interface through event based processing

Input Event Processing

struct input_event { struct timeval time; __u16 type; __u16 code; __s32 value; };

// Event Types #define EV_SYN 0x00 #define EV_KEY 0x01 #define EV_REL 0x02 #define EV_ABS 0x03 ... // Event Codes #define REL_X 0x00 #define REL_Y 0x01 #define REL_Z 0x02 ...

#include

4 event structs per sensor measurement uses code field to differentiate axis

PresenterPresentation NotesNotes: * recommend reading the header

* Types: ** are a little mismatched for sensor data ** EV_ABS is for absolute measurements- think Touchpad ** EV_REL is for relative measurements- think mouse ** EV_SYN is a sync record to say that These events all go together

* Codes: ** code tells you the axis

Next Slide: * see this in action with a demo

Demo: Input Event Framework

PresenterPresentation NotesNotes:Hands on demo at command line using * cat /proc/bus/input/devices * getevent /dev/input/eventX

Next Slide: * back to the presentation

Input Event Framework Resources

tools for debug

getevent /dev/input/event cat /proc/bus/input/devices

#include

further reading http://www.kernel.org/doc/Documentation/input/ http://www.kernel.org/doc/Documentation/input/event-codes.txt http://en.wikipedia.org/wiki/Evdev Internal input event handling in the Linux kernel and the Android userspace

PresenterPresentation NotesNotes: * input.h is an approachable header to read * get past mouse and keyboard focus * check out the links under further reading * standard tools, standard debug

Gotchas: * beyond x,y,z standards arent so standard * what if you want to return something like calibration data? * EV_ABS events can get dropped by framework * EV_REL events will get dropped if exactly the same as previous event

Next Step: * just showed how raw sensor data comes into the system, but how does android get a hold of it?

libsensor advertises available sensors and makes them available to

Sensor Service

controls sensors and reads data using Linux infrastructure

PresenterPresentation NotesNotes: * Sensor Service asks for things * libsensor handles these requests using the linux infrastructure

Next Step: * the first request: what sensors does this board have?

libsensor advertises sensors

struct sensors_module_t { struct hw_module_t common; int (*get_sensors_list)(struct sensors_module_t* module, struct sensor_t const** list); };

libsensor must define a sensors_module_t struct named

HAL_MODULE_INFO_SYM has a func ptr which returns a list of sensor_t strycts describing

the sensors

PresenterPresentation NotesNotes: * not immediately obvious from the header what to do * define HAL_MODULE_INFO_SYM * object oriented C with function pointers in the structs * comments removed here, and they help a little * will look at how this is commonly implemented later and it will be crystal clear

Next Step: * libsensors makes sensors available

libsensor makes sensors available

struct sensors_control_device_t { struct hw_device_t common; int (*activate)(struct sensors_control_device_t *dev, int handle, int enabled); int (*set_delay)(struct sensors_control_device_t *dev, int32_t ms); int (*wake)(struct sensors_control_device_t *dev); }; struct sensors_data_device_t { struct hw_device_t common; int (*data_open)(struct sensors_data_device_t *dev,

native_handle_t* nh); int (*data_close)(struct sensors_data_device_t *dev); int (*poll)(struct sensors_data_device_t *dev, sensors_data_t* data);

object oriented interfaces to

control and poll sensors

PresenterPresentation NotesNotes: * turning on/off , getting data from them * putting function pointers in structs * comments are removed to focus on methods * method names in blue * for each sensor, you declare one of each of these structs * is much less opaque when you see code that implements it

Next Step: * this is how data is made available to android, but what about data formats?

libsensor reads sensor drivers

typedef struct { union { float v[3]; struct { float x; float y; float z; }; ...

translate from

4 input_events to 1 sensors_vect_t

struct input_event { struct timeval

time; __u16 type; __u16 code; __s32 value; };

put into physical units: deg C, m/s^2

PresenterPresentation NotesNotes: * Sensor Manager asked for data, but need to go out to drivers and get it * input event framework is an easy way to get at our sensor data * drivers have a fixed point representation * android wants data in SI units: m/s^2, degrees Celsius,

Next Step: * how are we doing? Starting to make sense?

libsensor

advertises available sensors and makes them available to Sensor Service

controls sensors and reads data using Linux infrastructure

#include

PresenterPresentation NotesNotes: * recap of what libsensors does * libsensor interface in hardware/sensors.h * not to be confused with any other sensors.h, or linux lm_sensors

Next Step: * how do you actually implement those functions?

adapt an existing implementation

rowboat / hardware-ti-omap3 / rowboat-gingerbread / libsensors / sensors.cpp

rowboat / hardware-libhardware / include / hardware / sensors.h

this is the most common implementation, and is extendable for input event drivers

PresenterPresentation NotesNotes: * open both files, start in sensors.h * talked about input event framework because its the basis for the most common implementation * notice the helpful introductory comments on physical conventions * browse to sensors_module_t, which is implemented in sensors.cpp as sSensorList * later on in the file

Next Step: * rest of Sensors.cpp starts to rely on SensorBase classes to do the control and data structs

root of the implementation

best thing about it is that it works

PresenterPresentation NotesNotes: * public methods mirror a device driver * private methods for dealing with file handles, opening, closing * Best thing about it is that it works, extendable via copy-paste * Im not a fan, because encourages copy/paste * an example of why people dont like C++, but not the fault of C++

Next Step: * must derive a concrete class to utilize it

SensorBase and friends rowboat / hardware-ti-omap3 / rowboat-gingerbread / libsensors / SensorBase.h / AccelSensor.h / GyroSensor.h / InputEventReade

copy/paste if you

have an input event driver

PresenterPresentation NotesNotes: * example from TI repo with Accel and Gyro * much of the functionality is in sensor base * copy paste from the concrete classes AccelSensor GyroSensor * concrete classes have details like device name * InputEventCircularReader does the hard work of converting the input events into sensor_vec_t structs

Next Step: * encourage you do download this source code

libsensor

further reading http://www.kandroid.org/online-pdk/guide/sensors.html

source code

root / device / samsung / crespo / libsensors rowboat / hardware-ti-omap3 / rowboat-gingerbread / libsensors

OpenEtna / android_device_lg_eve / android_device_lg_eve / libsensors

PresenterPresentation NotesNotes: * here are three implementations of libsensor * TI BeagleBoard * Samsung Nexus One aka Crespo * an alternative, somewhat older, libsensors implementation on a phone Ive never heard of * Please share any libsensor specific documentation links you find with me * link from kandroid talks about building

Next Step: * more links..

Linux Infrastructure links

further reading Android Sensor PortingGuide Linux Industrial I/O Subsystem

kernel.org/.../spi/spi-summary kernel.org/.../i2c/summary

kernel.org/.../w1/w1.generic

Getting Started With UInput

PresenterPresentation NotesFurther Reading * recently released, hard to find guide from TI covers similar info as I have * IIO is an alternative to Input Framework, targeted towards high speed A/D drivers and Sensor drivers * Kernel documentation- surprisingly well written, not mired in the details * uinput for making virtual input devices, simulated sensor data

Next Step: * to wrap it all up

remember

keep drivers platform agnostic

leverage existing Linux infrastructure

implement the glue between Android and Linux

PresenterPresentation NotesNotes: * platform specific data goes into lib sensors

Gotchas:be cognizant of GPL and Android Open Source licensing issuesdefinitely GPL at the kernel levellikely AOSL in userland

Next Slide * what I didnt talk about thats going to impact you

whats next

Ice Cream Sandwich not that different Non input-framework based drivers Sensor Fusion daemons

now

Open Sensor Processing standards Dedicated Sensor Processors Sensor Fusion going beyond just orientation

future

PresenterPresentation NotesNotes: * didnt get time to talk about these things * happy to answer questions on them

Ice Cream SandwichAPIs largely the samesome sensor fusion includedEmerging Sensor Related Standards WC3StreamInputDedicated Sensor ProcessorsSensor HubsBig/Little processorsAlternative sensor configurations * wireless sensors * USB based sensors

Alternative driver frameworks * non-input event based drivers * io-ctl based sampling drivers * Linux IIO for more efficientSensor Fusion Daemons * Platform Testing and Verification * using uinput to simulate data

Questions?

rian.sanderson@gmail.com

Slide Number 1the problemthis talk should enable you tocreating sensors systems is an artdemo: BeagleTabbig picturedata flowapp level debugging sensor driverslinux infrastructureInput Event FrameworkInput Event ProcessingDemo: Input Event FrameworkInput Event Framework Resourceslibsensorlibsensor advertises sensors libsensor makes sensors availablelibsensor reads sensor drivers libsensoradapt an existing implementationroot of the implementationSensorBase and friendslibsensor Linux Infrastructure linksrememberwhats nextQuestions?