hacking for salone: drone races - di saverio; lippolis - codemotion milan 2016
TRANSCRIPT
![Page 1: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/1.jpg)
HACKING FOR SALONE: DRONE RACESEMANUELE / SIMONE
![Page 2: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/2.jpg)
EMANUELE DI SAVERIOPRINCIPAL DESIGN TECHNOLOGIST
Emanuele started writing software for mobile in the pre-iPhone era. Didn’t stop since then.
WHO’S ON THE STAGE
SIMONE LIPPOLISSENIOR DESIGN TECHNOLOGIST
Simone started writing software when COBOL was a hype word. Now his interests span from web-bots to data-visualization to embedded applications and IOT.
![Page 4: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/4.jpg)
![Page 5: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/5.jpg)
EVERYTHING STARTED ON A BLACK FRIDAY
![Page 6: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/6.jpg)
WE NEEDED
MORE ENTERTAINING
TO MAKE IT
![Page 7: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/7.jpg)
A CLASSIC GAME CAME IN OUR HELP
7
![Page 8: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/8.jpg)
![Page 9: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/9.jpg)
![Page 10: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/10.jpg)
![Page 11: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/11.jpg)
TODO LIST
TALK TO EACH OTHER
SENSE EACH OTHER
HARDER, BETTER, FASTER, STRONGER
![Page 12: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/12.jpg)
TALK TO EACH OTHER
![Page 13: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/13.jpg)
COMPONENTSARCHITECTURE
• Drones on wheel (JumpingSumo by Parrot)
• An open-security 5 Ghz Wi-Fi AP
• Android Smartphones
• A WebSocket Server
ControllerMobile App
JumpingSumolinux+busybox
Game ServerNodeJS
ControllerMobile App
JumpingSumolinux+busybox
ControllerMobile App
JumpingSumolinux+busybox
WebSocket
. .
![Page 14: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/14.jpg)
The Parrot robots work by creating an 802.11ac themselves - smartphones can connect to that and control though applications built with the Parrot SDK.This disables internet access for the smartphones - we needed to “hack” it.
HACKING CONNECTION
$ telnet 192.168.1.1
accesses the BusyBox environment to edit config files /etc/init.d/init_manager and dragon.conf
Drones will connect to your open WiFi at reboot, where they will find Android phones.
![Page 15: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/15.jpg)
The server doesn’t need to provide much functionality, but rather route events in a soft-realtime fashion to the players, with litte computation.• WebSockets help avoid TCP RTT• nodeJS because it can scale to gazillions of
eventsWe have 4 players at at time :)
A BROKER SERVER
![Page 16: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/16.jpg)
SENSEEACH OTHER
![Page 17: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/17.jpg)
WHERE IS MY DRONE?
BLE BEACON/FINGERPRINTING IR MARKER
TANGO
![Page 18: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/18.jpg)
LET DRONES “SEE”
The hardware platform features a front camera for still photos and streaming.Specs not great - we’ll need to design around.
Camera Spec
640x480
15 FPS
Fixed focus
Motion JPEG
![Page 19: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/19.jpg)
COLOR BLOBS
Using OpenCV as resource for highly-optimized algorithm library, we tried to detect color blobs by • thresholding• finding contours• centroidsand applying some heuristics
![Page 20: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/20.jpg)
FIDUCIAL MARKERS
We also res
B&W Square markers are much more robust and can apply adequate Hamming-distance encodings.As an added benefit, they are more robust with respect to luminosity.
![Page 21: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/21.jpg)
![Page 22: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/22.jpg)
IMAGE PROCESSING PIPELINE
PARROT SDK
IMAGE RE-ENCODE
ARTOOLKIT
MARKER COORDINATES
Each of the incoming frames from the drones has to go through heavy processing to detect a marker.
66 ms
3D MODELROTOTRASLATE COORDINATES
![Page 23: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/23.jpg)
HARDER, BETTER, FASTER, STRONGER
![Page 24: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/24.jpg)
OUT OF THE JAVALAND
First problem relates to the sheer computation power needed to conver binary image formats - Java implementation is just too slow.We had to port algorithm implementation to RenderScript to have a fully native implementation that performs acceptably.
JVM RS RS_MULTI
AVG 283,60 1,73 3,24
STDDEV 38,17 2,49 1,52
MEDIAN 262,11 0,86 2,78
MAX 447,57 10,04 15,18
![Page 25: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/25.jpg)
OUT OF THE GC LAND
The main source of delays in real-time applications like games is the Garbage Collector - we’re talking half second freeze on recent hardware.This means we cannot allocate memory in the game loop - Android memory profiler tools will help you discover and eliminate:• Bitmap.create()• new Array[] • Object creations / Autoboxing
![Page 26: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/26.jpg)
FULL CONTROL FLOW
PARROT SDK(CAMERA)
IMAGE RE-ENCODE
ARTOOLKIT
MARKER COORDINATES
3D MODELROTOTRASLATE COORDINATES
GAME STATE
NETWORK EVENTS
GAS PEDAL
USER INTERFACE
STEERING(GYRO)
BROKER SERVER
OTHERPLAYERSACTIONS
![Page 27: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/27.jpg)
FULL CONTROL FLOW
PARROT SDK
IMAGE RE-ENCODE
ARTOOLKIT
MARKER COORDINATES
3D MODELROTOTRASLATE COORDINATES
GAME STATE
NETWORK EVENTS
GAS PEDAL
USER INTERFACE
STEERING
BROKER SERVER
66 ms
33 ms
~ 15 s16 ms
~ 50 ms
STEERING(GYRO)
OTHERPLAYERSACTIONS
![Page 28: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/28.jpg)
Handling all the different events coming in at different rates would a very complex task to handle in a fully procedural way in a standard game loop:
REACTIVE STREAMS FTW
while (true) {
acquire_image();
process_markers();
get_steer_input();
get_other_players_input();
apply_cmds_to_drone();
apply_game_events();
update_ui();
}
.
.
.
.
![Page 29: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/29.jpg)
REACTIVE STREAMS FTW
Rx provides common semanthics to express and manipulate a collection of asynchronous events
![Page 30: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/30.jpg)
RXJAVA
http://www.slideshare.net/hazam/programming-sideways-asynchronous-techniques-for-android
(2014)
Plenty of good introductions to RxJava in the InterWebs
Exercises
https://github.com/jhusain/learnrxjava
![Page 31: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/31.jpg)
KOTLIN MAY BE THE ONE
We may have found a winner.
Kotlin is a deep language with many modern features, including Functional approaches.
The three best features for Kotlin on Android:
1. Overhead is at compile time, Runtime Overhead is minimal
2. Closures and SAM
3. Java Interoperability is a breeze
One things I’m still undecided if is good or bad: Nullability
FUNCTIONAL REACTIVE PROGRAMMING WITH KOTLIN ON ANDROID - BY GIORGIO NATILI
![Page 32: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/32.jpg)
![Page 33: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/33.jpg)
REFERENCES
https://github.com/frog/a-kart
http://designmind.frogdesign.com/2016/07/hacking-a-multi-game-user-drone-race/
SIMONE LIPPOLIS
EMANUELE DI SAVERIO
@simonelippolis
@hazam
GAME & UI DESIGN, SERVER DEVELOPMENT
MOBILE DEVELOPMENT, COMPUTER VISION
FEDERICO BRESSAN
ELENA MARENGONIRACETRACK, DRONE SKINDESIGN
RACETRACK, DRONE SKINDESIGN
![Page 34: Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016](https://reader031.vdocuments.net/reader031/viewer/2022020411/587080f91a28ab57368b660f/html5/thumbnails/34.jpg)
THANKS!