farmbeats: ai & iot for agriculture · •end-to-end iot system for seamless data-driven...
TRANSCRIPT
![Page 1: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/1.jpg)
FarmBeats: AI & IoT for Agriculture
Ranveer Chandra Sudipta Sinha
Microsoft Research
![Page 2: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/2.jpg)
Data-Driven Agriculture
2
![Page 3: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/3.jpg)
Data-Driven Agriculture
Ag researchers have shown that it:
• Improves yield
• Reduces cost
• Ensures sustainability
3
![Page 4: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/4.jpg)
But…
According to USDA, high cost of manual data collection prevents farmers from using data-driven agriculture
4
![Page 5: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/5.jpg)
IoT System for Agriculture
5
![Page 6: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/6.jpg)
Problem 1: No Internet Connectivity
• Most farms don’t have any Internet coverage
• Even if connectivity exists, weather related outages can disable networks for weeks
6
![Page 7: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/7.jpg)
Problem 2: No Power on the Farm
• Farms do not have direct power sources
• Solar power is highly prone to weather variability
7
![Page 8: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/8.jpg)
Problem 3: Limited Resources
• Need to work with sparse sensor deployments
• Physical constraints due to farming practices
• Too expensive to deploy, and
• Cumbersome to maintain
8
![Page 9: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/9.jpg)
FarmBeats
• End-to-end IoT system for seamless data-driven agriculture
9
FarmBeats Farm Services
![Page 10: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/10.jpg)
Outline
• End-to-end IoT system for seamless data-driven agriculture
• Challenges:▪ Network Connectivity on the Farm
▪ Low-cost aerial mapping and image analysis
▪ Limited Sensor Placement
▪ Internet Connectivity
▪ Power Availability
• Deployment in two farms in NY and WA for six months ()
10
![Page 11: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/11.jpg)
Outline
• End-to-end IoT system for seamless data-driven agriculture
• Challenges:▪ Network Connectivity on the Farm
▪ Low-cost aerial mapping and image analysis
▪ Limited Sensor Placement
▪ Internet Connectivity
▪ Power Availability
• Deployment in two farms in NY and WA for six months ()
11
![Page 12: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/12.jpg)
Challenge: Network/Internet Connectivity
(Farmer’s home/office) Cloud
12
![Page 13: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/13.jpg)
Challenge: Internet Connectivity
(Farmer’s home/office) Cloud
Sensors • Few miles away• Obstructed by vegetation, terrain etc.
13
![Page 14: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/14.jpg)
TVWS using Dynamic Spectrum Access (DSA)
• Determine available spectrum (white spaces)
• Transmit in “available frequencies”
• Detect if primary user appears
• Move to new frequencies
• Adapt bandwidth and power levels
Pow
er
Frequency
PU1
PU2
PU4
PU3
14
![Page 15: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/15.jpg)
MSFT Supported Pilot ProjectsCommercial Scale is the Next Step
SingaporeKenya
Uruguay
Philippines
Tanzania
Projects (15)Namibia
Ghana
200,000+ Population
Under Coverage
36,000 K-12 Students
Being Connected
38,500 University
Students Getting MSFT
Devices & Services
Taiwan
Indonesia
Scotland
Bhutan
Seattle Center
South AfricaBotswana
Launched 14 pilot projects and the global
Dynamic Spectrum Alliance. Objective: Trigger
regulatory changes and develop commercially
viable ecosystems for
TV White Spaces technology.
Regulatory changes achieved in USA &
Singapore with drafts in UK and Canada. Critical
progress made in ITU-D and ITU-R.
India
![Page 16: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/16.jpg)
TV White Spaces in the Farm
• What are the TV White Spaces?• Unused TV channels
• Benefits over Wi-Fi, Zigbee, etc• High throughput at long range
• Key insight for farms:• “lots” of TV spectrum is available, more than 100 MHz
• Just like Wi-Fi router covers the home, TVWS base station can cover the farm
dbm
Frequency
-60
-100
“White spaces”
470 MHz 700 MHz
![Page 17: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/17.jpg)
Outline
• End-to-end IoT system for seamless data-driven agriculture
• Challenges:✓Network Connectivity on the Farm
▪ Low-cost aerial mapping and image analysis
▪ Limited Sensor Placement
▪ Internet Connectivity
▪ Power Availability
• Deployment in two farms in NY and WA for six months ()
17
![Page 18: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/18.jpg)
Aerial imagery in precision agriculture
• Aerial imagery from drone, UAV, balloons, …
• Orthomosaics (2D image) or 3D point cloud
• Visual monitoring
• Crop inspection
• Many applications
18
![Page 19: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/19.jpg)
Aerial imagery in precision agriculture
19
3D point cloud reconstruction (using aerial 3D photogrammetry)
Cameras corresponding to video keyframes shown in blue
![Page 20: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/20.jpg)
Aerial imagery in precision agriculture
20
▪ Video from ~2 minutes flight▪ High-resolution RGB orthomosaic ▪ Resolution: 2—3 cm per pixel
![Page 21: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/21.jpg)
Aerial imagery in precision agriculture
▪Several advantages over satellite imagery▪Off-the-shelf drones
▪ Automatic and repeatable capture
▪Unaffected by cloud cover
▪High resolution imagery
▪ processing on an ordinary PC
21
Carnation, WA
India
![Page 22: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/22.jpg)
Processing multi-spectral imagery
22
Sequoia multi-spectral camera
RGB
NIR Red edge
Red Green
![Page 23: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/23.jpg)
Time-series of aerial maps & orthomosaics
23
▪ Repeated drone flights (uses same plan with GPS waypoints)▪ Completely automated pipeline using computer vision and image
processing technology
![Page 24: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/24.jpg)
Lower cost alternatives
▪UAVs are costly, have limited battery life
▪ Regulatory concerns in some countries
▪ Alternatives we are exploring▪ Tethered Helium balloons with smartphone payload
▪ Balloon manually moved around (person, tractor, …)
▪ Automatic video processing (same as described earlier)
▪ Automatic balloon time-lapse imagery▪ Long-term monitoring and analyzing change
▪Ongoing deployments in India
24
![Page 25: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/25.jpg)
Outline
• End-to-end IoT system for seamless data-driven agriculture
• Challenges:✓Network Connectivity on the Farm
✓Low-cost aerial image analysis
▪ Limited Sensor Placement
▪ Internet Connectivity
▪ Power Availability
• Deployment in two farms in NY and WA for six months ()
25
![Page 26: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/26.jpg)
Challenge: Sparse Ground Sensor Placement
▪ Ground Sensors deployed sparsely on the farm ▪ Reasons:
- Sensors are expensive, - Difficult to maintain too many- Farming constraints
Main idea:
Make predictions with a model that fuses UAV aerial imagery and ground sensor data
26
![Page 27: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/27.jpg)
Approach: Supervised Learning
▪Dataset▪Build geo-registered aerial map ▪ Store data from geo-registered ground sensors▪Compile dataset (X: image patches, Y: sensor data)
▪Learning stage:▪Train a Gaussian Process (X → Y)
▪Prediction stage:▪Process aerial map (in sliding window fashion)▪Extrapolate sensor values to areas with similar ▪ visual features
27
![Page 28: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/28.jpg)
Outline
• End-to-end IoT system for seamless data-driven agriculture
• Challenges:✓Network Connectivity on the Farm
✓Low-cost aerial image analysis
✓Limited Sensor Placement
▪ Internet Connectivity
▪ Power Availability
• Deployment in two farms in NY and WA for six months ()
28
![Page 29: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/29.jpg)
The Real World
(Farmer’s home/office)
Base Station
TV White Spaces
Cloud
Few miles
Sensors
• Weak Connectivity• Prone to outages
29
Wi-Fi, BLE
![Page 30: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/30.jpg)
What Services We Can Provide
30
Image stream Sensor data stream Notes, event, etc.Images, VideoWeather data
Yield Weather forecast
Pano3D
const.
Soil health
analysis
Activity tracking
Task reminder
Heatmap
Crop identification
Yield variation
Yield prediction
Harvest risk assess. Weed
detection
Pest inspection
Irrigation Scheduling
Livestock monitor
![Page 31: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/31.jpg)
Service Characteristics
• Large inputs
31
▪ Latency constraints
Data
Source
Daily
size
Sensor 70K
Drone video 10G
Drone image 4G
Camera 28M
Service Latency
Query sensor data seconds
Livestock mon. seconds
Irrigation sched. hours
Pest inspection hours
Variability analysis Days
![Page 32: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/32.jpg)
The Ideal World
32
![Page 33: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/33.jpg)
The Real World
(Farmer’s home/office)
Base Station
TV White Spaces
Cloud
Few miles
Sensors
• Weak Connectivity• Prone to outages
33
Wi-Fi, BLE
![Page 34: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/34.jpg)
FarmBeats Design
(Farmer’s home/office)
Base Station
TV White Spaces
Cloud
Few miles
Sensors
34
Wi-Fi, BLEFarmBeats Gateway
(IOT Edge)
Analytics
Seed vendorsAgronomistsFarmers…
Weather Data(Rain, Wind, Pollen)
Recommendations(daily best practices)
![Page 35: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/35.jpg)
Storage
FTP Server
FarmBeats Gateway (Windows PC)Sensors
Camera
Drone
MQTT Broker
Video Processor
Panorama Gen
HeatMap Gen
Sensor Interface Local Computation Ag Services
Precision Irrigation
Precision pH
Yield Prediction
Pest Infection
Precision Fertilizer
…
Streaming Analytics
IoT Hub
Storage
Web Server(Offline Access & 3-D Walkthrough)
Web Server
Drone Flight
PlannerCloud Sync
HTTP/ AMQP
EdgeCNN
• Can run offline• Unique Gateway services• Deep Learning at Edge• Component Migration
![Page 36: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/36.jpg)
Outline
• End-to-end IoT system for seamless data-driven agriculture
• Challenges:✓Network Connectivity on the Farm
✓Low-cost aerial mapping and image analysis
✓Limited Sensor Placement
✓Internet Connectivity
▪ Power Availability▪ Solar power, RF power, UAV battery life
• Deployment in two farms in NY and WA for six months ()
36
![Page 37: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/37.jpg)
Deployment
• Deployed in two farms • Carnation, WA (since April 2016)
• Essex, NY (six months)
• Farm area was 5 and 2000 acres
• Sensors:• Drones: DJI Phantom 2, DJI Inspire1, 3DR Solo
• Particle Photons with Moisture, Temperature, pH Sensors
• IP Cameras to capture RGB/IR imagery
• Cloud Components: Azure Storage and IoT Suite
SensorModule
SensorModule
37
![Page 38: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/38.jpg)
Deployment Statistics
• Used 10 sensor types, 3 camera types and 3 drone versions
• Deployed >100 sensors and ~10 cameras
• Collected >10 million sensor measurements, >0.5 million images, 100 drone surveys
• Resilient to week long outage from a thunderstorm
38
![Page 39: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/39.jpg)
High-resolution Orthomosaic from Essex, NY
Water puddle Cow excreta Cow Herd Stray cow
39
![Page 40: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/40.jpg)
Precision Map: Orthomosaic
40
![Page 41: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/41.jpg)
Precision Map : Moisture
41
![Page 42: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/42.jpg)
Precision Map : pH
42
![Page 43: FarmBeats: AI & IoT for Agriculture · •End-to-end IoT system for seamless data-driven agriculture •Challenges: Network Connectivity on the Farm Low-cost aerial mapping and image](https://reader030.vdocuments.net/reader030/viewer/2022041023/5ed649a70c1f140c715b6890/html5/thumbnails/43.jpg)
Precision Map: Accuracy
0
0.2
0.4
0.6
0.8
1
1.2
Temp (F) pH (0-14) Moist (0-6)
Mea
n E
rro
r
FarmBeats LeastCount
FarmBeats can accurately expand coverage by orders of magnitude using a sparse sensor deployment
43