automating everything with fme
TRANSCRIPT
![Page 1: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/1.jpg)
Automating
Everything with FME
![Page 2: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/2.jpg)
Agenda
• What is Automation?
• Data Loading
Scenarios– On Demand
– Scheduled
– Real Time
• Working with Big Data
• Making Data Available
![Page 3: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/3.jpg)
![Page 4: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/4.jpg)
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
![Page 5: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/5.jpg)
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
Episode #2:
Transformation Menace
These systems are too different and I can’t easily move
data between them.
![Page 6: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/6.jpg)
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
Episode #2:
Transformation Menace
These systems are too different and I can’t easily move
data between them.
Episode #3:
Validation Strikes Back
The data in these systems sucks. I need to filter out the
bad, fix it, and make sure only good stuff gets moved.
![Page 7: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/7.jpg)
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
Episode #2:
Transformation Menace
These systems are too different and I can’t easily move
data between them.
Episode #3:
Validation Strikes Back
The data in these systems sucks. I need to filter out the
bad, fix it, and make sure only good stuff gets moved.
Episode #4:
Automation Awakens
I want data to be moved automatically. It should
happen each night, as soon as data is available, or on-
demand if someone needs it right then.
![Page 8: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/8.jpg)
FME Server is automation.
![Page 9: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/9.jpg)
What is Automation?
Running an
FME WorkspaceDelivering DataTriggering an event
![Page 10: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/10.jpg)
Your Top 3 Automations*
1. Loading Data
2. Processing Large Data
Volumes
3. Making Data Available
* based on 5 years of survey data from FME Server Customers
![Page 11: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/11.jpg)
Loading Data• On Demand
• Scheduled
• In Real Time
![Page 12: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/12.jpg)
Example: Loading On Demand
![Page 13: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/13.jpg)
Example: Scheduled Replication
“FME Server is the caretaker that lets us pay attention to other things.”
![Page 14: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/14.jpg)
Example: Loading in Real Time
![Page 15: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/15.jpg)
Data Loading Tips
✓ Validate data before it’s loaded and
immediately report the quality of a user’s
submission.
✓ For duplicate systems, nightly replication
avoids disruptions during the day.
✓ Keeping systems connected in real time is
better for insight/transparency.
![Page 16: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/16.jpg)
Processing Data Volumes.
Large
Colossal
Big
Gigantic
Huge
Enormous
Gargantuan
![Page 17: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/17.jpg)
Big Data Symptoms
❏ Difficulty figuring out how to handle all your data.
❏ So much data but not enough ability to process it.
❏ Regular use of the term terabyte.
❏ Work with imagery, LiDAR, networks,
live streams.
❏ Performing tasks like modelling, spatial
relationships, change detection, large-
scale map production.
![Page 18: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/18.jpg)
Example: Big Networks (10,000 route queries!)
![Page 19: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/19.jpg)
Example: Big Maps
“The volume of data required to be processed needed an automation workspace run on FME Server.”
![Page 20: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/20.jpg)
Big Data Tips
✓ Look for ways to improve performance in
your FME workspaces.
✓ Use a dedicated FME Server machine.
✓ Use 64-bit FME for full access to memory
resources.
✓ Use multiple FME Engines for concurrent
processing.
✓ Use multiple machines, if you got ‘em.
✓ Estimate the total time and frequency to
decide if automation is worth it.
![Page 21: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/21.jpg)
Example: Big Maps in the Cloud
![Page 22: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/22.jpg)
Tip: Put Big Data in the Cloud.
✓ Scalable ● On-demand processing power.
✓ Reliable● 99.99%+ uptime.
✓ High performance● On-demand upgrade of processing power.
✓ Cost effective● Pay only for what you need.
![Page 23: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/23.jpg)
22 Minutes to Destination
Example: Real-time Networks
Brake pressure OK
Fuel Level at 60%
JMS
![Page 24: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/24.jpg)
“Fast Data” Tips
✓ Use multiple FME Engines for
scalability.
✓ Use message queue software
(JMS, SQS).
✓ Keep FME Engines always
running.
![Page 25: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/25.jpg)
Making Data Available.
![Page 26: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/26.jpg)
Consider: Do you need a
map focus, or a
metadata focus?
![Page 27: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/27.jpg)
Consider: Can you offer
more than just the data?
![Page 28: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/28.jpg)
● FME Server was created for automation.
● Automation can mean event triggers, running a
workflow on a schedule, or making data available on
demand.
● Leverage the full processing power of FME Desktop, FME
Server, and FME Cloud to get the most out of your
automation scenarios.
![Page 29: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/29.jpg)
More info about all of the stories seen here:
● On-Demand Data Loading example (Swedish EPA): http://www.safe.com/fmeuc/automated-quality-controls-with-fme/● Scheduled Replication example (San Jose): http://www.safe.com/customers/san-jose-water-company/● Data Loading in Real Time example (San Jose): http://www.safe.com/presentation/oracle-database-triggers-and-fme-server-for-
task-tracking/● Big Networks example (London Fire Brigade): http://www.1spatial.com/wp-
content/uploads/2015/11/London_Fire_Case_Study_FINAL.pdf● Big Maps example: http://www.safe.com/presentation/national-map-production-with-mapnik-rasterizer/ ;
http://www.imgs.ie/ordance-survey-ireland● Big maps in the cloud example (Weather): http://www.slideshare.net/SafeSoftware/fme-around-the-world-46744668/53 ;
http://blog.safe.com/2015/02/event-driven-architecture-aws-lambda-s3/● “Fast data” railway example: http://blog.safe.com/2014/12/125000-mph-railways/● On-demand analytics: https://prezi.com/33sm_x4x6wvz/fme-on-demand-and-automated-spatial-reporting/● Open data, map focus: http://mslapps.mt.gov/geographic_information/data/databundler/ui_cdp● Open data, metadata focus:
http://findit.york.ca/opendata/portal/#/search?disp=f3566bc6&view=card&sortdir=desc&fq=fss_Categories:Spatial%5C%20Data
![Page 30: Automating Everything with FME](https://reader031.vdocuments.net/reader031/viewer/2022021816/589d6acd1a28abd91d8b5e07/html5/thumbnails/30.jpg)
Thank you!
Questions?