building data mashup solutions with nyc open data...copyright © 2019, oracle and/or its affiliates....

Copyright © 2019, Oracle and/or its affiliates. All rights reserved.

Building Data Mashup Solutions with NYC Open Data

Marc Sewtz Senior Software Development Manager Oracle Application Express June 19th, 2019


•Marc Sewtz • Senior Software Development Manager

• Oracle Application Express / Database Tools

• Oracle Germany, Hamburg 1998-1999

• Oracle USA, New York, NY 1999 - today

• Member of the APEX Dev. Team since 2002

• Twitter: @msewtz

• LinkedIn: http://www.linkedin.com/in/msewtz

2


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.

3


Agenda

Introduction

NYC Open Data

Socrata Open Data API

Low Code Development

Building the App

1

2

3

4

5

4


Introduction

NYC Open Data



Building the App

1

2

3

4

5

5

Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 6

What Problem are we trying to solve?


7

Information available on the Cloud

Data stored in Enterprise Database

How to make sense of disparate data,

gain insights and make decisions?

?


8

? New York State

Math and ELA Test Results (Open Data APIs)

Real World Example:

Choosing a School in New York City

New York City Public School Directory

(stored in Oracle DB)


What Will you Learn?


Interfacing with REST-based

Pubic Open Data APIs

Building Responsive Web Apps on the Oracle Database

Building a Data Analytics and

Decision-Making App with Oracle APEX

Low Code App Dev with Oracle APEX App


Introduction

NYC Open Data



Building the App

1

2

3

4

5

11


Municipal Open Data

• Municipal governments produce and commission large amounts of data and information every day:

– 311 calls, financial data, demographics, public transportation information, crime statistics, building permits, ….

• Making these datasets available to the public enables these institutions to become more transparent and accountable to their citizens.

• By encouraging the use and free distribution of datasets, governments are:

– Inviting citizens to more actively engage with public agencies

– Promoting the creation of innovative, citizen-centric services and solutions.

What is Municipal Open Data?

12


Open Data - New York City

The NYC Open Data portal provides access to over 2000 public data sets.

Mission Statement:

Our Mission: Open Data For All

Open Data is an opportunity to engage New Yorkers in the information that is produced and used by City government. We believe that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker.

opendata.cityofnewyork.us

13


Introduction

NYC Open Data



Building the App

1

2

3

4

5

19


The Socrata Open Data API

• Socrata provides services to make existing government data discoverable, usable, and actionable for government workers and the public.

• Solution allows public sector institutions to automate the flow of data from their primary line of business systems, into publicly accessible servics.

• The Socrata Open Data API (SODA) is the most widely adopted government open data solution.

Overview

20



• Example: 2013-2017 NYC School Math Results: https://data.cityofnewyork.us/resource/stka-4ti9.json

• All resources are accessed through a common base path of /resource/ along with their dataset identifier.

• All datasets have a unique identifier - eight alphanumeric characters split into two four-character phrases by a dash.

• Once you’ve got your API endpoint, you can add on filtering and SoQL parameters to filter and manipulate your dataset.

Finding your API endpoint

21

Adapted from: https://dev.socrata.com/consumers/getting-started.html

https://dev.socrata.com/docs/filtering.html

https://dev.socrata.com/docs/queries.html

https://dev.socrata.com/consumers/getting-started.html




The Socrata Open Data API Finding your API endpoint



• SODA APIs are self-describing — schema and contents of the dataset itself determines how it can be queried.

• Any field within the data can be used as a filter, simply by appending it to the API endpoint as a GET parameter.

• For example, get Math test results for specific school:

• https://data.cityofnewyork.us/resource/stka-4ti9.json?dbn=15K088

Simple Filters

24







25



• “Socrata Query Language” (SoQL) is a simple, SQL-like query language designed for making it easy to work with data on the web.

SoQL Queries

26

Parameter Description Default In $query

$select The set of columns to be returned, similar to a SELECT in SQL All columns SELECT

$where Filters the rows to be returned, similar to WHERE No filter WHERE

$order Column to order results on, similar to ORDER BY in SQL Unspecified order ORDER BY

$group Column to group results on, similar to GROUP BY in SQL No grouping GROUP BY

$having Filters the rows that result from an aggregation No filter HAVING

$limit Maximum number of results to return 1000 LIMIT

$offset Offset count into the results to start at, used for paging 0 OFFSET

$q Performs a full text search for a value. No search N/A

$query A full SoQL query string, all as one parameter N/A N/A


https://dev.socrata.com/docs/queries/select.html

https://dev.socrata.com/docs/queries/where.html

https://dev.socrata.com/docs/queries/order.html

https://dev.socrata.com/docs/queries/group.html

https://dev.socrata.com/docs/queries/having.html

https://dev.socrata.com/docs/queries/limit.html

https://dev.socrata.com/docs/queries/offset.html

https://dev.socrata.com/docs/queries/q.html

https://dev.socrata.com/docs/queries/query.html






• Example: NYC School Demographics - 5-year trend:

– https://data.cityofnewyork.us/resource/ 98et3mve.json? $select= year, sum(total_enrollment), avg(poverty_2)& $group=year& $order=year

SoQL Queries

27

[

{

"avg_poverty_2": "76.4030635838150289",

"sum_total_enrollment": "1046041",

"year": "2013-14"

},

{

"avg_poverty_2": "75.0199435028248588",


"year": "2014-15"

},

{

"avg_poverty_2": "73.8934626038781163",


"year": "2015-16"

},

{

"avg_poverty_2": "73.6072956664838179",


"year": "2016-17"

},

{

"avg_poverty_2": "77.0819956616052061",


"year": "2017-18"

}

]


Introduction

NYC Open Data



Building the App

1

2

3

4

5

28


Low Code

29

Low code platforms enable rapid delivery of business

applications with a minimum of hand-coding and less

upfront investment in setup, training, and deployment.

Forrester Wave 2016

“ ”



30

Low barrier of entry

Less coding

High Productivity

Concentrate on business requirements

Release apps faster with minimal investment

Why use Low-Code Frameworks?


• Many Open Data portals provide web-based data viewers for basic research and analysis.

• Real power of Open Data is in REST APIs, making data available to be consumed by other systems and services.

• An easy-to use, Low-Code PaaS enables a broad audience of users to build and publish their own solutions and services.

• Cloud-based Low Code tools allow anyone to get started in minutes and makes services available to anyone instantly.

Low Code Development Why use Low-Code Frameworks with Open Data?


Oracle APEX Low Code Development with Oracle Application Express (APEX)

32

Leverages SQL skills and Oracle Database capabilities

Easy to learn, develop and deploy

All-inclusive AppDev and Database development IDE

Used to build apps to manage and visualize data

Popular framework in Oracle Cloud, on-premises, and other Clouds

Integrates with Oracle JET


Oracle APEX Oracle APEX is suitable for…

33

Rapid Application Development

Filling gaps in corporate systems

Modernization of legacy systems

Replacing spreadsheets

Large mission-critical apps for thousands of users


Introduction

NYC Open Data



Building the App

1

2

3

4

5

34


Learn More:

apex.oracle.com

Free, Cloud-Based Application Development

building data mashup solutions with nyc open data...copyright © 2019, oracle and/or its affiliates....

Documents