9/30/2014

1

1

Overview of the EMF Refresher Webinar Series

Introduction to the EMF

Working with Data in the EMF – viewing & editing

Inventory Data Analysis and Reporting

2

EMF Resources

EMF User's Guide

http://www.cmascenter.org/emf/internal/guide.html

EPA's CoST website

http://www.epa.gov/ttn/ecas/cost.htm

CoST User's Guide

http://www.cmascenter.org/help/model_docs/emf_cost/2.5.1/CoST_UsersGuide_2012-08-01_Final.pdf

9/30/2014

2

3

Roadmap for Today's Webinar

What is the Emissions Modeling Framework (EMF)?

What can the EMF do?

What challenges can the EMF help solve?

How is data stored/managed in the EMF?

How do I get started in the EMF?

What can EMF tell me about the data?

How can I view and export data?

4

What is the EMF?

Software package built to help in emissions work

Manage emissions data files

Track changes to data files

Create summaries and reports from inventories

Perform QA tasks on inventories

Create controlled and/or future-year inventories

9/30/2014

3

5

How the EMF Works

The EMF client application runs on your local computer

The client application communicates with the remote EMF server

The remote EMF server stores the emissions data and does the number crunching

6

What Can the EMF Do?

Analyze emissions inventory data; for example:

Create inventory summary reports by source characteristics like county, SCC, plant ID

Compare inventory data from one year to another year

Check inventory records for bad values like unknown or missing SCC codes, or negative emissions values

9/30/2014

4

7

What Can the EMF Do?

The EMF includes the Control Strategy Tool (CoST) - covered in separate webinars

Estimate emissions reductions and costs associated with control strategies

Apply control measures to create controlled inventories

Apply growth factors to create future year inventories – growth factors EPA or regional

Customize inventory temporalization to your analysis

8

What Can the EMF Do?

The EMF can manage the inputs and settings needed to run emissions modeling tools like SMOKE

SMOKE converts emissions inventory data into air quality model inputs by applying allocation factors

We will not be covering this topic in this webinar series.

9/30/2014

5

9

What Problems Can the EMF Help Solve?

How do we organize all the various files that are used?

How can everyone see the data that's available and find the data that they need?

How do we track changes? Who made the change, what was changed, and why was it changed.

When updates are made, how do we make sure we use the latest version of the data?

10

What Problems Can the EMF Help Solve?

How do we share information with others?

How do we ensure that created output is reproducible?

How do we keep track of the analysis and QA that's been done for a particular file?

9/30/2014

6

11

Poll: Which of these is notsomething the EMF can do?

a) Keep track of all the inventory files for a project

b) Create a summary of an inventory by county and SCC code

c) Project an inventory from 2011 to 2030

d) Create a summer weekday inventory from an annual inventory

12

How is Data Stored in the EMF? Data files are imported into the EMF to create datasets

During import, the EMF reads the raw data from the file into the EMF database

When you import a file, you tell the EMF what dataset type to use

The EMF knows the format of the data file based on the dataset type

Column names and what kind of data (string, integer, decimal, etc.) is in each column

Data to read must exactly match the dataset type column definition

9/30/2014

7

13

Kinds of Data Managed by the EMF

Emissions inventories

Measured or estimated emissions by source

Emissions data can have different temporal resolutions like annual, monthly, daily, hourly, or seasonal

Data can be base year emissions, future year emissions, and/or controlled emissions

The EMF can read several emissions inventory files in different formats

Knows about the different types of emissions sources like on-road mobile, nonroad, nonpoint, and point

14

Inventory Dataset Types in the EMF

Flat File 2010 [Nonpoint|Point|Activity]

Inventories containing annual or month-specific emissions in different FF10 formats

Flat File 2010 [Nonpoint|Point] Daily

Day-specific inventories in FF10 format

ORL [Fire|Nonpoint|Nonroad|Onroad|Point] Inventory

Annual inventories in different ORL formats

9/30/2014

8

15

Kinds of Data Managed by the EMF

Allocation factors used to calculate more finely detailed estimates from the original emissions inventory data; for example:

Gridding surrogates: factors to allocate county-level emissions to grid cells

Speciation profiles: factors to split emissions by pollutant into chemical mechanism species

Temporal profiles: factors to estimate hourly emissions based on the original temporal resolution of the emissions inventory data

16

Kinds of Data Managed by the EMF

Growth factors used to project a base year inventory to a future year

Control measures to model control technologies

Effectiveness information for the control technology (i.e. control efficiency, rule penetration)

Factors to calculate the engineering costs associated with implementing particular measures

9/30/2014

9

17

Kinds of Data Managed by the EMF

Reference data, including:

State and county codes and names for reporting emissions summarized by geographic area

County latitude and longitude coordinates that can be included in output so data can be visualized using GIS tools

Names for codes like SCC, SIC, MACT, NAICS, etc.

18

Poll: We talked about several kinds of data. Which didn't we discuss?

a) Seasonal point source emissions

b) Hourly temperature data

c) Factors to estimate gridded emissions from county-level emissions

d) Control measure efficiencies

9/30/2014

10

19

Starting Up the EMF

Each time you launch the EMF client you will be prompted to login with your username and password

The first time you start the EMF client you'll need to create a username and password and enter your profile information

20

Main EMF Window

9/30/2014

11

21

EMF Windows The Status window will display information

about tasks the EMF is running

The Downloads window will be used when downloading data or reports from the EMF

22

The Dataset Manager The Dataset Manager is the main interface for

listing the datasets that have been imported into the EMF and finding the dataset you want to work with

To open the Dataset Manager, open the Manage menu at the top of the main EMF window and select Datasets

Initially, no datasets are listed; for now, we'll use the Show Datasets of Type pull-down menu to select “All”

9/30/2014

12

23

The Dataset Manager

24

The Dataset Manager

The Dataset Manager displays a summary of information about each dataset

aka Dataset Properties

The Dataset Manager lets you select and work with datasets

In the homework, you'll explore a few datasets and get familiar with the EMF user interface

9/30/2014

13

25

Viewing the Properties of a Dataset

Select a dataset by checking the checkbox in the Select column next to the dataset name

Click the Edit Properties button to open the Dataset Properties window

The Summary tab will be displayed (see next slide)

26

Dataset Properties: Summary Tab Name: nonroad_caps_2005v2_jul_orl_nc

Description: header of inventory file

Project: assigned when dataset was imported

Creator: user who created the dataset

Dataset Type

Time Period Start and End: determined from file header

Temporal Resolution: determined from dataset type

Sector: set by dataset type

Region: assigned by user

Country: determined from file header

9/30/2014

14

27

Dataset Properties: Summary Tab Status: imported from file vs. created by

EMF

Dates last modified, accessed, and created

Intended Use: public vs. private

Default Version: will discuss in a moment

28

Poll: Which of these properties is not stored for datasets?

a) Date the dataset was created

b) The user who created the dataset

c) Temporal resolution

d) Spatial resolution

e) Geographic region the dataset covers

9/30/2014

15

29

Viewing vs. Editing

Viewing an item: looking at the item but can't change its information

Editing an item: can change information about the item

Only one user can edit a dataset at a time but multiple users can view a dataset

You must edit a dataset in order to change anything about it including adding or running QA steps (we'll discuss more later)

30

How to Find Datasets Use the Show Datasets of Type pull-down menu in

the Dataset Manager

This pull-down menu does have an “All” option to see all the datasets in the system

9/30/2014

16

31

How to Find Datasets

If you have a lot of datasets of a given type, use the Name Contains field

You can type in a string to match the names of datasets

For example, typing 2020 will match all datasets whose name contains “2020”

You can use wildcards in this field: pt*2020 matches datasets whose name contains “pt” followed by “2020”at some point

ptnonipm_2020cc_nc_sc_va_09nov2009_v0

32

How to Find Datasets

Advanced button displays the Advanced Dataset Search window

Match datasets by name, description, creator, project

Find datasets by QA step or data values

Datasets must match all criteria specified (boolean AND)

9/30/2014

17

33

How to Find Datasets

Another option is to filter the list of datasets in the Dataset Manager

Click the Filter toolbar button to display the Filter Rows dialog

Create a filter to only show rows where the Type column contains the word “Inventory”; note that filter criteria are case-sensitive

34

How to Find Datasets

Edit your user profile to limit the list of dataset types that the EMF shows you

In the main EMF window, open the Manage menu and select My Profile to see the Edit User window

In the Edit User window, you'll see a list of Visible Dataset Types and a list of Hidden Dataset Types

Select visible dataset types and click the Hide button to move the selected types to the hidden list

9/30/2014

18

35

How to Find Datasets

36

Poll: How can you find datasets by name?

a) Advanced dataset search

b) Create criteria to filter the rows displayed in the Dataset Manager

c) Enter a search term in the Name Contains field in the Dataset Manager

d) All of the above

9/30/2014

19

37

Dataset Properties: Data Tab

Provides access to the raw data associated with the dataset

Shows information about each version of a dataset

Click View to view the raw data (will be covered in more detail later)

38

Dataset Properties: Keywords Tab

Keywords store additional metadata about datasets

The keywords pass info to the EMF to be used for certain processing like including prefixes or suffixes when exporting a dataset or setting up indexes of the raw data

9/30/2014

20

39

Dataset Properties: Notes Tab

Shows notes that have been made about the dataset

Notes can be questions, answers, comments, etc.

Create a new note by clicking the Add button

40

Creating a Dataset Note

Enter a summary of your note

Select the note type and enter the details

If your note references a previous note (like answering a previous question), select the referenced note

Select which version of the dataset your note refers to

9/30/2014

21

41

Dataset Properties: Revisions

Displays information about revisions that have been made to the raw data in the dataset

Revisions are automatically created when the raw data is edited

42

Dataset Properties: History

Displays a record of when the dataset was exported for use in modeling runs

9/30/2014

22

43

Dataset Properties: Sources

Displays information about where the data associated with this dataset came from (i.e. the original file that was imported)

Also shows which table the raw data is stored in in the database

44

Dataset Properties: QA Steps

Shows a list of QA steps that have been created for this dataset

QA steps represent different actions performed like summarizing the data by county and pollutant or creating a list of unique SCCs in the data

Allows you to add, edit, copy, and delete QA steps

We will discuss QA steps in the next webinar

9/30/2014

23

45

Poll: True or False?

1) You can't view a dataset while someone else is editing it.

2) Every dataset in the EMF uses at least one table in the database to store its raw data.

3) Dataset notes are associated with a particular version of a dataset.

46

Dataset Actions

Refresh: checks with the EMF server and gets the latest information for the dataset

Other users can be editing datasets while you're working so you may need to refresh to see their latest changes

Save: save the changes you've made to a dataset

Export: export the raw data to a file (next slide)

Close: close the Dataset Properties window; if you have unsaved changes, you'll be asked if you want to discard them

9/30/2014

24

47

Exporting Datasets

Exporting a dataset creates a file containing the data

The exported file format is determined by the dataset's type, i.e. an FF10 Nonpoint Inventory dataset will be exported in FF10 Nonpoint format

In the Dataset Manager window, select the datasets you would like to export and click the Export button to display the Export customization window

48

Exporting Datasets

9/30/2014

25

49

Exporting Datasets

Check the Download files to local machine?Checkbox

Creates file on the EMF server that will be automatically downloaded to your computer

The file name is generated from the dataset name

You can enter a file name prefix to add to the file name

If the dataset has multiple versions, you can choose which version you would like to export

Exporting Datasets

• Choose datasets to export

• Download files to local machine? CHECK

• Optional File Name Prefix

• Choose Version

• Filter to limit rows exported

50

9/30/2014

26

51

Exporting Datasets

In the Export dialog, click the Export button to begin the export

Check the Status window for messages:Completed export of SmokeFlatFile_NONPOINT_20140912.csv to

C:\EMF_State_Data\downloads\StateEMF\SmokeFlatFile_NONPOINT_20140912_csv_27sep2014_v0 in 13.104 seconds. The file will start downloading momentarily, see the Download Manager for the download status. No. of records in database: 9,389,860; Exported: 60,460

52

Exporting Datasets In the main EMF window, open the Window menu and

select Downloads to open the Download Manager

Once the progress bar reaches 100%, your download is complete

Right click the filename and choose Open Containing Folder to open the folder where the file was downloaded

9/30/2014

27

Exporting Datasets Local Machine Download

TIP: Re-name with .csv extension to easily open as a spreadsheet

53

Exporting DatasetsOpening as a Spreadsheet

54

9/30/2014

28

Coming up…

• Homework: links to be provided in follow-up email

• EMF Refresher, Part 2:

• October 21, 1:30 – 3:00

• How do I QA and summarize inventory data?

• What is a QA step?

• Creating a QA step: new and from templates

• Comparing datasets

• Exporting data in KMZ format to view in Google Earth

55

Preview of EMF Refresher, Part 2QA step results:• What are the statewide totals by pollutant?• For each pollutant, which counties and

SCCs?• EMF bonus – SCC descriptions

56

9/30/2014

29

57

Acknowledgement

Institute for the Environment from University of North Carolina at Chapel Hill