Managing and

Sharing Imagery in

ArcGISKristen Hocutt-Esri

Gerry Kinn- Esri

Mark Romero- Esri

Charlene Sylvester- US Army Corp of Engineers

Topics covered

• Introduction to Imagery

• Architecture

• What is the mosaic dataset and why you should use it?

• Management:

• How to manage imagery on a personal level?

• How to manage imagery for a group?

• How to manage imagery for large scale operations?

• Sharing:

• Imagery using mosaic datasets/image services.

• Image Services and Cloud resources

Tour of the

Living AtlasDemonstration

Professional

Imagery &

Geospatial

Analysts

Enterprise

ArcGIS is a Comprehensive Imagery Platform

Empowering users to make informed decisions

System of

Record

Manage and process

all your imagery

System of

Insight

Extract Information

from Imagery

System of

Engagement

Share imagery products and

information

Content:

ArcGISOnline

Partners, Org

LiDAR

Thermal

Panchromatic

Multi-Spectral

RADAR

Full Motion Video (FMV)

Aerial

Satellite

Making Imagery Accessible and Valuable

ArcGIS Integrates imagery from multiple sources

Manage and Serve Imagery from a Wide Range of Sensors

Scaling the use of imagery

• One image at a time

- Files

- Image Analysis Window (IAW) / Image Ribbon

- Geoprocessing

• Collections of images

- Mosaic Datasets

- Image Analysis Window (IAW) / Image Ribbon

- Geoprocessing

• Imagery as a Service

- Web

- Image Analysis Window (IAW) / Image Ribbon

- Developer APIs / Web apps

- ArcGIS Online / Portal

What are my choices when working with imagery

To mosaic a dataset or mosaic dataset?

Anatomy of the Mosaic Dataset

The optimum model for managing large image and raster collections

Image Management

AerialSatellite

Raster Maps

Elevation

LiDAR

Imagery

Bathymetry

Mosaic

Dataset

Imagery Management

Primary Information Model

• Raster Dataset

• Raster Types

• Mosaic Datasets

• Image Services

Processing• Raster Functions• Raster Analytics• Authored - Geoprocessing Tools

Raster DatasetArcGIS Imagery Information Model

• Represents a basic image with basic behavior.

• read and write image storage (pixels) and metadata

• Comprehensive Support

• Read image files directly – no need to convert

“It’s what you get if you drag a GeoTIFF into ArcMap.”

Raster TypeArcGIS Imagery Information Model

• Represents the logic for a particular sensor or image product coming from a vendor.

Applanix

CADRG

ECRG

CIB

DMCii

DTED

Formosat-2

GeoEye-1

HRE

IKONOS

ISAT

Kompsat-2

Kompsat-3

Landsat 1-5

MSS

Landsat 1-5

TM

Landsat 7

ETM+

Landsat 8

LAS

NITF

Pleiades-1

Quickbird

RapidEye

Radarsat 2

Sentinel-1

SOCET

(SUP)

SPOT 5

SPOT 6

WorldView-

1

WorldView-

2

WorldView-

3

WorldView-

4

…

• Catalog of

- Imagery

- Associated metadata

- Processing functions to be applied

• Stored in Geodatabase

• Authored using ArcGIS for Desktop

• Enables

- Dynamic Mosaicking

- On-the-fly processing

What is a Mosaic Dataset?ArcGIS Imagery Information Model

“You web enable your imagery by publishing your Mosaic

Dataset as an Image Service.”

Raster ProductArcGIS Imagery Information Model

• The Raster Product is the primary information model component which makes it easy to use

intelligence which is provided by the Raster Type

• It’s role is to represent Imagery Information Model intelligence as products and product

information to the user

• A Raster Product

- enables ArcGIS user interface shortcuts to well known band combinations and processing chains

- allows the user to think about products and not files

• Sensor and/or Format Specification specific

- they are based on Raster Type(s)

• ArcMap Catalog window / ArcCatalog

Image ServiceArcGIS Imagery Information Model

• The Image Service is the primary information model component which web enables imagery

• It’s role is to provide Imagery as a Service (one aspect of GIS as a Service)

Pictures

Pixels

Metadata

Image as a Service

Download / Upload

Creating a

Mosaic DatasetDemonstration

What drives Imagery Management?

• Collection

• Modality

• Size

• Images

• Collections

• Metadata

• Geometry

• Formats

• Bandwidth

• Services: Imagery, functionality

• Storage models: Local, network, cloud

• Process frameworks: GP vs. RF vs. RA

Data

Architecture

Server

Desktop

Collections

of Imagery

& Rasters

Dynamic Mosaicking & On-the-fly Processing

Working with Mosaic Datasets

On-The-Fly ProcessingCreating Multiple Products from a Single Source

fx

Raster FunctionArcGIS Imagery Information Model

Image Raster Function

Pixel BlockVisualization or Analytical

“Virtual” Product

Cache or New Image

ScreenDesktop, Mobile, Web

Raster Function“Chain”

Raster Function or Geoprocessing Tool?ArcGIS Imagery Information Model

• Geoprocessing Tools

- Esri does not implement what I need as a Raster Function

- My processing requires integration of feature data (vectors)

- I have complex GP Models (conditionals, iterations, custom script tools)

- Algorithms which are not well suited for block level processing (cost distance)

• Raster Functions

- If you can, you should (storage savings, time savings, flexibility to change,..)

- Esri provides all the Raster Functions I need to produce my products

- It’s acceptable in my application to use an ephemeral or intermediate results

- Visualization

- Analysis results which can be consumed per request based on an AOI

ArcGIS Builds Higher Quality Products

• Color Correction using the Mosaic Dataset

ArcGIS Builds Higher Quality Products

• Automatic seamline generation using the mosaic

dataset

Information Flow in an Image Service

Raster Types

MosaicDataset

ImageryNativeForm

Points

to original

ImageryRasterFunctions

Harvests Metadata

ArcGISServer

Collection

Raster Products

Publishing

Desktop Web Device

ImageService

ArcGISDesktop

On-the Fly

ProcessingDemonstration

Sharing Mosaic Datasets with ArcGIS Enterprise

Sharing Mosaic Datasets

• Share by Value

- Imagery is copied to the ArcGIS

Server configuration store

(e.g. C:\arcgisserver\config-

store\services)

- Data will be converted into File

Geodatabase

- NOT recommended for large image

collections

Sharing Imagery

Image Server

Server

configure store Publish

Copy

Image(s)

• Share by reference

- Register a file share and/or

enterprise geodatabase(s) with the

server data store

- Data is not moved

- Recommended for publishing

mosaic dataset or large single

images or large single raster data

Sharing Imagery

Mosaic Dataset

And/Or

Image Server

Server

configure store Publish

Registered

Imagery

217217217

200200200

255255255

000

163163163

131132122

2396553

110135120

1129256

62102130

1025648

130120111

237237237

8011927

252174.59

“The views, opinions and findings contained in this report are those of the authors(s) and should not be construed as an official Department of the Army position, policy or decision, unless so designated by other official documentation.”

Prepared by Charlene Sylvester

For 2018 ESRI Federal User Conference

20-21 March 2018

USACE NATIONAL COASTAL MAPPING PROGRAM:IMAGERY MANAGEMENT WITH CLOUD SERVICES

28JOINT AIRBORNE LIDAR BATHYMETRY

TECHNICAL CENTER OF EXPERTISE

28

Stennis International AirportKiln, MS

OPERATIONS

RESEARCH AND DEVELOPMENT

Hardware

Data exploitation

Procedures

Surveys

Software

Algorithms

Aircraft

People

Jennifer.M.Wozencraft@usace.army.mil 228-806-6044

19th Annual JALBTCX Airborne CoastalMapping and Charting Technical Workshop

26-28 June 2018Providence, RI

29

Lid

ar

Bath

ym

etr

y a

nd T

opogra

phy

West P

alm

Beach, F

L, 2016

Goals• Develop regional, repetitive, high-

resolution, high-accuracy elevation

and imagery data

• Build an understanding of how the

coastal zone is changing

• Facilitate management of sediment

and projects at a regional, or

watershed scale

31EMERGENCY RESPONSE OPERATIONS AND

RAPID DATA PRODUCT PRODUCTION

2008 - Ike

2012 - Sandy

2016 - Matthew

2017 - *Irma

Deployments for regional-scale, post-storm damage assessments:

2004 - Charley, Frances, Ivan, Jeanne

2005 - *Dennis, Katrina, Rita, Wilma

*JALBTCX deployments for FEMA

Acquire Process Edit Classify Initial QC Final QC Production Delivery

32

BASIC NCMP DATA PRODUCTS

• JALBTCX standard data

product suite

• Evolve with improvements

in technology

• Promote ease of use and

re-use

• Horiz: GCS NAD83 (2011)

Vert: NAVD88 (GeoidXX)

33

USACE District

Office

USACE GRiD USGS EROS

Data Center

USGS Florida

Science Center

NOAA OCM &

NCEI

By request to

jalbtcx@usace.army.mil

NATIONAL COASTAL MAPPING PROGRAM

DATA SHARING

Stennis International AirportKiln, MS

https://griduc.rsgis.erdc.dren.mil/griduc/corpsmap/ https://coast.noaa.gov/dataviewer/

ArcGIS Server and managed database on

Microsoft Azure

Map Services Feature Services

Image Services

34DATA SHARING WITH IMAGE SERVICES

STEP 1: PUSH DATA INTO THE CLOUD

Stennis International AirportKiln, MS

ArcGIS Server and managed database on

Microsoft Azure

• Products pushed to Azure include

DSMs, DEMs and true-color camera

mosaics

• Bandwidth challenges overcome with

upgrade to fiber connection

• Remote desktop limitations overcome

with utilization of an ftp server

• 5 GB of data products, comprised of

5815 files, uploaded to Azure in 1

hour

35DATA SHARING WITH IMAGE SERVICES

STEP 2: CREATE SOURCE MOSAIC DATASETS

• Products organized on Azure by

product type, then by state, then

by effort (i.e. 2010_NCMP_ME)

• SOURCE mosaic dataset

templates are pre-populated with

fields necessary to support time-

enablement and querying

Field Definition

Year Year of data acquisition

Start_Date First data acquisition day

End_Date Last data acquisition day

State Two-letter state

abbreviation

Effort Effort under which data

was collected (i.e. NCMP)

36DATA SHARING WITH IMAGE SERVICES

STEP 3: BUILD SOURCE MOSAIC DATASETS

• Workflow uses

ArcCatalog and

ArcMap, and follows

best practices in

Esri’s Image

Management

Guidebook http://esriurl.com/7169

• Use fully-qualified

path name to data

store containing

raster products, so

ArcGIS Server

machine can

access the data

store

37DATA SHARING WITH IMAGE SERVICES

STEP 4: QC & ATTRIBUTE SOURCE MDS

• Verify data coverage

and troubleshoot any

display issues related

to missing, corrupt or

stale overviews

• Populate fields based

on JALBTCX best

practices and the

Effort’s metadata

Field Value

GroupName NCMP, Emergency

Response, or District

ProductName 1mGrid or

BareEarth_1mGrid

DATA SHARING WITH IMAGE SERVICES

STEP 5: ADD SOURCE MDS TO DERIVED MDS

• Stop image service prior

to adding SOURCE

MDS to DERIVED MDS

• Utilize ‘Table’ Raster

Type

• Don’t forget to re-start

the image service once

complete ;)

38

DATA SHARING WITH IMAGE SERVICES

STEP 6: REVIEW DERIVED MDS SYMBOLOGY

• Apply the JALBTCX

standard colormap to

the derived mosaic

dataset using a raster

function template

• Developing a custom

color map for the

entire coastal U.S. is

hard!

Beach

Highest

Elevations

Deepest

Depths

39

DATA SHARING WITH IMAGE SERVICES

STEP 7: REVIEW THE IMAGE SERVICE

• Verify time-

enablement on

the image service

performs as

intended

40

NATIONAL COASTAL MAPPING PROGRAM

WEB APP – LIDAR ELEVATION COMPARISON

41

http://bit.ly/2yE0Lo7

42

https://arcg.is/1Sf90G

NATIONAL COASTAL MAPPING PROGRAM

WEB MAP – TRUE-COLOR IMAGERY VIEWER

43NATIONAL COASTAL MAPPING PROGRAM

WEB APP – BEACH VOLUME CHANGE VIEWER

44

SUMMARY

• USACE, under a National Coastal Mapping Program executed by JALBTCX,

provides high-resolution, high-accuracy lidar elevation and imagery data along

sandy shorelines of the U.S. on a recurring basis.

• Data products include point clouds, surface models, elevation models,

shoreline vectors, true-color imagery mosaics and hyperspectral imagery

mosaics. These products are available for use by Federal, State and Local

partners, Industry and the Public.

• Data products have traditionally been shared using external hard drives.

JALBTCX recently piloted the sharing of these products using Esri Image

Services in conjunction with Esri-managed Cloud Services (EMCS).

• While JALBTCX followed Esri’s guidance and best practices, early lessons

learned included the (1) acquisition of a fiber internet connection, (2) use of

fully-qualified paths when building mosaic datasets, (3) specification of custom

fields to support time-enablement and querying, and (4) application of custom

color maps with raster function templates.

• Next steps include (1) migrating the EMCS infrastructure to the DoD L2 Cloud,

(2) automating the ingestion of data products into mosaic datasets and

JALBTCX’s geospatial data inventory, (3) improving time-enablement for

performance in web maps and (4) including more datasets and additional data

products.

45

Charlene.S.Sylvester@usace.army.mil 228-252-1115 NCMP Operations

QUESTIONS?

Tim

e-lapse o

f a n

ight flig

ht, L

ong Isla

nd, N

Y,

Septe

mber

2017

For More Details: www.esriurl.com/imageryworkflows

Print Your Certificate of AttendancePrint stations located in the 140 Concourse

Tuesday Wednesday12:30 pm – 6:30 pm GIS Solutions Expo Hall B

5:00 pm – 6:30 pm GIS Solutions Expo SocialHall B

10:30 am – 5:15 pm GIS Solutions Expo Hall B

6:30 pm – 9:00 pm Networking ReceptionSmithsonian National Portrait Gallery

Download the Esri Events

app and find your eventSelect the session

you attended

Scroll down to find the

feedback section

Complete answers

and select “Submit”

Please Take Our Survey in the Esri Events App