Cyberinfrastructurefor

Astronomy

Roy Williams

US National Virtual Observatory ProjectCalifornia Institute of Technology

Toward a “new astronomy”

• Past: Observations of small, carefully selected samples

Trends

• Future dominated by detector improvements

Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in Megapixels, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels.

• Moore’s Law growth in CCD capabilities

• Gigapixel arrays on the horizon

• New Detector Technologies (e.g., STJ)

•Improvements in computing and storage will track growth in data volume

• Investment in software is critical, and growing

New astronomy: Data Federation

New Astronomy: Data Mining

• Future: Multi-wavelength data for millions of objects, allowing us to:– Discover significant patterns from the analysis of

statistically rich and unbiased image/catalog databases

– Understand complex astrophysical systems via confrontation between data and sophisticated numerical simulation

New Astronomy: New Views

• Simultaneous access to – multi-wavelength archives, – advanced visualization and – statistical analysis tools

New Astronomy: Multispectral Imagery

Crab Nebula.3 channels: X-ray in blue, optical in green, and radio in red.

Moffet Field California. 224 channels from 400 nm to 2500 nm

The Virtual Observatory

International VOhttp://ivoa.net

US VOhttp://us-vo.org

Interoperability

VizieR: Contains more than 4000 astronomical catalogues consisting of one or several tables.

“Give me all tables containing the V magnitude in the Johnson system.”

144 different names for Johnson V.

But first, what can you do now?

Science applications

– Resource discovery: Registry– Data discovery and integration: DataScope– Database queries and cross-correlation:

OpenSkyQuery– Spectrum browsing and analysis: Spectrum

Services– Source extraction / cross-correlation: WESEX,

WCS fixer– Sky coverage of catalogs: Catalog Coverage

Maps– On-demand mosaicking: Mosaic Service

NVO Registry Portal

Find source catalogs, image archives, and other astronomical resources registered with the NVO

A Registry is a distributed database of Virtual Observatory resources: primarily access services for catalog, image, and spectral data, but also descriptions of organizations and data collections. There are several coordinated registry implementations that share information by harvesting each other's resources. This registry is at STScI in Baltimore, MD.Searches for resources can be done by keyword, or advanced queries can be expressed in the SQL language. The registry is open for humans through web forms, or machines through SOAP web services.

DataScope

Using the NVO DataScope scientists can discover and explore hundreds of data resources available in the Virtual Observatory. DataScope uses the VO registry and VO access protocols to link to archives and catalogs around the world. Users can immediately discover what is known about a given region of the sky: they can view survey images from the radio through the X-ray, explore archived observations from multiple archives, find recent articles describing analysis of data in the region,

Discover and explore data in the Virtual Observatory

find known interesting or peculiar objects and survey datasets that cover the region. A summary page provides a quick précis of all of the available data. Users can download images and tables for further analysis on their local machines, or they can go directly to a growing set of VO enabled analysis tools, including Aladin, OASIS, VOPlot and VOStat.

The “view-data” link goes directly to DataScope.

See the event context from all major surveys.

See archival images corresponding to the latest gamma-ray bursts

DataScope with VOEvent

The results have already been collected from across the VO as soon as the GRB happens.

OpenSkyQuery

Cross-match your data with numerous catalogs

OpenSkyQuery allows you to cross-match astronomical catalogs and select subsets of catalogs with a general and powerful query language. You can also import a personal catalog of objects and cross-match it against selected databases.

Spectrum Services

Search, plot, and retrieve SDSS, 2dF, and other spectraThe Spectrum Services web site is dedicated to spectrum related VO services. On this site you will find tools and tutorials on how to access close to 500,000 spectra from the Sloan Digital Sky Survey (SDSS DR1) and the 2 degree Field redshift survey (2dFGRS). The services are open to everyone to publish their own spectra in the same framework. Reading the tutorials on XML Web Services, you can learn how to integrate the 45 GB spectrum and passband database with your programs with few lines of code.

Web Enabled Source Identification with Cross-Matching (WESIX)

Upload images to SExtractor and cross-correlate the objects found with selected survey catalogs.

This NVO service does source extraction and cross-matching for any astrometric FITS image. The user uploads a FITS image, and the remote service runs the SExtractor software for source extraction. The resulting catalog can be cross-matched with any of several major surveys, and the results returned as a VOTable. The web page also allows use of Aladin or VOPlot to visualize results.

An essential part of an astronomical image is the mapping between pixels and sky — the so-called WCS information. This NVO service takes an image with approximate WCS and computes the exact WCS, using major catalogs such as USNO-B and SDSS.

USNO-B1USNO-A2SDSSNOMAD

Input image

• Source detect• Match to catalog• Fit parameters

Exact WCS

Client Remote NVO Service

Refine an image with approximate pixel mapping to the exact WCS solution

WCS Fixer

Catalogue of Stellar Spectral Classifications (Skiff, 2005)

Find catalogs and source density for the position or object name of interest

An all-sky map showing distribution of sources across the sky. Dynamically generate a custom map for specific region and selection of catalogs. Composite all-sky overlay maps show which regions are covered by all of the selected catalogs.

Catalog Coverag Maps

Try it out!

• Start at http://us-vo.org• Develop your own VO-enabled software

with the NVO Summer School software kit

• For developers: come to the 2006 NVO Summer School in Aspen

DataScope

NVO: Spectrum Services

OpenSkyQuery

VOPlot

Under the hood…

… standards

Regis

try

Vie

ws

Serv

ices

View:Client’s Frame of Reference

File/Blob TableMIME type

SemanticsFITS

XMLS

pect

ral L

ines

Sour

ce C

atal

og

Spec

trum

Imag

e

CharacterizationR

egio

nE

vent

Dat

acub

e

Services, Filesystems, VOSpace

Space/Time

Portals, Applications, Interfaces

VOTable

Edu

cati

on

Service: Interacting with remote resources

Grid& Globus

Protocols (SOAP, REST, GET)Accounts & Security

AsyncC

one

Imag

e

Certificates

Dat

acub

e

WWW

Spe

ctru

m

Cat

alog

Reg

istr

y

VOSpace

Cus

tom

....

Contracts (WSDL)

Service Requests

Query Language

Files and Databases

internet

Example Web Service

name= M51x= 10y= 10sky_survey=DSS2_redmime_type=download_gif

http

://arc

hiv

e.e

so.o

rg/d

ss/d

ss/im

ag

eClient interface

Request: Keyword/value

Response: image/gif

Serviceendpoint

Web Services

• Remote execution

• Structures in and out

• Self-defined (WSDL)

• Click or code

Mosaicking enginefrom services

Logical SIAP

NVO Registry

Physical SIAP

Computing

Portal

Trust

Request:survey,region,[login]

Sandbox

web access

Image Data

Service

Registry:Publish, Find & Use

IVO identifiers

Con

e

Org

aniz

atio

n

Eve

nt p

ubli

sher

Harvest

Prov

enan

ce

Imag

e

Com

pute

Ser

vice

Registry Curation

Spec

trum

App

lica

tion

Etc

etc

....

Service

Cus

tom

....

Skyn

ode

Registryof

Registries

Distributed Registry

Caltech

NCSA

STScI/JHU

HEASARC

Astrogrid

CDS

JapanVO

Ongoing harvesting May 05

ESO

CfA

NOAO

DataScope

• Federates multiple cone, SIAP services

Registry

OAIPublish

Registry

OAI QueryRegistry

OAIPublish

DataScope

1

2

3

4Caltech

NCSA

JHU/StSci

Goddard

Palomar-QuestA repeating sky surveyCaltech-Yale

Palomar-Quest SurveyDjorgovski, Baltey, Drake, Graham, Mahabal, Williams

P48 Telescope

Caltech Yale

NCSA

Transient pipeline computing reservation at sunrise for immediate followup of transients

Synoptic survey massive resampling for ultrafaint detection of hi-Z quasars

NCSA and Caltech and Yale run different pipelines on the same data

50 Gbyte/night ALERT

Griffith Observatory“big picture”

158 feet

Show tile b14

158 feet

158 feet

Secret Sauce is Software

VOEventInstant Messaging for Astronomy

• Automated architecture• Structured packets• Trust

1604 image of Cas A

Transient Sources in the Sky

• Asteroids• Variables and binaries• Supernova• Gamma-ray burst• Lensing events• Gravitational waves• etc etc and:

• NOT YET THOUGHT OF ....?

• http://voeventnet.caltech.edu/transients/

Gamma-ray Burst Satellite Feed

subscribersubscriber

author/publisher

socket

subscriber

emailtelephone

Single provider Structured, for machine understandingFixed structuresStandard transports

Caltech

Exeter, UK

Los Alamos

Tucson

SDSS SNe U Washington/Stanford

PairitelBerkeley

Palomar P60Caltech

Palomar-Quest Caltech

RaptorLANL

Liverpool TelescopeLa Palma

UKIRTHawaii

AuthorPublisherRepositoryRelaySubscriber

FaulkesHawaii/Australia

OGLE IIIPoland

GCNNASA/GSFC

Microlensing Survey Exeter

CTIO/KPNO surveys

Sources of Events

• GCN, (Swift, Integral, etc)• Palomar-Quest (Caltech), Raptor (LANL),

etc• Supernova factories• Variable star observers (AAVSO)• non-photon:

– LIGO, neutrinos, cosmic rays, etc

• Future– PannStarrs, LSST

VOEvent Structure• Who

• Publisher, Contact, etc• Subscribers will use PublisherID to select

• WhereWhen (== STC)• Can be simple eg RA, Dec, eg UTC• Can be sophisticated, eg multiple frames, near objects, etc

• What• Hierarchy of named parameters• Units, Semantic type (UCD)• References, Descriptions

• How• How was the evidence gathered: camera, telescope, etc

• Why• probability list of interpretation

– supernova, comet, asteroid, .....

• Citation• Link to other VOEvent: Followup, Supercede, Retraction, • Link to support data

Google-like interfaces (coming)

• Prototype pan/zoom at Pittsburgh

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Questions?