neon obs school 11-aug-2005 archival data and virtual observatories 1 virtual observatories...or how...

NEON Obs School11-Aug-2005 Archival Data and Virtual Observatories 1

Virtual Observatories

...or how to do your research from a beach in the Bahamas rather than stuck up a cold

mountain in the middle of the night...


Who I amJohn Taylor: [email protected] on the AstroGrid and VOTech Virtual Observatory projectsMember of Edinburgh's Wide Field Astronomy Unit – curates catalogue and image data from SuperCOSMOS and (soon) WFCAM and (less soon) VISTASoftware engineer NOT Astronomer


OutlineThe data deluge Why science is changing The scale of the problem and technical issues

Catalogues How to use them VOTables, FITS files Visualisation tools

The Virtual Observatory AstroGrid Demonstration


I: The astronomer's world is changing...

Previously an Astronomer was a “radio astronomer” or a “x-ray astronomer”. Now you need to be an expert in all wavelengths.The move towards Survey AstronomyFederating data from different instruments.Vast quantities of auto-generated dataSee: http://grist.caltech.edu/sc4devo/What_is_the_VO.pdf


Supernova Remnant Cassiopeia-A – a 300

year old Supernova

Shocks seen in the X-ray Chandra image

Heavy elementsseen in the optical

Dust shows in the IR

Images from Chandra Science CentreSlide: Nic Walton, AstroGrid Project Scientist

Mapping e¯s in the magnetic field as revealed by Radio data


Data Deluge

(Ex-)Concorde(15 km)

CD stack with1 year LHC data(~ 20 km)

SuperCOSMOS: 2.5 Tb

WFCAM: 14Tb per yearVISTA: 4Tb per weekPhysicists, Astronomers, Biologists all facing the same problemSteve Lloyd: http://www.gridpp.ac.uk/talks/Data_Deluge_Brunel_2005.ppt


Nearby Spiral Galaxy

x-ray optical radio


II: Catalogue and Image ServersCone searchesData formats: FITS and VOTablesVisualising the data: Topcat and AladinMore sophisticated searches using SQLDemo of the SuperCOSMOS Science Archive


Example: SuperCosmos Science Archive

Scans of the UK Schmidt telescope platesCatalogue data & Image data3 wavebands, 2 epochsOver 1 billion objects2.5 terabytes in size


Data FormatsFITS – the astronomer's jpeg images/spectra/tables Includes metadata

VOTable “xml” format for

tables The VO standard

UCDs describe what a

quantity “is”


Visualising the dataTOPCAT For viewing and

manipulating tables, especially VOTables

Create new columns Filter the sources Output the table in

different formats

Aladin Good for viewing

image files Overlay catalogue

tables Create RGB images Access data archives

such as Vizier Aims to be a general

“Virtual Observatory” portal


Complex queries: SQLSQL is a general language for querying databases

select ra,dec,mag FROM sources where ra>0 and ra<1 and dec>-45 and dec<-44 and mag>23

give me ra, dec and mag values(use * to get everything)

table name(each db will have several) the following

conditions are true


SQL IIPlenty of resources on the web for learning SQL e.g. http://www.w3schools.com/sql/default.aspThere is an astronomy variant of SQL called ADQL (astronomy data query language)It's basically SQL with a few extra bits thrown in (a CIRCLE keyword for doing cone searches, and an XMATCH keyword for cross-matching data between databases)[Cross Matching: Taking two or more different database tables from (maybe) different sources and joining them together based on their RA and DEC]


Some databasesSIMBAD - Basic data on objectshttp://simbad.u-strasbg.fr/Simbad SuperCOSMOS Science Archive http://surveys.roe.ac.uk/ssa/ SDSS (Sloan Digital Sky Survey) http://www.sdss.org/2mass (2 micron all sky survey) http://www.ipac.caltech.edu/2mass/XMM http://xmm.vilspa.esa.es/external/xmm_data_acc/xsa/index.shtml


Databases of databasesVizier http://vizier.u-strasbg.fr/viz-bin/VizieRDatascope http://heasarc.gsfc.nasa.gov/cgi-bin/vo/datascope/init.plNED (Nasa Extragalactic Database) http://nedwww.ipac.caltech.edu/OpenSkyQuery http://openskyquery.net/Sky/skysite/


This presentation will be available at:

http://wiki.astrogrid.org/pub/Main/JohnTaylor/neon.sxi


A recap on yesterdayIs the SSA now behaving?Last night's observing group: how to use SQL to get your list of galaxies from SDSS

select g.ra,g.dec,g.objId, s.z from Galaxy g, ELRedShift s where g.specObjID=s.specObjID and s.z>0 and s.z<0.01 and g.r<10 and g.ra>0 and .........


III: Virtual Observatories :

What's a virtual observatory?What do you need to make one?AstroGrid – glue for the Euro-VODemo


The International Virtual Observatory


Virtual Observatory“The VO vision can be summed up as the desire to make all archives speak the same language − all searchable and analysable by the same tools, all data sources accessible through a common interface, all data held in distributed databases that appear as one.”Andy Lawrence, 09/2003

Take all the world's Astronomical data, and all the world's applications and computing power and make it appear that they reside on the user's desktop.


Science Cases – Brown Dwarf Location

Brown Dwarfs are cool, faint and (if can see them) closeRoughly speaking: Query optical and infrared catalogues for objects

that are “bright” in the infra red, but not in the optical and have large proper motions.

Requires use of more than one catalogueMore: “The AstroGrid 10” http://wiki.astrogrid.org/bin/view/Astrogrid/ScienceProblems


AstroGrid Components

PAL

PALPAL

CEA CEA


Example workflow: calculating photometric redshifts


AstroGrid Components


Demo


Contacts and info

Contacts: John Taylor [email protected] Institute for Astronomy,

Royal Observatoy of Edinburgh

Background Info on AstroGrid http://www.astrogrid.org

Download the Euro-VO AstroGrid software from: http://software.astrogrid.org


What now?

AstroGrid Release 1 is up and running now

http://www.astrogrid.org/release

http://wiki.astrogrid.org/bin/view/Astrogrid/ReleaseV1Pages


Locating data and applications

IVO-standard registryRegistries harvest from each otherXml-based query language ADQL


WorkflowOrchestrate servicesQueries and workflows are designed using the portal – other clients on the wayWork is run remotely and asynchronously Archives searched and results manipulatedResults are stored in a virtual file system Queries and workflows can be re-used and shared

A PPARC funded project

AstroGrid Intro & Demo

John TaylorInstitute for Astronomy,

Edinburgh


Building a VO - Layer Diagram

neon obs school 11-aug-2005 archival data and virtual observatories 1 virtual observatories...or how...

Documents

edinburghneon obs school11

virtual observatories

archival data

vo layer diagram