introduction to observational catalogues & databases duncan law-green (ledas archive scientist)...

Introduction to Observational Catalogues & Databases

Duncan Law-Green(LEDAS Archive Scientist)

21st February 2007

IntroductionIntroduction

What this seminar is aboutWhat this seminar is aboutIntroduction to selection of online astronomical Introduction to selection of online astronomical data resources to help with your research. data resources to help with your research. Outline of data formats and catalogue tools. Outline of data formats and catalogue tools. Practical demos.Practical demos.

What this seminar is not aboutWhat this seminar is not aboutData reduction and analysis techniques.Data reduction and analysis techniques.Computer science of databases, SQL Computer science of databases, SQL programming, how to build your own databases.programming, how to build your own databases.

Introduction to Observational Catalogues & Databases: 21/02/2007

Use of Catalogues/ArchivesUse of Catalogues/Archives Literature searchesLiterature searches

Build science case for your proposalBuild science case for your proposal Archive retrievalArchive retrieval

Previous observations of your objectsPrevious observations of your objects Feasibility studiesFeasibility studies

Is the observation possible with your instrument?Is the observation possible with your instrument? Observation planningObservation planning

Source positions, instrument FOV, guide stars etc.Source positions, instrument FOV, guide stars etc. Class studiesClass studies

Filter catalogues for interesting sources/outliersFilter catalogues for interesting sources/outliers Cross-correlationCross-correlation

Compare source positions in different cataloguesCompare source positions in different catalogues

Data & MetadataData & Metadata

DataDataA set of measured parameters for a source.A set of measured parameters for a source.

MetadataMetadata Data Data aboutabout the dataset, provides vital context info. the dataset, provides vital context info.

Examples include: coordinate epoch, date of Examples include: coordinate epoch, date of observations, observing mode, filter bandpass, observations, observing mode, filter bandpass, pipeline processing performed etc.pipeline processing performed etc.

Data without metadata is Data without metadata is uselessuseless..Some data formats preserve metadata, some don't.Some data formats preserve metadata, some don't.


Data Formats IData Formats I

ASCII ASCII (CSV/TSV: Comma/Tab-Separated Variables)(CSV/TSV: Comma/Tab-Separated Variables)

Advantages:Advantages: Easy to generate, easy to read. Easy to generate, easy to read. Will ingest directly into, e.g. ExcelWill ingest directly into, e.g. Excel

Disadvantages:Disadvantages: No direct metadata support. No direct metadata support. No integrity support. No integrity support. No documentation. No documentation. Bulky.Bulky.

Data Formats IIData Formats II

FITS (NOAO), HDS, NDF (Starlink)FITS (NOAO), HDS, NDF (Starlink)

Advantages:Advantages: Structured formats, include metadata. Structured formats, include metadata. Multi-dimensional (tables, images, datacubes). Multi-dimensional (tables, images, datacubes). Well-defined formats, good software support. Well-defined formats, good software support. Binary format, compact.Binary format, compact.

Disadvantages:Disadvantages: Varying compatibility between FITS, HDS, NDF.Varying compatibility between FITS, HDS, NDF.Conversion may affect metadata.Conversion may affect metadata.No semantics.No semantics.

Data Formats IIIData Formats III

VOTable (IVOA, AstroGrid)VOTable (IVOA, AstroGrid)

Advantages:Advantages: Structured format, metadata+Structured format, metadata+semanticssemantics. . Human-readable, supported by modern software. Human-readable, supported by modern software. Uses XML, existing tools to generate and check Uses XML, existing tools to generate and check integrity.integrity.

Disadvantages:Disadvantages: Uses XML – very bulky. Uses XML – very bulky. Multidimensional support awkward. Multidimensional support awkward. Standards evolving.Standards evolving.

SemanticsSemantics

What a data column means in physical termsWhat a data column means in physical terms Unified Content Descriptors (UCDs)Unified Content Descriptors (UCDs)

Created by IVOA (International Virtual Observatory Created by IVOA (International Virtual Observatory Alliance) as standard controlled vocabulary of Alliance) as standard controlled vocabulary of keywords to describe physical nature of table keywords to describe physical nature of table columns. Current system “UCD1+”columns. Current system “UCD1+”

R.A. (Main):R.A. (Main): pos.eq.ra;meta.main pos.eq.ra;meta.main Source ID:Source ID: meta.id;srcmeta.id;srcRadio flux ratio:Radio flux ratio: phot.flux;em.radio;arith.ratiophot.flux;em.radio;arith.ratio

UCDs are feature of well-constructed VOTables. UCDs are feature of well-constructed VOTables. Intended to ease automated data handling, “workflows”Intended to ease automated data handling, “workflows”

VOTableVOTable<?xml version="1.0"?><VOTABLE version="1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://www.ivoa.net/xml/VOTable/VOTable/v1.1"> <COOSYS ID="J2000" equinox="J2000." epoch="J2000." system="eq_FK5"/> <RESOURCE name="myFavouriteGalaxies"> <TABLE name="results"> <DESCRIPTION>Velocities and Distance estimations</DESCRIPTION> <PARAM name="Telescope" datatype="float" ucd="phys.size;instr.tel" unit="m" value="3.6"/> <FIELD name="RA" ID="col1" ucd="pos.eq.ra;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Dec" ID="col2" "pos.eq.dec;meta.main" ref="J2000" datatype="float" width="6" precision="2" unit="deg"/> <FIELD name="Name" ID="col3" ucd="meta.id;meta.main" datatype="char" arraysize="8*"/> <FIELD name="RVel" ID="col4" ucd="src.veloc.hc" datatype="int" width="5" unit="km/s"/> <FIELD name="e_RVel" ID="col5" ucd="stat.error;src.veloc.hc" datatype="int" width="3" unit="km/s"/> <FIELD name="R" ID="col6" ucd="phys.distance" datatype="float" width="4" precision="1" unit="Mpc"> <DESCRIPTION>Distance of Galaxy, assuming H=75km/s/Mpc</DESCRIPTION> </FIELD>

continued on next slide...

VOTable (cont.)VOTable (cont.)

<DATA> <TABLEDATA> <TR> <TD>010.68</TD><TD>+41.27</TD><TD>N 224</TD><TD>-297</TD><TD>5</TD><TD>0.7</TD> </TR> <TR> <TD>287.43</TD><TD>-63.85</TD><TD>N 6744</TD><TD>839</TD><TD>6</TD><TD>10.4</TD> </TR> <TR> <TD>023.48</TD><TD>+30.66</TD><TD>N 598</TD><TD>-182</TD><TD>3</TD><TD>0.7</TD> </TR> </TABLEDATA> </DATA> </TABLE> </RESOURCE></VOTABLE>

continued from previous slide...

TreeviewTreeview File format viewerFile format viewer

Can read multiple file formats, display hierarchical Can read multiple file formats, display hierarchical structures, expand and collapse nodes with click of structures, expand and collapse nodes with click of mouse. Some basic plotting, image, stats routines.mouse. Some basic plotting, image, stats routines.

Literature SearchLiterature Search ADS: NASA Astrophysics Data ServiceADS: NASA Astrophysics Data Service

3 bibliographic databases3 bibliographic databasesAstronomy & AstrophysicsAstronomy & Astrophysics (1.2 million) (1.2 million)PhysicsPhysics (3.6 million) (3.6 million)ArXiv e-printsArXiv e-prints (400,000) (400,000)

Searchable by author, subject, title, object, Searchable by author, subject, title, object, abstract text, full-text...abstract text, full-text...

Links to full PDFs of articles, object catalogues, Links to full PDFs of articles, object catalogues, data tables.data tables.

MyADS Update Service: subscribe to updatesMyADS Update Service: subscribe to updates

Not peer-reviewed!

adsabs.harvard.edu

““Telegrams”Telegrams”

IAU Circulars: Central Bureau for IAU Circulars: Central Bureau for Astronomical TelegramsAstronomical TelegramsCentral clearinghouse for info on transient events Central clearinghouse for info on transient events (comets, solar system bodies, novae, supernovae (comets, solar system bodies, novae, supernovae etc.). Subscribe by email. etc.). Subscribe by email. Search via ADSSearch via ADS

Astronomers' TelegramAstronomers' TelegramPrimarily high-energy transient events (GRBs etc.). Primarily high-energy transient events (GRBs etc.). Subscribe via email or RSS. Searchable web Subscribe via email or RSS. Searchable web interface, mirror @ LEDASinterface, mirror @ LEDAS

Data ServersData Servers Catalogue ServersCatalogue Servers

Surveys and article data. Searchable by position, filter by Surveys and article data. Searchable by position, filter by various parameters, output data in ASCII, FITS, VOTable. various parameters, output data in ASCII, FITS, VOTable. (examples: NED, HEASARC, LEDAS, ViZieR)(examples: NED, HEASARC, LEDAS, ViZieR)

Image ServersImage ServersImages of the sky at various wavelengthsImages of the sky at various wavelengths.. Output data in Output data in bitmap (GIF,JPG,PNG) or FITS image. May or may not be bitmap (GIF,JPG,PNG) or FITS image. May or may not be “science grade”. (examples: DSS-I/II, SDSS, SkyView, Aladin)“science grade”. (examples: DSS-I/II, SDSS, SkyView, Aladin)

Archive ServersArchive ServersRepository of public data from particular observatory or Repository of public data from particular observatory or mission. Various formats, may need specialist software or mission. Various formats, may need specialist software or training to interpret. (examples: Hubble, MERLIN, Chandra, training to interpret. (examples: Hubble, MERLIN, Chandra, LEDAS).LEDAS).

AladinAladin Advanced image and catalogue search systemAdvanced image and catalogue search system

Plot catalogue search results directly on imagePlot catalogue search results directly on image

Highly versatile, write scripts for repetitive Highly versatile, write scripts for repetitive operations.operations.

Launch via CDS website, LEDAS site or directly on Launch via CDS website, LEDAS site or directly on desktop.desktop.

RTFM!RTFM!


AladinAladin


Plane stack

Object data

Preview

Button bar

Main window

AladinAladin


Multiview option

Splitscreen button

TOPCATTOPCAT Catalogue plotting, editing and filtering toolCatalogue plotting, editing and filtering tool Cross-correlations between cataloguesCross-correlations between catalogues


TOPCATTOPCAT

Cross-correlationCross-correlation

Search for matching positions in 2 or more Search for matching positions in 2 or more catalogues (e.g. “does this X-ray source have a catalogues (e.g. “does this X-ray source have a radio counterpart?”)radio counterpart?”)

Consider positional uncertainties, statistical Consider positional uncertainties, statistical probability of chance coincidenceprobability of chance coincidence

Convenient tool for 2, 3, 4-way catalogue Convenient tool for 2, 3, 4-way catalogue matches in TOPCAT (Joins -> Pair Match etc.)matches in TOPCAT (Joins -> Pair Match etc.)


Virtual Observatory (VO)Virtual Observatory (VO)

International project to simplify access to International project to simplify access to astronomical catalogues and archives. astronomical catalogues and archives. Coordinated by IVOA.Coordinated by IVOA.

Standard set of access commands (“protocols”), Standard set of access commands (“protocols”), all databases “appear” the same on the network.all databases “appear” the same on the network.

UK VO project UK VO project AstroGridAstroGrid, developed additional , developed additional software for distributed search, distributed software for distributed search, distributed storage, workflows etc.storage, workflows etc.


AstroGrid WorkbenchAstroGrid Workbench Workbench: Desktop Java application for VO Workbench: Desktop Java application for VO

searches and data processing. searches and data processing.

Check availability of dataCheck availability of data

Execute simultaneous searches across multiple Execute simultaneous searches across multiple catalogues/serverscatalogues/servers

Construct Construct “workflows”“workflows”: drag+drop editing of : drag+drop editing of data gathering/reduction/analysis pipelines.data gathering/reduction/analysis pipelines.

Save results to “MySpace”, temp scratch spaceSave results to “MySpace”, temp scratch space


VOPlotVOPlot

Product of VO India projectProduct of VO India project

Reading, interactive plotting and filtering tool for Reading, interactive plotting and filtering tool for catalogue data (primarily VOTables)catalogue data (primarily VOTables)


And finally...And finally...

Stellarium (Stellarium (http://www.stellarium.org/http://www.stellarium.org/))Free planetarium software, impress your friends!Free planetarium software, impress your friends!

..

..

..

Questions to Questions to [email protected]@star.le.ac.uk

Seminar slides, URLs to appear on my webspaceSeminar slides, URLs to appear on my webspacehttp://www.star.le.ac.uk/~dlg/http://www.star.le.ac.uk/~dlg/


introduction to observational catalogues & databases duncan law-green (ledas archive scientist)...

Documents

metadata data

data column

data reduction

data formats iiivotable

data formats iascii

data formats iifits

structured formats

automated data handling