us-vo.org/vim vim visual integration and mining a tool to explore your sources roy williams (...
TRANSCRIPT
![Page 1: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/1.jpg)
us-vo.org/vim
VIMVisual Integration and MiningA Tool to Explore Your Sources
Roy Williams (Caltech)And the NVO Portal Team
THE US NATIONAL VIRTUAL OBSERVATORY
Part of the NVO Portal
![Page 2: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/2.jpg)
us-vo.org/vim
Vim is anNVO Portal Component
Sources and Resources
Find sources (== sky positions)Find resources (== catalog services)Look up sources with resources
![Page 3: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/3.jpg)
us-vo.org/vim
Start with your sources(currently <1000)
• Multiple named sky positions– Simple text (RA,Dec,ID)– Text wizard (NVO table importer, today 17.10)
– VOTable of positions (local or url)– Push positions from NVO portal
![Page 4: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/4.jpg)
us-vo.org/vim
Got Text Table?
Make a VOTable from plain text using the table wizard.
http://us-vo.org/tablewizard
![Page 5: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/5.jpg)
us-vo.org/vim
Getting Data about your sources
![Page 6: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/6.jpg)
us-vo.org/vim
hover mouse to enlarge
Skyview cutouts in many wavelengths
Cen
A @
840
MH
z
![Page 7: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/7.jpg)
us-vo.org/vim
Skyview cutouts in many wavelengths
Abell clusters in optical and X-ray
![Page 8: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/8.jpg)
us-vo.org/vim
Spectral Collections from JHU brokered by NRAO:
• Arecibo Maser Catalog
• 2dFGRS
• SDSS DR5
Hover mouse on thumbnail to see larger image
Spectra from VO SSAP services
![Page 9: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/9.jpg)
us-vo.org/vim
Getting Data about your sources
Proximity search
AJAX progress
![Page 10: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/10.jpg)
us-vo.org/vim
MulticoneProximity Search =user-supplied sources + radius
returns VOTable
radius
![Page 11: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/11.jpg)
us-vo.org/vim
VIM tables structure– Multiple matches per source– Can join to sources (outer or nearest)
RA Dec ID
sources Cat1 Cat2
![Page 12: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/12.jpg)
us-vo.org/vim
Shakbazian galaxy groupsprox search in Simbad
![Page 13: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/13.jpg)
us-vo.org/vim
Table column metadata
NED metadata
![Page 14: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/14.jpg)
us-vo.org/vim
![Page 15: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/15.jpg)
us-vo.org/vim
Compute new columns, eg joint IR-optical colors
Choose which columns to
display
![Page 16: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/16.jpg)
us-vo.org/vim
Visualization by download & use Topcat
![Page 17: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/17.jpg)
us-vo.org/vim
Architecture
browser Vim
personal persistent storage
upload sourcesHTML + AJAX
Catalogs (cone)Images (Skyview)Spectra (SSAP)
All the relevant information about your sources-- mashups from the VO-- kept for you in persistent storage-- view, mine, download
python api
Registry
Inventory
NVO portal
VOservices
![Page 18: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/18.jpg)
us-vo.org/vim
• Auto-script:• Clicking builds script• Can edit script and rerun
![Page 19: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/19.jpg)
us-vo.org/vim
# got some sdss sources from Vizier, a 12 minute radius with rmag<21# about 773 sources foundsources.run(bench, open(‘mysources.votable').read())
# run 2mass prox search at 2 arc secondsproxsearch.run(bench, survey='2MASS-PSC', sr=0.00055556, tablename='_2mass', verb='1', maxthreads='10')
# merge with the sdssjoin.run(bench, tablename='_2mass', match='best')
# select rows with non-null j magnitude (i.e. thos in 2mass)# we get abouyt 148 remainingselectrows.run(bench, tablename='sources', criterion='! NULL__2mass_j_m')
# compute 2mass-J - SDSS-raddcol.run(bench, tablename='sources', colname='JminusR', expression='_2mass_j_m - rmag')
# compute 2mass-H - SDSS-zaddcol.run(bench, tablename='sources', colname='Hminusz', expression='_2mass_h_m - zmag')
# then download sources.votable from Utility/ViewAllFiles, and plot with topcat
Vim auto-scripting
![Page 20: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/20.jpg)
us-vo.org/vim
• WebServer or Personal install• Pure python webserver (CherryPy)
• Scalable• Auto-script, edit, rerun paradigm• Powered by Stilts (2,000,000,000 rows !)
• Simple sharing• Share your workbench by emailing a URL
http://envoy5.cacr.caltech.edu:8888/?benchID=7462948574635185948473627485958433
• Can write-protect with password
VIM Features
![Page 21: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/21.jpg)
us-vo.org/vim
• Content– Any cone search (all 1000’s in VOregistry)
– Cutouts from Skyview– NVO Inventory services– Spectra via SSAP (from NRAO)
• Thumbnails and images and FITS
• Display– Column selection, Row sort/select– Images small-hover-large– Tools and metadata choice– AJAX progress for data fetching …but not Safari :-(
VIM Features
![Page 22: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/22.jpg)
us-vo.org/vim
NVO Portal:Using the registry to get catalogs
Either:START at Vim, Catalogs / Find Catalog
Services (your workbench remembered)Or:Collect Ivorns from registry and Vim can resolve them Catalogs / Add Catalog
by Ivorn,Or: choose “Export results to VIM” from another portal component
![Page 23: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/23.jpg)
us-vo.org/vim
NVO Portal:Finding catalogs by spatial correlationInput catalog is Brightest BL Lac (38)
NVO Portal:Finding catalogs by spatial correlationInput catalog is Brightest BL Lac (38)
Science Question:Are BL Lac associated with GRB?
Science Question:Are BL Lac associated with GRB?
![Page 24: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/24.jpg)
us-vo.org/vim
my sources
Catalog A:Perfect correlation
Catalog C:No correlation
Catalog B:Weak correlation
NVO Portal:Inventory service
(1000’s of catalogs ingested)
![Page 25: Us-vo.org/vim VIM Visual Integration and Mining A Tool to Explore Your Sources Roy Williams ( Caltech) And the NVO Portal Team T HE US N ATIONAL V IRTUAL](https://reader035.vdocuments.net/reader035/viewer/2022070307/551b88c7550346d6338b5ab4/html5/thumbnails/25.jpg)
us-vo.org/vim
Questions?
VIM is pure python under CherryPy serverVIM relies on Stilts and Skyview and YahooUI and NVO Portal
Thank you BobGretchenJohnMarkMikeTom et al!