livermore labs - from video to knowledge
Post on 03-Apr-2018
225 Views
Preview:
TRANSCRIPT
-
7/28/2019 Livermore Labs - From Video to Knowledge
1/8
4 Lawrence Livermore National Laboratory
S&TR April/May
-
7/28/2019 Livermore Labs - From Video to Knowledge
2/8
5Lawrence Livermore National Laboratory
&TR April/May 2011 Persistent Surveillance
Livermores Persistics
data-processing pipeline
combines graphics-based
computer hardware
and clever software to
extract meaning from
wide-area overhead
surveillance video.
Unmanned ril vhicls, such sth ros Prtor Rpr, rrwritig th ruls of bttl i Irq
afghist r big cosir for
oitorig U.S. borrs. Oprt by
rot cotrol, sotis thouss of
kilotrs fro th bttlfil, th ircrft
us ultipl crs to trck stroy
th y.
Ths ros c fly svrl kilotrs
high, hovr ovr trgts for 24 hours
logr, r oft r with bobs
ir-to-grou issils. By fiig,
trckig, oitorig popl, vhicls,
vts of itrst o cotiuous
bsis with wi-r vio crs, th
ircrft provi prsistt itlligc,
survillc, rcoissc (ISR).
I thory, prsistt ISR covrg c
prvt is fro vig ovrh
survillc systs, thrby bligfst cisio kig for tcticl issios,
with ruc risk to U.S. forcs
ocobtts.
a w Livror coputtiol syst
is sig to hlp th dprtt of
dfs othr gcis oitor ts
of squr kilotrs of trri fro th
skis, with sufficitly high rsolutio
for trckig popl vhicls for
y hours t ti. Th syst, cll
Prsistics, proiss to ovrco svr
growig probl: th ovrwhlig
volu of vio t grt by
or ovrh igry. militry
wi-r vio survillc crs
such s Costt Hwk agl Fir
r collctig vr-icrsig outs
of vlubl vio igry ovr lrg
gogrphic rs. Howvr, th tchology
to proptly procss, stor, xtrct
ig fro t, th trsit this
ifortio ovr log istcs for furthr
lysis, hs lgg fr bhi.
Data Overwhelm Capabilities
Ssig cpbilitis for wi-r
vio rcoissc hv volv to
fr xc our wilst xpcttios,
sys physicistShil Viy, priciplivstigtor for Prsistics puty
progr irctor of dfs i
Livrors Globl Scurity Pricipl
dirctort. Thr xists orous
growiggp btw th out
of ifortio cotiuously collct
by xistig ssors our bility to
quickly tk vtg of tht ifortio
Unmanned aerial vehicles, such as the Predator shown
here, use multiple cameras for persistent surveillance.
(Courtesy of Department of Defense.)
-
7/28/2019 Livermore Labs - From Video to Knowledge
3/8
S&TR April/May Persistent Surveillance
Th Prsistics t-procssig pipli
uss tchiqu cll pixl-lvl s
ig corrspoc to stbiliz vio,
liit prllx (th slight iffrcs i
th pprt positios of objcts viw fro
iffrt crs), coprss bckgrou,
provi uprct subpixl
rsolutio of ovig objcts of itrst.
ds ig corrspoc coprs th
sc fro o pixl to othr corrctsfor iiviul iffrcs.
Inside Persistics
avc ccurt lysis tchiqus
r urgtly by th popl who
r tsk with ctgorizig, ixig,
ottig, rwig coclusios fro
th ptbyts of t collct ily ithtr oprtios.
Coputr scitist mrk duchiu,
o of Prsistics ky rchitcts, sys,
Th t-procssig ifrstructur for
tiol scurity is ot sig for th
outs typs of t big grt
by u ril ros rltiv to
th scl of hu rsourcs vilbl to
lyz th through covtiol VCR
plybck styl of viwig. I itio,
th couictio bwith supportig
t trsissio fro ir to grou
th rchiv storg cpbility r uchtoo slow or too sll to support fst-
turrou hu lyss.
duchiu sys, Svrl yrs
go, dprtt of dfs grs
prst us with this itrstig
procssig chllg. Th Livror
scitists kw thy to work
o th softwr si to gt th ost
ifortio out of th strig vio
igry. Th grtst opportuity for
hlpig tckl this chllg ly i th
vlopt of vc lgoriths (short
coputr progrs) iplt o w
coputr rchitcturs.
Prsistics, th prouct of svrl
yrs ffort, is iovtiv t-
procssig pipli tht tks
riclly iffrt pproch to
rssig th vio-t ovrlo
chllg. (S th box t right.)
Th tchiqu rtis th lvl
of til cssry for tctig
olis whil t th s ti
coprssig th uchgig
bckgrou vrythigi otio by bout 1,000 tis
without losig prtit
ifortio. as such, th pproch
liorts th rth of couictio
bwith for trsportig vio without
losig ig fility. I, Prsistics
6 Lawrence Livermore National Laboratory
If w kow prcisly how y two v
igs rlt to o othr, w c r
prllx tri wht is bckgrou
wht is ovig, icrs sigl-to-o
rtio, icrs rsolutio to s ti
o vhicls, sys coputr scitist mr
duchiu. Th biggst bfit, howv
is th bility to coprss t without
throwig wy iportt ifortio.
Th tchiqu llows for igstbility, which is sstil for ccurt
(left) In the manual stitch image, the road does not line up between the collected images.
(right) Dense image correspondence corrects for intensity variations between cameras to
provide a seamless image.
tchology c prouc subpixl
rsolutio for th bckgrou y
ovrs (popl vhicls), thrby
llowig for itiol lyss of
suspicious ctivitis. a sigl pixl ccorrspo to ywhr fro svrl
squr trs to lss th 1 squr tr
of rl stt.
Th Prsistics rchitctur c support
r rl-ti oitorig for tcticl
cobt issios s wll s forsic
lysis of pst vts. Its lysis
lgoriths prit survillc systs to
str t ky popl, vhicls, loctios,
vts for hours v ys t
ti whil utoticlly srchig
with usurpss til for olis
or prslct trgts. Th Livror
brkthrough cobis optiiz
hrwr fturig th wst grti
of grphics chips (typiclly us forcoputr gig) with iovtiv
lgoriths. So lgoriths focus o
coprssig t whil othrs lyz
strig vio cott to utotic
xtrct its of itrst.
Th syst stbilizs icoig vi
igry through xtrly ccurt
clibrtio of obor crs
corrctio of pixl-to-pixl itsity
vritios. I this wy, osic
fro hurs of iiviul ssors
-
7/28/2019 Livermore Labs - From Video to Knowledge
4/8
7Lawrence Livermore National Laboratory
S&TR April/May 2011 Persistent Surveillance
(left) Persistics allows analysts to automatically follow individual
vehicles. In this scenario, a vehicle is tracked after a meeting
at a presumed safe house (red box) as it travels (green line) to
another building. (above) Persistics can extract a road network
from flowing traffic and show vehicle motion in a heat map
representation of traffic density to automatically discern if the traffic
pattern changes. Spurious tracks can be caused by the glint of
sunshine and other factors.
high-rsolutio objct itifictio
rckig. It lso prits slss stitchig,
procss tht opticlly cobis igs
ro ultipl crs to crt virtul
rg-fort cr. To o this, iiviul
rs ust b fily clibrt to ccout
or rltiv stitch positio itsity
iffrcs, ls istortio, pixl-to-pixl
ritios. Th rsultt osic bhvs lik
sigl ssor for Prsistics to prfor thcssry hug t ructio.
Persistics uses parallax-removing pixel-
level dense correspondence to register
two images and detect what has changed
in the time period between capturing
each image. The left image shows the
original aerial video view. In the center,
two images of the aerial view have been
corrected with the traditional method of
global registration. Because of parallax
motion, some buildings as well as movingvehicles are detected. The right image
corrected with dense correspondence
shows only vehicles in motion (the bright
areas); Persistics has removed the
static background.Global registration Dense correspondence
Spurious tracks
a rquirt for Prsistics thousfol
t ructio is th cotiul rvisit of
survillc pltfor ovr spcifi r. Th
icrsig cohrc of sc built up ovr
y psss proucs high-fility bckgrous
fro which it is sy to spot ovrs. By hvig
ircrft circl th s r, Prsistics
c clcult th ffct of prllx, which
cuss th prspctiv to costtly rott. Th
corrct ig provis stright-ow viwof highly til trri. Rovig prllx
ks it uch sir to tri wht is i
rlity ovig i th bckgrou, sys Prsistics
projct lr Holgr Jos. Currt procssig
tchiqus us globl rgistrtio, which grts
y fls-positiv ovrs (popl vhicls).
Prsistics is fficit i llottig procssig
rsourcs to whr thy r ost . For
xpl, it c utoticlly irct itiol
coputig powr to trck prticulr covoy of
vhicls i rl ti or kp wtch ovr group ofmris who r slp i poorly f r.
-
7/28/2019 Livermore Labs - From Video to Knowledge
5/8
S&TR April/May
8 Lawrence Livermore National Laboratory
(a) The Autonomous Real-Time GroundUbiquitous Surveillance Imaging System
(ARGUS-IS) has four stabilized lenses
(telescopes), which together use 368 5-million-
pixel cell phone cameras. (b) A mosaic of the
368 individual images is stitched together to
form one wide-angle seamless image from which
details can be extracted.
Persistics software is currently housed on a high-density 12-node minicluster, providing an aggregate
40 trillion floating-point operations per second of computing capability. Each node contains a
combination of central processing units and seven graphics processing units. Pictured are (left) Sheila
Vaidya and Holger Jones.
c b rprst s ig fro
sigl cr. This procur llows t
syst to xploit th sptil tpo
rucy i th t ts out
ovrs fro th sttic bckgrou.
In Use at Ground Stations
I collbortio with th ntiol
Gosptil-Itlligc agcy, dfs
avc Rsrch Projcts agcy
(daRPa), air Forc, ary, svr
dprtt of dfs lbortoris,
Livror is icorportig th Prsistic
pipli ito t-procssig grou
sttios f with vio t fro Cost
Hwk vio crs obor both
u ircrft. Rct
of Costt Hwk igry hv vlitPrsistics pproch, ostrtig
1,000-tis coprssio of rw vio
collctios whil itiig high fi
I copriso, coprssio of still i
c ruc t cott tfol, whil
str vio coprssio c chiv
bst 30-tis t ructio.
alysts workig t grou sttio
will itrct with th trsitt irbo
vio t. For xpl, Prsistics
hs b itgrt ito th air Forc
Rsrch Lbortoryvlop Pursu
viwr to llow lysts to p, zoo
rwi, qury, ovrly ps o(a)
(b)
Persistent Surveillance
-
7/28/2019 Livermore Labs - From Video to Knowledge
6/8
9Lawrence Livermore National Laboratory
S&TR April/May 2011 Persistent Surveillance
cll th autooous Rl-Ti Grou
Ubiquitous Survillc Igig Syst
(aRGUS-IS). This syst to tct
trck vts o bttlfils i urb
rs c covr 100 squr kilotrs,
sigifictly grtr ril footprit
th currt systs. Th aRGUS-IS
vio-collctio rt of 12 hrtz (frs
pr sco) is cosirbly grtr th
prvious systs, which oprt t 2 hrtz.
aRGUS-IS copriss 368 crs of
bout 5 illio pixls ch, iticl tothos us i cll phos. Th crs
oprt togthr bhi four high-qulity
tlscop lss. I ll, aRGUS-IS
hs 1.8 billio pixls copr with
4 illio pixls i th first Soo ssor
(th prcssor to aRGUS-IS) vlop
t Livror i 2003, 176 illio pixls
ifortio fro visul t. (S S&TR,
novbr 2004, pp. 1219.)
Persistics Coming On Board
although th currt Prsistics
cofigurtio is sig for grou-
sttio procssig, Livror rsrchrs
r workig to shrik th siz, wight,
powr rquirts to prit outig
of th syst obor survillc crft
withi two yrs. dvloprs r xplorig
th tr-off btw t coprssio,ig fility, coputig rquirts
to succssfully volv th xt-grtio
rchitctur tht will support obor
flight procssig of ssiv t sts.
Prsistics is big furthr hc to
work with daRPas wst grtio of
rl-ti prsistt survillc cpbility
tt. With this viwr, thy c us
rsistics to k rqusts such s, Giv
th frs tht rcor this vhicl
ro o to two oclock this ftroo, or
Show ll th vhicls tht stop t thisoctio toy. Sys Prsistics projct
r Holgr Jos, With Prsistics,
lysts c tri th rltioships
tw vhicls, popl, builigs,
vts.
Prsistics t-procssig ouls rly
corcil vio-procssig hrwr
sig for coputr grphics, vio
itig, gs. GPUs [grphics
rocssig uits] cotiu to grow i
owr, whil shrikig i siz rgy
quirts, sys Jos. I fct, thir
rocssig powr is outpcig tht oftrl procssig uits (CPUs), which
u ost coputr oprtios. Wr
iig th GPU tchology wv,
ys Jos.
Viy l forr projct tht
srch how GPUs ight b
rogr us i kowlg-
iscovry pplictios rlvt to tiol
curity. W rliz ths procssors
ritiolly sig for fst rrig
f visul siultios, virtul rlity,
coputr gigcoul provi
fficit solutios to so of th ost
hllgig coputig s fcig th
tlligc ilitry couitis,
ys Viy. (S S&TR, novbr 2005,
p. 1920.)
Th Prsistics softwr is currtly
ous o 12-o iiclustr,
ch o cotis cobitio
f CPUs GPUs. To hlp t
SR rquirts i cost-ffctiv
r, th Livror Prsistics
hs optiiz cobitio of
icroprocssors high- grphicsrs fou i both gig boxs
y prsol coputrs. Wh
obi with high-sp twork
oftwr tools writt i op-sourc (ot
or-propritry) co, th clustrs
utprfor lrgr or xpsiv
ropritry gis i xtrctig
ARGUS-IS imagery presentsnew challenges because of the
sheer amount of data collected.
The inset reveals people and
vehicles extracted from this
aggregate wide-angle view of
Quantico, Virginia.
-
7/28/2019 Livermore Labs - From Video to Knowledge
7/8
S&TR April/May
10 Lawrence Livermore National Laboratory
Persistent Surveillance
i th ost vc Soo ssor
vlop i 2007, 800 illio pixls
i th ssor vlop i 2009 by th
msschustts Istitut of Tchologys
Licol Lbortory.Prsistics c siultously
cotiuously tct trck th otio
of thouss of trgts ovr th aRGUS-IS
covrg r of 100 squr kilotrs.
aRGUS-IS c grt svrl trbyts
of t pr iut, hurs of tis
grtr th prvious-grtio ssors.
Util ow, w h o prcticl wy to
stor tht uch t, sys Jos. With
Prsistics, w hv iovtiv tho
to coprss th quivlt of thouss of
hr rivs to just fw rivs.
Decades of Visualization Research
about 30 popl r ivolv i th
Prsistics ffort, icluig coputr
scitists, lctricl girs, optics
xprts, sttisticis, chi-lrig
spcilists who rw ig out of
visul t, cor Livror strgth. Th
Persistics researchers are
developing the capability to
produce three-dimensional images
from overhead video data.
Lbortory hs b couctig rsrch
i ISR for two cs. Prvious fforts
iclu sigig vc ssors
lrg, lightwight optics; lyzig spc
igry; ivtig istruts totct vrsril clsti ctivitis
ssocit with wpos of ss
structio. I itio, Livror
coputr scitists r vlopig
thos to g lrg volus of
visul t utot tchiqus to
xtrct ig k iscovris
fro tht ifortio for tiol
scurity pplictios.
aog othr ccoplishts,
rsrchrs hv pior thos to
loct rs of itrst i thr-isiol
siultios of uclr wpos physicsgrt by suprcoputrs prforig
trillios of oprtios pr sco.
Lbortory scitists hv lso vlop
coputr-utot tchiqus to srch
for icroscopic flws i glss optics
us i th ntiol Igitio Fcility, th
worls ost rgtic lsr.
my frl gcis r itrst
i Prsistics pplictios, icluig th
dprtt of ergy, dprtt of
dfs, dprtt of Hol
Scurity. Collbortios i ISR for ucsfgurs trty oitorig supp
xt to itrtiol orgiztios su
s th Itrtiol atoic ergy ag
th Coprhsiv nuclr-Tst-B
Trty Orgiztio.
Ovr th pst c, Livror
rsrchrs hv stui th vxig
issu of ssor output growth fr
outpcig hu cpbilitis to igst
rct to th collct t i ti
r. erly work, fu through
th Lbortory dirct Rsrch
dvlopt Progr, focus ovlopig softwr tchiqus tht
coul first quit th jittry vio tk
obor ovig irbor vhicl
bufft by th tosphr th
coprss th t. Th rsrch ffort
grw ito th dprtt of ergy
projct cll Soo, which vlop
-
7/28/2019 Livermore Labs - From Video to Knowledge
8/8
11Lawrence Livermore National Laboratory
S&TR April/May 2011 Persistent Surveillance
b crri loft. Tht ffort shoul b
sir this yr bcus th siz of
GPUs is xpct to shrik yt gi, with
powr cosuptio ruc i hlf by lt
2011. as w ri th GPU tchologywv, w shoul hv th syst up i
th ir by 2013, sys Viy. W pl
to str to grou oly th ifortio
tht is prtit to th usr, which
y rquir syrgy of ultipl ssors
i th ir to provi robust stits of
whts iportt.
Prsistics tchology will soo b
vilbl s op sourc to th
govrt subcotrctors. Th
tchology is oulr to llow futur
plug ply, s Livror scitists
vlop itiol utot tchiqus.For xpl, thy r rsrchig wys
to k possibl th thr-isiol
viwig of trgts, which coul
furthr hc t coprssibility.
Thy r lso xplorig thos
to ovrly ultipl ssor iputs
icluig ifrr, rr, visul
t th rg t to obti
ultilyr sssst.
Viy ots tht u ircrft
hv ostrt thir ISR vlu for
yrs i afghist Irq. as U.S.
solirs rtur ho, th rol of ovrh
vio igry i by Prsistics
tchology is xpct to icrs.
Prsistics coul lso support issios
t ho, such s oitorig scurity
t U.S. borrs or gurig ports
rgy prouctio fcilitis. Clrly, with
Prsistics, vio s kowlg
strgth tiol scurity.
Arnie Helle
Key Words: agl Fir;autooousRl-Ti Grou Ubiquitous Survillc
Igig Syst(aRGUS-IS); Costt Hwk;grphics procssig uit (GPU); itlligc,survillc, rcoissc (ISR);Prsistics; pixl; Prtor; Rpr; Soo.
For further information contact Sheila Vaidya
(925) 423-5428 (vaidya1@llnl.gov).
qulity. Viy rclls, Our ibility to
trsport th t to grou i fficit
r h bco bottlck.
Prsistics provis ot oly fficit
s of stbilizig coprssigvio to trsport it cross liit
bwith chls but lso bck-
oly tctio bhvior lysis
to iffrtit btw orl
borl pttrs of bhvior.
Fast Forward or Rewind
Prsistics vio procssig is vt-
riv, ig its lgoriths r
sig to tct i rl-ti pottilly
iportt vts i th strig vio
t. Th pipli c tct ovig
popl vhicls trck th ovry hours. autotic rcogitio
of crti scrios c provi rly
wrig of thrt grtly strgth
tcticl cobt support. For xpl,
lli forcs coul itrcpt trrorists i
th procss of pltig iprovis
xplosiv vic or sttig up for
lli bs ttck.Livror scitists r
lso vlopig utot thos for
itifyig pttrs of bhvior tht coul
iict vitios fro orl socil
culturl pttrs s wll s tworks
of subvrsiv ctivity.
at th othr of th spctru,
Prsistics supports forsic lyss.
Shoul vt such s trrorist ttck
occur, th rchivl igry of th public
spc coul b rviw to tri
iportt tils such s th ot
bob ws plc or wh suspct cs
th trgt r. With sufficitly high-
rsolutio igry, lw-forct or
ilitry usr coul o y zoo i o
iiviul fc i hvily popult
urb virot, thus itifyigth ttckr.
Looking Ahead
Livror rsrchrs cotiu to
optiiz th Prsistics pltfor so it c
wi-r ssors for oitorig uclr
oprolifrtio. Th Soo Prsistt
urvillc Syst ftur wi-r
iws t high rsolutio, rl-ti obor
t procssig, high-prforcisuliztio t th rcivig . I 2006,
h Soo Projct t rciv
R&d 100 awr for thir iovtio. (S
S&TR, Octobr 2006, pp. 45.)
Popl i th dprtt of dfs
ot xcit bcus thy rliz thy
oul us th s tchiqus to look
or trrorist ctivitis, sys duchiu.
2005, th vio-cr ffort tht
g with Soo ws pss to th
dprtt of dfs, which vlop
agl Fir for th air Forc mri
Corps Costt Hwk for th ary.aRGUS-IS, built by Bae Systs, Ic., is
h wst igig syst.
durig Soos vlopt,
daRPa grs tur to Livror
o xplor vus for rucig th
ssiv volu of t collct by ths
w ssors without scrificig ig
top related