jsoc pipeline processing system components
DESCRIPTION
Pipeline Program, “module”. JSOC Science Libraries. Utility Libraries. DRMS Library. Record Cache. Record Management. Keyword Access. Link Management. Data Access. JSOC Pipeline Processing System Components. Pipeline processing plan. Pipeline Operator. SUMS Disks. - PowerPoint PPT PresentationTRANSCRIPT
Page 1 JSOC Peer Review 17Mar2005
JSOC Pipeline Processing System Components
Database Server
SUMSStorage Unit
Management System
DRMSData Record
Management SystemSUMS Tape Farm
SUMS Disks
Pipeline Program, “module”
Record Manage
ment
Keyword Access
Data Access
DRMS Library
Link Manage
ment
Utility LibrariesJSOC Science
Libraries
Record Cache
PUIPipeline User
Interface
Pipeline processing
plan
Processing script, “mapfile”
List of pipeline modules with needed datasets for input, output
Pipeline Operato
r
Processing History Log
Page 2 JSOC Peer Review 17Mar2005
Storage Unit Management Sybsystem (SUMS) API
• SUM *SUM_open(char *dbname)
Start a session with the SUMS
• int SUM_close(SUM *sum)
End a session with the SUMS
• int SUM_alloc(SUM *sum)
Allocate a storage unit on the disks
• int SUM_get(SUM *sum)
Get the requested storage units
• int SUM_put(SUM *sum)
Put a previously allocated storage unit
• int SUM_poll(SUM *sum)
Check if a previous request is complete
• int SUM_wait(SUM *sum)
Wait until previous request is complete
Page 3 JSOC Peer Review 17Mar2005
SUM_get() API Call Sequence
DRMS Client
SUM Server
SUM Library
SUM_get()
get Procedure
resultpend
Sample SUM Call
Oracle Server
get storage units
offline
Tape Server
retrieve storage units
resultpend
SUM_poll() orSUM_wait()
ack
SUM_flow.vsd
online loc
ack
retrieve storage units
online loc
response Procedure
Page 4 JSOC Peer Review 17Mar2005
HMI Dataset Sequence
tlm packets
camera& hk data~1700bytes
HMI 55Mbs
filtergrams
n:132MBImage
n:m
observables
32MBImage
/home/jim/soffice/user/work/dataset_seq3.sxd
tlm
lev0
lev1
Ancillary Data
HK
Page 5 JSOC Peer Review 17Mar2005
Pipeline User Interface (PUI) Block Diagram
Data Product
Plan Table
Data Query
Oracle DB
Map BuildingScripts
Map Execution
PUI GUI
pui_svc
pipelineexecution
Page 6 JSOC Peer Review 17Mar2005
Pipeline User Interface (PUI)
Page 7 JSOC Peer Review 17Mar2005
Pipeline User Interface (PUI)
Page 8 JSOC Peer Review 17Mar2005
Pipeline User Interface (PUI)
Page 9 JSOC Peer Review 17Mar2005
Pipeline User Interface (PUI)
Page 10 JSOC Peer Review 17Mar2005
Pipeline User Interface (PUI)
Page 11 JSOC Peer Review 17Mar2005
HK Data Flow
MOCDDS
Science SOC T & C SOC
GSFC
dailyLR
realtimeLR APID
Lockheed
WSC
Stanford
socket ftp
realtimeHR APID
ftpdaily LR APID
FDS/PLANNING
HK DefinitionTables
HK KeywordDB
realtime LR APID
FDS/PLAN
Page 12 JSOC Peer Review 17Mar2005
CM with CVS