data handling, compression and telemetry formatting...data formats • transmission scheme and...
TRANSCRIPT
Data Handling, Data Handling, Compression and Compression and Telemetry FormattingTelemetry Formatting
Institute of Space Studies of Catalonia (IEEC)Institute of Space Studies of Catalonia (IEEC)University of Barcelona (UB)University of Barcelona (UB)
Technical University of Catalonia (UPC)Technical University of Catalonia (UPC)
Presented by Jordi Portell (IEEC/UPC)Presented by Jordi Portell (IEEC/UPC)
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
ScienceScience Data Data Pipeline Pipeline in in GaiaGaia
Instruments [Instruments [CCDsCCDs]]
Data Handling [PDHS/PDHE]Data Handling [PDHS/PDHE]
Formatting and codificationFormatting and codification
Data compressionData compression
TM system / RFTM system / RFDB
MultiplexingMultiplexing
ASTROASTRO MBPMBP RVSRVS
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Payload Payload Data Data Handling SystemHandling System•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
Full design available (system level)Full design available (system level)–– Modules andModules and submodulessubmodules–– Data fluxData flux
Consistent with GASS, GIBIS and PDHEConsistent with GASS, GIBIS and PDHEA good reference for the final mission: A good reference for the final mission: Processes/modules that generate and Processes/modules that generate and modify the data in the payload ofmodify the data in the payload of GaiaGaiaASTRO and PDHU described. SPECTRO can ASTRO and PDHU described. SPECTRO can easily be included with a similar designeasily be included with a similar design
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
PDHS PDHS SchematicSchematic•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Data Data Formats for SimulationsFormats for Simulations•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
GAIAGAIA--BCNBCN--007007: : ““OnOn--board and telemetryboard and telemetry data data formatsformats””RecommendedRecommended Data Data Formats for simulations implying the Formats for simulations implying the Telemetry StreamTelemetry Stream::
–– GASSGASS–– Telemetry formatting and packetingTelemetry formatting and packeting–– Data Data compressioncompression–– ......
Current versionCurrent version (1.5b):(1.5b):–– ASTRO (GDAASASTRO (GDAAS--2)2)–– MBP MBP andand RVS RVS temptative codingtemptative coding
Forthcoming versionsForthcoming versions::–– MBP+RVS MBP+RVS stablestable (GDAAS(GDAAS--2). HK+2). HK+AttitudeAttitude??–– FlightFlight--realisticrealistic–– ......
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Data Data Formats for SimulationsFormats for Simulations•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations CodingCoding//transmission scheme fortransmission scheme for
RVS RVS shouldshould be be decideddecided::–– NowNow 1 1 patchpatch / 1 / 1 starstar ((simplisticsimplistic))–– TransmissionTransmission by by readoutreadout time?time?–– Transmission ofTransmission of 1 1 patchpatch / / star star excluding overlapsexcluding overlaps??
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Transmission schemeTransmission scheme•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
GAIAGAIA--BCNBCN--006006: : Proposal ofProposal of a a transmissiontransmission, , multiplexing and coding scheme for the payload of Gaiamultiplexing and coding scheme for the payload of Gaia::
–– Compilation of requirementsCompilation of requirements–– Multiplexing ofMultiplexing of data data sourcessources–– Integration within Packet TelemetryIntegration within Packet Telemetry–– RobustRobust ((transmission errorstransmission errors, , packet orderpacket order))–– Coding ofCoding of data data fields complyingfields complying CCSDS CCSDS standardsstandards
Optimized forOptimized for a a minimal bandwidth consumptionminimal bandwidth consumptionPrepared forPrepared for data data compression applied afterwardscompression applied afterwardsAlgorithm for compiling theAlgorithm for compiling the data data from instrumentsfrom instrumentsClock imperfections willClock imperfections will be be taken into accounttaken into account ((now now being studied underbeing studied under a Master a Master ThesisThesis))
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
MultiplexingMultiplexing•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
RobustnessRobustness•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Source PacketsSource Packets ((PacketPacket TM)TM)•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Telemetry modelTelemetry model•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
Model of theModel of the average average telemetry consumptiontelemetry consumption((simplisticsimplistic))
Optimization of codification parametersOptimization of codification parameters: : ResolutionsResolutions, , length oflength of Time Time SlotsSlots, , maximum sourcesmaximum sources in a SP...in a SP...MatLab simulations for optimizing these parametersMatLab simulations for optimizing these parametersGUI GUI framework for optimizing telemetry parametersframework for optimizing telemetry parameters
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Optimization of parametersOptimization of parameters•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Average Average resultsresults•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations Preliminary optimizationsPreliminary optimizations::
–– HigherHigher time time resolutionsresolutions, CCSDS, CCSDS--compatible data compatible data fieldsfields–– > 2.5 > 2.5 kbpskbps average average savingsaving (GDAAS2(GDAAS2--like modified codinglike modified coding))–– > 32 > 32 kbpskbps average average savingsaving (GDAAS2 original (GDAAS2 original codingcoding))–– Adaptive codification recommendedAdaptive codification recommended ((depending on star ratedepending on star rate))
ASTRO average data ASTRO average data raterate: ~1.9 : ~1.9 MbpsMbps (no (no comprcompr.).)
< Only averaged (simplistic) simulations >
“Telemetry coder” under development...“flight-realistic” telemetry curves
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Data Data compressioncompression•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• Telemetry Telemetry modellingmodelling and and optimizationoptimization
•• Data Data CompressionCompression
•• Pipeline of Pipeline of Telemetry Telemetry SimulationsSimulations
Redesign ofRedesign of GaiaGaia has relaxed the requirements:has relaxed the requirements:Lower spin rate Lower spin rate lower star rate lower star rate llowerower data ratedata rate
Anyway:Anyway: ComprCompr. ratio of ~. ratio of ~44 is required (is required (losslesslossless))
Preliminary results are satisfactory:Preliminary results are satisfactory:
Differential Differential adaptiveadaptive prepre--compressorscompressors+ standard compression algorithms+ standard compression algorithms
3.0 to 3.83.0 to 3.8using GASS binary data (raw data),using GASS binary data (raw data), GaiaGaia--1 design1 design
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Data Data compressioncompression•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data Data CompressionCompression
•• Pipeline of Pipeline of Telemetry Telemetry SimulationsSimulations
Telemetry stream readyTelemetry stream readyfor including thefor including the DataDataCompressionCompression ““blackblack boxbox””::
New designs coming soonNew designs coming soon::–– Use Use of newof new data data fromfrom GASS GASS usingusingGaiaGaia--2 2 and other improvementsand other improvements–– Improved preImproved pre--compressorscompressors
–– Persistency of TreesPersistency of Trees & & DictionariesDictionaries are are taken into accounttaken into account(data (data integrity even with txintegrity even with tx. . errorserrors))–– Cosmic rays mustCosmic rays must be be taken into accounttaken into account–– Optimized algorithmsOptimized algorithms((onon--board processing requirementsboard processing requirements!)!)
Data Compression
...
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
Telemetry pipelineTelemetry pipeline•• Payload Data Payload Data
Handling System Handling System (PDHS)(PDHS)
•• Data FormatsData Formats•• Transmission Transmission
scheme and scheme and multiplexingmultiplexing
•• TelemetryTelemetrymodellingmodelling and and optimizationoptimization
•• Data CompressionData Compression•• Pipeline of Pipeline of
Telemetry Telemetry SimulationsSimulations
Telemetry coder
DataCompression
[GAIA-BCN-007]
Txt2Bin
Binarydata
[GAIA-BCN-006]
Format Mplex. Packet
Source Packets (uncompressed science data)
[CCSDSPacket TM]
SPs (compr.sci. data) Tr. Frames
composerTFs [Transmiss.
Channel]TFs with errors
[CCSDSPacket TM][HRTF][GAIA-BCN-001 and ...]
Ground StationReceive Error Correct.
TFs OK
(GDAAS)
(GDAAS)
DB
ASCIIoutput
Raw sci. data
[Errors, cosmic,windows,
SourceId...]
[Error rate]
GASS
[Errors, cosmic,windows,
SourceId...]
DecoderSPsTFs
[PDHS, clocks...]
Gaia SWG/RVS Meeting J. Portell et al.Nice (France), 10-12 December 2003
ConclusionsConclusionsPDHS PDHS designdesign finishedfinished
Data Data sourcessources, , modifiers and main processes identifiedmodifiers and main processes identified[ SPECTRO [ SPECTRO instrument toinstrument to be be includedincluded ]]
Data Data formattingformatting defineddefinedStandarisation for simulators makingStandarisation for simulators making use use of telemetry streamof telemetry stream
[ [ ConsistentConsistent MBP MBP andand RVS data RVS data mustmust be be includedincluded ]][ [ Document must evolve towards flightDocument must evolve towards flight--realisticrealistic ]]
Transmission schemeTransmission scheme defineddefined, , modelled and optimizedmodelled and optimizedImprovementsImprovements in data in data qualityquality, , reliability and compatibilityreliability and compatibilityReducedReduced data data raterate
[ [ ‘‘Telemetry CoderTelemetry Coder’’ toto be be developeddeveloped ]]
Key factors forKey factors for data data compressioncompression are are identifiedidentifiedDevelopment of an optimal system is well plannedDevelopment of an optimal system is well planned
[ Final Data [ Final Data Compression System to Compression System to be be developeddeveloped ]]
Telemetry pipelineTelemetry pipeline identifiedidentifiedTelemetry simulations willTelemetry simulations will be be easily concatenatedeasily concatenatedImprovementsImprovements//updatesupdates//adaptationsadaptations can be can be easily introduced between simulatorseasily introduced between simulators