r2r qa for ctd profile data - unols
TRANSCRIPT
R2RQAforCTDProfileData
StatusUpdateReportCyndyChandler(WHOI)
RollingDecktoRepository(R2R)
November2016AnnualRVTECMeeOng
CTD–ConducOvity,Temperature,Depth
Sea-Bird911plushUp://www.seabird.com/sbe911plus-ctd (Plueddemann,WHOI;OOIKN222,2014)
CTDdatafiles
• VesselresidentCTDunitswithaddiOonalopOonalsensors(dissolvedO2,PAR,transmissometer,fluorometer)
• CTDdata(andsetup)filesets• Setoffilesforeachcast(CTDdeployment)
• ConOnuousprofiledata(downandup)• ‘boUlefiles’:boUlefirerecords
slide4of23
R2RQAforCTDprofiledata• TeamMembers
CarolinaNobre(WHOI,PhysicalOceanographyDepartment)CyndyChandler(WHOI,BCO-DMO,R2R)LauraStolp(WHOI,ShipboardScienOficServices)
• Generalupdateonprojectstatus:– BatchSeaBirdProcessing
• GeneratesprocessedCTDfiles– CTDQAscript(python)thatrunsonCTDfilesfromeachcruise
• generatesR2RQAsummaryreport(XML)Runon978cruisedatasets(rawCTDdatathathadbeenbrokenoutfromcruisedistribuOons)
– ProcessedCTDdataproductsgeneratedfor853(ofthe978)
R2RQAforCTDprofiledata
IniOalresultsofQAprocessing:• About10%couldnotbeassessed
• missingfiles,PARsensorissues
• CTDdatafrom853cruisesprocessed• ~40%ofthecruisesareallgreenlights• ~30%ofthecruiseshaveatleastoneyellowlight• ~30%ofthecruiseshaveatleastoneredlight
slide6of23
• SeabirdModules• datcnv• wildedit• Celltm• filter• loopedit• Binavg(2db)• Split
• ParametersExtracted– Temperature– ConducOvity– Oxygen– AlOmeter– …
RawFiles->SeabirdProcessing->ProcessedCTD
n RawFilesn .hex/.datn .con/.xmlconn .hdrn .bln .NAVn ,,,
n .cnvFilesn No.ofStaOonsn BoUlesFiredn Max/mincast
depthn Sensor
ConfiguraOon
– Cruisemetadata– ValidityoftheGPSdata– PresenceofallexpectedrawSeaBirdfiles– Presence/validityofdate/Omeentries
– Sensors:– PresenceofRedundantSensors– DataRanges(comparisonwithmanufacturer
specsforeachsensor)– SensorListandCalibraOonDates
– DataCoverage:– NumberofstaOonsdetected– DetecOonofboUlesfired– Minimumpressuretest(castswentbelow5m)
QA–Metadata,SensorsDataCoverage
QAProcessSteps
Processhasevolvedoverseveralyears;informedbyfeedbackfromChrisPaver(NCEI),shipboardtechs,R2Rpartners,WHOIPOscienOstsanddatamanagers• AbilitytohandledifferentversionsSeaBirdSeaSon• Abilitytomodify/create.psafilesandinsertthemintobatchprocessing(SeaSavesonwaresetupforWindows)
• Addedseabirdmodules(Wildedit,LoopEdit,BinAvg,Split)togenerateprocessedCTDprofiledataproduct(2decibar,bin-averaged,down/upcastfiles)
slide13of23
QAProcessSteps• R2RNavprocessingvalidatestemporalbounds(start/end
dates)andfillsinspaOalbounds(WESNbox)• CTDQAprocesschecksagainstthosespaOalandtemporal
cruisebounds,andidenOfiesoutliercastsintheXMLreport• R2RbreakoutprocessreadstheCTDQAXMLreportand
updatesSQLdatabase(ifneeded),andresubmitsfilestoNCEIifnecessary(newversion)
• InaddiOontoindividualtestresults,CTDQAreportinfoblockincludeswhichcasts/filesfailedtests
• BoththeLevel0(originalraw)andLevel1(processed)willbesubmiUedtoNCEI(oneQAreportappliestoboth)
slide14of23
QAProcessStepsCreateR2RQualityAssessment(QA)RollupReport• XMLformat,compliantwithR2Rv1.0schema• BasicprovenanceinformaOon(cruise,filesetID)• InfoResultsSummary:• TotalRawFiles,TotalProcessedFiles
BoUlesFired(True/False)%ofCastswithBoUlesFiredNumberofSensorsDetectedSensorInfolist:SensorName,SensorID,CalibraOonDateCreatesuiteofsensor-specificplots
• FileSetInfo:basiccruiseandinstrumentmetadata• Filemanifest:filename,IDandchecksumvaluesCTDrawandprocesseddataplusQAfilessubmiUedtoNCEI
slide15of23
QAProcessSensorTests
• Min/maxrange(sensormanufacturerspecs)• Gap• Spike• ConstantValue• GradientTest• OutlierTest• DualSensorPresenceTest(TempandCond)• DualSensorDifferenceTest(TempandCond)
slide16of23
QAProcessResults
• CTDQAreport(XML)• Sensor-specificQAplots• QAdashboarddisplay(accesstoQAresults)• Product:“ProcessedSeabirdCTDfiles”
• ConsistentCTDdataproductfromeverycruise• Downanduptracedatafileforeachcast• .psa(processingfile)• ConReports(ASCIIversionwithfullinstrumentmetadata,includingallsensorinfo)
slide17of23
FutureEnhancements
• ConOnueevaluaOngwhysomecruisesarefailingcertaintests(adjustprocess)
• Reviewtestresultswithdomainexperts• ExposetheCTDQAtestresultsonthepublicdashboardview
Hurdles(andtherehavebeenafew…)
• Syntaxofusersupplieddata• Example:**LaOtude:3136.32
• MismatchedFileNames• DeckTests(missingfiles,emptyfiles)
– (missingfilesareredforexample)
• PARsensorinconsistency(detailsnextslide)
PARSensorIssue
TheabilitytocollectPARdataisdeterminedbytwoseparate’switches’,oneonthedeckunitandtheotherinthe.confileIfthetwodon’tmatch(switchforPARdataisonintheconfilebutnotonthedeckbox,orviceversa)youendupwitheithernoPARdataorbaddata.