fsr – frustrating summary records
DESCRIPTION
FSR – frustrating summary records. file. ^. R. Lambert. Idea. Book-keeping. Log file. Provenance information. Log file. Log file. File Metadata. Log file. File Metadata. File Metadata. File Metadata. Data file. File Metadata. Provenance information. Data file. Event data. - PowerPoint PPT PresentationTRANSCRIPT
Rob Lambert, NIKHEF Core Soft, 26th September 2012 1
FSR – frustrating summary records
R. Lambert
file
^
Log file
Idea
Rob Lambert, NIKHEF Core Soft, 26th September 2012 2
Event data
File Metadata
File MetadataData file
Event data
Data file
Log file
File Metadata
Log file
File Metadata
Log file
File Metadata
Book-keeping
Provenance information
Provenance information
Gaaah!
Rob Lambert, NIKHEF Core Soft, 26th September 2012 3
… and these are justthe savannah bugs/tasks
Current Structure
Rob Lambert, NIKHEF Core Soft, 26th September 2012 4
/FileRecords /Event
…FSR …/GUID /GUID
FSR …/GUID /GUID
FSR …/GUID /GUID
Event …Event …Event …
FSR
FSR
FSR
Event …Event …Event …
Current Structure
Rob Lambert, NIKHEF Core Soft, 26th September 2012 5
/FileRecords /Event
…FSR …/GUID /GUID
FSR …/GUID /GUID
FSR …/GUID /GUID
File MetadataCustom algorithms
Provenance InformationAutomatically Created
Event DataCustom Algorithms
Open File
Current Procedure
Rob Lambert, NIKHEF Core Soft, 26th September 2012 6
/Event
FSRs
Provenance
Output FileTES
/Event
FSRs
Execute Finalize
/Event
FSRs
Provenance
Input File(s)
+
+
Pack
Provenance
FSRsFSRs
Current Streaming
Rob Lambert, NIKHEF Core Soft, 26th September 2012 7
Output File(s)
TES
/Event/…
/Event
FSRs
Provenance
Input File(s)
+
+
/Event/…
Open File Execute Finalize
+
+ /Event/…
/Event
FSRs
Provenance
/Event
FSRs
Provenance
/Event
FSRs
Provenance
Pack
Pack
Pack
Provenance
FSRsFSRs
FSRs
Observations FSRs are sparse, Events are chunky
FSRs have many levels in the tree, Events have few levels
FSRs encode information in the structure, only one “event”
Rob Lambert, NIKHEF Core Soft, 26th September 2012 8
/FileRecords /Event
…FSR …/GUID /GUID
FSR …/GUID /GUID
FSR …/GUID /GUID
Event …Event …Event …
Observations FSRs are sparse, Events are chunky
FSRs have many levels in the tree, Events have few levels
FSRs encode information in the structure, only one “event”
Rob Lambert, NIKHEF Core Soft, 26th September 2012 9
/FileRecords /Event
…FSR …/GUID /GUID
FSR …/GUID /GUID
FSR …/GUID /GUID
Event …Event …Event …
Physical Problems Many levels = many baskets
Rob Lambert, NIKHEF Core Soft, 26th September 2012 10
Physical Problems Many levels = many baskets
Basket size is waaaaaaay too large
Rob Lambert, NIKHEF Core Soft, 26th September 2012 11
FSR , ~128 bits
Basket , ~262144 bits
FSR , ~128 bits
Basket , ~262144 bits
Physical Problems Many levels = many baskets
Basket size is waaaaaaay too large
Only one “event” written, basket size never optimized
Rob Lambert, NIKHEF Core Soft, 26th September 2012 12
I’m sure this is fine. Don’t worry about it. Ask me again in 9 events time.
Physical Problems Many levels = many baskets
Basket size is waaaaaaay too large
Only one “event” written, basket size never optimized
Slow to navigate
Rob Lambert, NIKHEF Core Soft, 26th September 2012 13
Physical Problems Many levels = many baskets
Basket size is waaaaaaay too large
Only one “event” written, basket size never optimized
Slow to navigate
Huge memory footprint (Gigabytes)
Massive file size increase (Hundreds of Megabytes)
Crazy computing time (30 minutes in finalize)
Rob Lambert, NIKHEF Core Soft, 26th September 2012 14
Open File
Merging in Production
Rob Lambert, NIKHEF Core Soft, 26th September 2012 15
/Event
FSRs
Output FileTES
/Event
Execute Finalize
/Event
FSRs
Provenance
Input File(s)
+
+
Pack
Provenance
FSRsFSRs
FSRsMerge
Best Case Scenario Normal Production
Rob Lambert, NIKHEF Core Soft, 26th September 2012 16
RAW SDST DST
uDST
Brunel DaVinci DaVinci
DaVinci
Reconstruction Stripping+Streaming Merging
DST
uDST
~5 x1 FSRs0 Daughter1 Level
~5 x1 +1 FSRs1 Daughter2 Levels
~5x1+1 FSRs0 Daughter1 Level
Worst Case Scenario MC Filtering
Nominally A=1, B~10, C~100… nominally 5,000 FSRs!
Rob Lambert, NIKHEF Core Soft, 26th September 2012 17
SIMGauss DIGIBoole
DST LHCb DST
DST
DaVinci
DSTMoore
“A” FSRs0 Daughter1 level
A+2 FSRs1 Daughters2 levels
A+3 FSRs2 Daughters3 levels
A+4 FSRs3 Daughters4 levels
(A+4)xB+1 FSRs3*B Daughters5 levels
((A+4)xB+1)xC+1 FSRs3*B*C Daughters6 levels
B C
Brunel
Immediate prospects Current work around:
Delete and clean up FSRs (might as well not write them) Parse and merge the XMLSummaries instead
Fixes required Resurrect provenance information Write different FSRs out depending on stream
New smarter EventCountFSR IOFSR (prototype written) Similar IO information to the XMLSummary Stores input GUIDs with number of events read Stores vectors of information for daughter files
Creating IOFSR done by new FSR writer (prototype t.b.d.)
Rob Lambert, NIKHEF Core Soft, 26th September 2012 18
With new RootCnv Heard a nice talk last week on basket optimization
Repeat my older tests
Well done Markus and Ivan! But it’s still not great for FSRs
Rob Lambert, NIKHEF Core Soft, 26th September 2012 19
SetupProject DaVinci v32r2p1Time gaudirun.py ~rlambert/public/forMarkus/fsrMemLeak/options.py#20 files with one event each, but 4681 FSRs in total
Existing RootCnv Time (usr+sys) / s VMEM / MiB File Size / MiB
With FSR Cleaning 39.7 995 2.4
No FSR Cleaning 619.9 2704 15.2
RootCnv Head Time (usr+sys) / s VMEM / MiB File Size / MiB
With FSR Cleaning 22.5 936 2.1
No FSR Cleaning 116.7 1073 5.0
Conceptual problems
X Provenance information requires complicated navigation … and then we throw it away anyway. Great.
X Requires a second writer and separate service
X Sequencing!1. Keep output file open
2. Create FSR object and register on TES during finalize
3. Write FSR object to file Very complicated once Streaming is involved
X All FSRs in output streams are identical Output files don’t necessarily have the same metadata
X Different structure suggests different optimal working point
Rob Lambert, NIKHEF Core Soft, 26th September 2012 20
Fixing any of these will fix FSRs for good
The Future?
Rob Lambert, NIKHEF Core Soft, 26th September 2012 21
10,000 FSRsSparse Trees in TES
ROOT can’t handle sparse trees
Sparse Trees Persisted
ROOT Tree used
Deconstruction
Rob Lambert, NIKHEF Core Soft, 26th September 2012 22
10,000 FSRsSparse Trees in TES
ROOT can’t handle sparse trees
Sparse Trees Persisted
ROOT Tree used
Fixing any of these will fix FSRs for good
1. Merging (what we do right now)• Currently throws away provenance information • Requires very very careful sequencing for finalization order• Requires each FSR type to have some associated merger
• OK for small number of FSR types and a lot of manpower• … there are several other options, though…
Deconstruction
Rob Lambert, NIKHEF Core Soft, 26th September 2012 23
10,000 FSRsSparse Trees in TES
ROOT can’t handle sparse trees
Sparse Trees Persisted
ROOT Tree used
Fixing any of these will fix FSRs for good
2. Merging (automatic, and IOFSR)• Provenance information will be kept• Have merging done by a service or tool, on FSR baseclass• Do this somehow automatically and always cleanup correctly
• OK, but needs a lot of thought about how best to implement
Deconstruction
Rob Lambert, NIKHEF Core Soft, 26th September 2012 24
10,000 FSRsSparse Trees in TES
ROOT can’t handle sparse trees
Sparse Trees Persisted
ROOT Tree used
Fixing any of these will fix FSRs for good
3. Event-like FSRs• Treat each FSR as an event, file GUID instead of event number• Write to the same location several times, a mini event container• Completely changes FSR mechanics of Gaudi
• Good for any purpose.
Sparse Trees Persisted
Fixing any of these will fix FSRs for good
4. Flattening (LHCb “Packed” FSR)• Write a master class which holds information at one top level• Let there be only one “FSR” which holds everything• Frequent and invasive schema migration for new FSRs
• Good solution for up to thousands of FSRs
Deconstruction
Rob Lambert, NIKHEF Core Soft, 26th September 2012 25
10,000 FSRsSparse Trees in TES
ROOT can’t handle sparse trees
ROOT Tree used
Sparse Trees Persisted
Fixing any of these will fix FSRs for good
5. Flattening (LHCb Packed FSR)• Each writer creates a /FileRecords/Packed location• Vector of data objects, only write out this location• Complicated to work around existing Gaudi FSR system
• Good solution for up to thousands of FSRs
Deconstruction
Rob Lambert, NIKHEF Core Soft, 26th September 2012 26
10,000 FSRsSparse Trees in TES
ROOT can’t handle sparse trees
ROOT Tree used
Sparse Trees Persisted
Fixing any of these will fix FSRs for good
6. Flattening (Gaudi Packed FSRs)• Let the TES and Persistent classes be different• Have the persistent class be a __flattened__ tree (auto-unpack)• Requires invasive re-coding of parts of Gaudi
• A good solution for up to thousands of FSRs
Deconstruction
Rob Lambert, NIKHEF Core Soft, 26th September 2012 27
10,000 FSRsSparse Trees in TES
ROOT can’t handle sparse trees
ROOT Tree used
Fixing any of these will fix FSRs for good
7. Abandon Root trees completely• Don’t use data objects at all, just append an ntuple to the file• Avoids all baskets and other problems• Completely changes FSR mechanics of Gaudi
• The only solution if 10,000 FSRs are needed
Sparse Trees Persisted
Deconstruction
Rob Lambert, NIKHEF Core Soft, 26th September 2012 28
10,000 FSRsSparse Trees in TES
ROOT can’t handle sparse trees
ROOT Tree used
ROOT can’t handle sparse trees
Fixing any of these will fix FSRs for good
8. Get sparse trees in Root to work properly• Have basket size and optimization tuneable tree-by-tree• Needs work from the ROOT team• Only possible on new stack ~6-month time scale
• Perfect solution, no new LHCb/Gaudi code required
Sparse Trees Persisted
Deconstruction
Rob Lambert, NIKHEF Core Soft, 26th September 2012 29
10,000 FSRsSparse Trees in TES
ROOT Tree used
Fixing any of these will fix FSRs for good
There is a possible solution for each problem …
What do we persue?
Fixing the whole idea of FSRs is very desirable
Not biting the bullet now, means more manpower later …
The Future?
Rob Lambert, NIKHEF Core Soft, 26th September 2012 30
10,000 FSRsSparse Trees in TES
ROOT can’t handle sparse trees
Sparse Trees Persisted
ROOT Tree used
Summary FSRs are a new addition to Gaudi, not yet robust
MC Filtering is being addressed IOFSR to keep provenance information
Currently to add a new FSR type is very complicated
We can work around the problems, but shouldn’t we fix this part of our software which is underperforming? Automatic merging Event-like FSRs Flattening (Three possible directions) Abandon trees altogether Fix sparse trees
Rob Lambert, NIKHEF Core Soft, 26th September 2012 31
End Backups are often required
Rob Lambert, NIKHEF Core Soft, 26th September 2012 32
Discussion
Rob Lambert, NIKHEF Core Soft, 26th September 2012 33