fast 2d detector control at esrf beamlines
DESCRIPTION
Fast 2D Detector Control at ESRF beamlines. David Fernández Alejandro Homs Emmanuel Papillon Manuel Pérez. Vicente Rey Cyril Guilloud V. Armando Sole Assen Kirov. Talk outline. Introduction ESRF FReLoN CCDs 2D control arquitecture Espia card based systems Espia card FReLoN 2k, Maxipix - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/1.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Fast 2D Detector Control at ESRF beamlines
David Fernández Alejandro Homs
Emmanuel PapillonManuel Pérez
Vicente ReyCyril Guilloud V. Armando SoleAssen Kirov
![Page 2: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/2.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Talk outline
•Introduction
ESRF FReLoN CCDs
2D control arquitecture
•Espia card based systems
Espia card
FReLoN 2k, Maxipix
Dalsa, Sarnoff, Aviex
•Latest developments
Driver improvements
Data saving
Common CCD acq. platform
![Page 3: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/3.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
FReLoN 1000 (1996):
• 1024 x 1024 - 14 bit• 4 parallel readout channels• Readout time: 100 ms
FReLoN 2000 (1998):
• 2048 x 2048 - 14 bit• Readout time: 220 ms
FReLoN – Fast Readout & Low Noise
![Page 4: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/4.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
FReLoN 1000/2000 acquisition setup
Gigabit Ethernet
TACO Device serverTACO Device server
SPECSPEC
![Page 5: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/5.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Princeton Sensicam, PCO Photometrics Matrox Meteor II [H. Witsch, J. Meyer] Trixell Pixium [L. Claustre] Andor ImagePro [A. Gotz] Direct-Show
Other CCD-like devices supported
Mar CCD [L. Claustre] IDS Falcon [D. Miraut, J. Meyer] Firewire (IEEE 1394) [A. Gotz] Dalsa Dalstar
![Page 6: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/6.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
ADSC Q315r for MX beamlines
Gigabit Ethernet
Local Gigabit
![Page 7: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/7.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Designed to interface the FReLoN 2k Collaboration ESRF + SECAD, S.A. 2 Gbps fiber optic link (> 100 m) PCI 64 bits / 66 MHz ~180 MB/s maximum data rate Bus master + Scatter Gather + IRQ Driver source code available rewritten!
Can write on 32-bit addresses only (4 GB)
Espia card
![Page 8: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/8.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Control workstation (2005)
Transtec (2005): Dual Intel Xeon 2.6 GHz, HT 2 GB ECC DDR-RAM 3 x 73.4 GB Ultra320 SCSI 15Krpm SuSE 8.2 + Linux 2.4.20-64GB-SMP
• Remote NFS through 1 Gbps link 90 MB/s• 2 disks in software RAID 0 100 MB/s (8 MB files)• 4 disks in software RAID 0 150 MB/s (1 GB files)
for ultra-fast tomography
![Page 9: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/9.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
New FReLoN Generation
FReLoN 2k (ATMEL): No Mux – only one fiber optic! Frame Transfer Mode 2048 x 1024 @ 16 fps 66 MB/s
FReLoN 4M (KODAK): Only 4 channels – Full Frame Mode
FReLoN HD (PSB-2): 2048 x 1024 @ 32 fps ~125 MB/s
![Page 10: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/10.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
• Connects to the Espia• 2 x Camera Link connectors:
• Base configurationMultiplexed• Multiple pixel packing formats
• Test image generator @ ~180 MB/s
Camera Link: FOCLA
![Page 11: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/11.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Frame transfer technology 1024 x 1024 @ 60 fps 120 MB/s 1 x Camera Link [Base]
Connector Compact! Image not reconstructed FOCLA does reconstruction
Dalsa Pantera 1M60
![Page 12: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/12.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Dalsa setup
Fast online displayImage reconstruction
![Page 13: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/13.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
512 x 512 @ 300 fps 150 MB/s 512 x 128 @ 500 fps 125 MB/s 2 x 8 ADCs – 12 bit 2 x Camera Link [Base]
Connectors Separate RS-232 serial line
Sarnoff CAM512
![Page 14: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/14.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Sarnoff setup
Camera Link
Fiber optic
RS-232
![Page 15: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/15.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Sarnoff results (2007)• 300 fps• 75 frames• 250 ms• 40x
slowdown
No sound! B&W Flickering
(room light)
ID15: Tomography
in 0.5 sec @ 500 fps
![Page 16: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/16.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
The real image
Espia #1
Espia #2
![Page 17: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/17.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Image reconstruction in kernel
Espia #1
Espia #2
![Page 18: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/18.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
“Very High memory” support
Physical Memory
Low RAM0 MB
3.2 GBPCI
Very high RAM
800 MB
4 GB
5.8 GB
High RAM
32 bit
boundary
• Driver always has 32-bit aux. buffers• If next target frame has 64-bit pages:
Program each Espia to write on aux buffer
Frame end (all adapters finished) copy from aux. into target buffer
Inform waiting threads• Adapters are not synchronous• Memory copy with IRQ disabled
Kernel thread
Implementation in driver
High frame rate many image buffers
![Page 19: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/19.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Aviex PCCD-8448
2048 x 3584 @ 1.2 fps 18 MB/s
Binning 16 x 16 8 fps
![Page 20: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/20.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Software image reconstruction
ADC
ADC
ADC
ADC
• Per frame operation• Warranted only once in a multi-frame (big) buffer• Callback thread safe
![Page 21: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/21.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Medipix2 / Maxipix
Pixel detector photon counting 256 x 256 – 13 bit 0.3 ms readout time 1000 fps 130 MB/sMax: 1400 fps 180 MB/s
Chipboard: University of Geneve PRIAM: ISG/ESRF Interfaced to the Espia
Espia
![Page 22: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/22.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Maxipix results (2007)
• 1000 fps !!
![Page 23: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/23.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Multichip MaxipixMedipix2 array 5 x 1 1280 x 256
1000 fps 650 MB/s PRIAM multiple FO links Future configs: 2 x 2, 5 x 2
![Page 24: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/24.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
New control workstation
OPALE server from Ecrin Systems (FR):
Trenton Single Host Board and backplane Dual Quad Core Intel Xeon @ 2.33 GHz 8 GB of FB DDR2 3 x 140 GB SAS disks @ 15 Krpm
Chelsio T320 10 Gbps Ethernet controller
![Page 25: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/25.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Driver evolution
32-bit application 3GB address space 8GB RAM Dynamic mapping of buffersMove to 64-bit architecture No so trivial ESRF/Linux 2.1 [Red Hat Enterprise Linux 5]Kernel version 2.6.18Copy to very-high-memory: 1.3 GB/s
![Page 26: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/26.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Sarnoff @ ID15
512 x 512 12 bits @ 300 fps 150 MB/sWrite to local disks: 300 MB/sWrite to remote disks: 300 MB/s
NetApp FAS6070NetApp
FAS6070
10 Gbps Ethernet10 Gbps Ethernet
![Page 27: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/27.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
“La Bestia”Dell PowerEdge 2900 Similar to Ecrin/Trenton + 16 GB RAM 4 + 6 = 10 disks 530 MB/s (800 MB/s raw) Super Sarnoff: 2x FOCLA test imageAcquisition rate: ~360 MB/s
Running
![Page 28: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/28.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Turbo Bestia 4 + 6 + 8 = 18 disks Single RAID 0 array 840 MB/s raw 3 RAID 0 arrays + parallel write 1.5 GB/s raw
4x
6x
8x
![Page 29: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/29.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Espia Next Generation
PCI-Express 8x lanes 4 fiber optic links @ 250 MB/s 1
GB/s Full access to 64-bit memory Maximum register compatibility
![Page 30: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/30.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Common CCD acq. Library
Reuse of common code generic procedures + interfaces Software “features” fallback if hardware has limited
capabilitiesHigh level Python implementationCommon TACO CCD interface Interfaces: implicit explicit Image processing and data reduction as plugins (C++) Extensible interface through user-defined “features”
(Maxipix) Generic TANGO interface Collaboration framework (ALBA)
![Page 31: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/31.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Current status
x 10 + 1[+1] (C++)
x 1 (Python)
x 20
x 5 + 1
![Page 32: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/32.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Conclusions
Significant number of supported 2D detectors This number keeps raising … New detectors demand high performance control Experience in high speed acquisitions
Stable low level driver (Sarnoff failed only once…)Functional high level interface
Never-ending project … (sub-projects)Common image acquisition library will help!
![Page 33: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/33.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Acknowledgements
SECAD, S.A. F. Hereson
Instrument Support Group:
J.C. Labiche E. Collet L. Siron J.J. Thevenin D. Pothin C. Ponchut J.M. Rigal J. Clement P. Fajardo
Computing Services:
P. Makijarvi B. Regad P. Pinel L. Garçon F. Andrei D. Gervaise S. Ohlsson B. Lebayle
![Page 34: Fast 2D Detector Control at ESRF beamlines](https://reader035.vdocuments.net/reader035/viewer/2022062304/568140c8550346895dac932c/html5/thumbnails/34.jpg)
A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008
Thank you!