High Bandwidth Data Acquisition and Network Streaming in VLBI

Jan Wagner Guifreacute Molera et alTKK Metsaumlhovi Radio Observatory

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 2

bull EXPReS funds to connect stations at 1G10Gbull 1G fiber up to capacity with Tsunami UDP (2006) or JIVE vlbi_udp (2007) real-time transport

protocols

bull ToDo new system for gt1 Gbps rates

Basic e-VLBI Data AcquisitionIntroduction

bull Basic 8ndash16 stations rates 128ndash512 Mbits 24h 1ndash4 TBstation all to correlatorbull VLBI observed galaxyrsquos noise recorded to disks shipped to correlatorbull e-VLBI transfer station data via fast research networks two alternatives

bull post-observation record locally transfer to correlator laterbull real-time stream to the correlator for temp storage or processing

1Apr08 JIVE vlbi_udp incoming aggregate rate record

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 3

Goals for High(er) Bandwidth DAQbull Observe a broader radio spectrum

bull broader spectrum =gt higher data ratebull correlation result and final image improvesbull high EXPReS goal 1248 Gbps per station

bull Higher rates =gt need at leastbull new analog-to-digital converter system (current 10+ years old)bull faster higher capacity localremote data storage systemsbull suitable transport protocols - stations to correlator

bull Ultimately e-VLBI data processing in real-timebull get rid of high capacity station data storagebull JIVE hardware correlator successful in many le512 Mbps real-time runsbull distributed software correlators more natural with data streaming

Introduction

C Phillips e-VLBI Scientific Benefits ATNF CSIRO memo 2004

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 4

High Bandwidth DAQ DevelopmentDAQ steps

Digitize wide-band signal ndash Stream over 10G ndash Capturestore with PC(s)

Development Environment

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5

DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform

bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards

Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC

=gt 32 Gbps ADC data 20 Gbps net

bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development

bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6

DAQ ndash Streamingbull TCP is not a good streaming choice

bull No unreliable best-effort delivery no delivery time constraints

bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient

35kYears1Tbps8Kbull congestion control none data thinning change sampling mode

bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)

=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh

overall iBOB test designhellip =gt

~ 1 f

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7

DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8

DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on

~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet

=gt Inexpensive 1ndash4 Gbps data storage system

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9

DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid

bull Essential for performance in 10G stream recordingbull 10G NIC throughput

bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers

=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput

bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads

=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 2

bull EXPReS funds to connect stations at 1G10Gbull 1G fiber up to capacity with Tsunami UDP (2006) or JIVE vlbi_udp (2007) real-time transport

protocols

bull ToDo new system for gt1 Gbps rates

Basic e-VLBI Data AcquisitionIntroduction

bull Basic 8ndash16 stations rates 128ndash512 Mbits 24h 1ndash4 TBstation all to correlatorbull VLBI observed galaxyrsquos noise recorded to disks shipped to correlatorbull e-VLBI transfer station data via fast research networks two alternatives

bull post-observation record locally transfer to correlator laterbull real-time stream to the correlator for temp storage or processing

1Apr08 JIVE vlbi_udp incoming aggregate rate record

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 3

Goals for High(er) Bandwidth DAQbull Observe a broader radio spectrum

bull broader spectrum =gt higher data ratebull correlation result and final image improvesbull high EXPReS goal 1248 Gbps per station

bull Higher rates =gt need at leastbull new analog-to-digital converter system (current 10+ years old)bull faster higher capacity localremote data storage systemsbull suitable transport protocols - stations to correlator

bull Ultimately e-VLBI data processing in real-timebull get rid of high capacity station data storagebull JIVE hardware correlator successful in many le512 Mbps real-time runsbull distributed software correlators more natural with data streaming

Introduction

C Phillips e-VLBI Scientific Benefits ATNF CSIRO memo 2004

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 4

High Bandwidth DAQ DevelopmentDAQ steps

Digitize wide-band signal ndash Stream over 10G ndash Capturestore with PC(s)

Development Environment

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5

DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform

bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards

Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC

=gt 32 Gbps ADC data 20 Gbps net

bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development

bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6

DAQ ndash Streamingbull TCP is not a good streaming choice

bull No unreliable best-effort delivery no delivery time constraints

bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient

35kYears1Tbps8Kbull congestion control none data thinning change sampling mode

bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)

=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh

overall iBOB test designhellip =gt

~ 1 f

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7

DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8

DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on

~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet

=gt Inexpensive 1ndash4 Gbps data storage system

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9

DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid

bull Essential for performance in 10G stream recordingbull 10G NIC throughput

bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers

=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput

bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads

=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 3

Goals for High(er) Bandwidth DAQbull Observe a broader radio spectrum

bull broader spectrum =gt higher data ratebull correlation result and final image improvesbull high EXPReS goal 1248 Gbps per station

bull Higher rates =gt need at leastbull new analog-to-digital converter system (current 10+ years old)bull faster higher capacity localremote data storage systemsbull suitable transport protocols - stations to correlator

bull Ultimately e-VLBI data processing in real-timebull get rid of high capacity station data storagebull JIVE hardware correlator successful in many le512 Mbps real-time runsbull distributed software correlators more natural with data streaming

Introduction

C Phillips e-VLBI Scientific Benefits ATNF CSIRO memo 2004

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 4

High Bandwidth DAQ DevelopmentDAQ steps

Digitize wide-band signal ndash Stream over 10G ndash Capturestore with PC(s)

Development Environment

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5

DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform

bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards

Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC

=gt 32 Gbps ADC data 20 Gbps net

bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development

bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6

DAQ ndash Streamingbull TCP is not a good streaming choice

bull No unreliable best-effort delivery no delivery time constraints

bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient

35kYears1Tbps8Kbull congestion control none data thinning change sampling mode

bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)

=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh

overall iBOB test designhellip =gt

~ 1 f

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7

DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8

DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on

~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet

=gt Inexpensive 1ndash4 Gbps data storage system

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9

DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid

bull Essential for performance in 10G stream recordingbull 10G NIC throughput

bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers

=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput

bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads

=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 4

High Bandwidth DAQ DevelopmentDAQ steps

Digitize wide-band signal ndash Stream over 10G ndash Capturestore with PC(s)

Development Environment

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5

DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform

bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards

Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC

=gt 32 Gbps ADC data 20 Gbps net

bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development

bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6

DAQ ndash Streamingbull TCP is not a good streaming choice

bull No unreliable best-effort delivery no delivery time constraints

bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient

35kYears1Tbps8Kbull congestion control none data thinning change sampling mode

bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)

=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh

overall iBOB test designhellip =gt

~ 1 f

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7

DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8

DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on

~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet

=gt Inexpensive 1ndash4 Gbps data storage system

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9

DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid

bull Essential for performance in 10G stream recordingbull 10G NIC throughput

bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers

=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput

bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads

=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5

DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform

bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards

Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC

=gt 32 Gbps ADC data 20 Gbps net

bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development

bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6

DAQ ndash Streamingbull TCP is not a good streaming choice

bull No unreliable best-effort delivery no delivery time constraints

bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient

35kYears1Tbps8Kbull congestion control none data thinning change sampling mode

bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)

=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh

overall iBOB test designhellip =gt

~ 1 f

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7

DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8

DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on

~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet

=gt Inexpensive 1ndash4 Gbps data storage system

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9

DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid

bull Essential for performance in 10G stream recordingbull 10G NIC throughput

bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers

=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput

bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads

=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6

DAQ ndash Streamingbull TCP is not a good streaming choice

bull No unreliable best-effort delivery no delivery time constraints

bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient

35kYears1Tbps8Kbull congestion control none data thinning change sampling mode

bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)

=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh

overall iBOB test designhellip =gt

~ 1 f

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7

DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8

DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on

~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet

=gt Inexpensive 1ndash4 Gbps data storage system

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9

DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid

bull Essential for performance in 10G stream recordingbull 10G NIC throughput

bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers

=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput

bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads

=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7

DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8

DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on

~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet

=gt Inexpensive 1ndash4 Gbps data storage system

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9

DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid

bull Essential for performance in 10G stream recordingbull 10G NIC throughput

bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers

=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput

bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads

=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8

DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on

~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet

=gt Inexpensive 1ndash4 Gbps data storage system

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9

DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid

bull Essential for performance in 10G stream recordingbull 10G NIC throughput

bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers

=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput

bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads

=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9

DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid

bull Essential for performance in 10G stream recordingbull 10G NIC throughput

bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers

=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput

bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads

=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording

Basic Data Acquisition System

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10

Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN

bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements

bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)

Basic Data Acquisition System

=gt Improved rates for ldquooff-linerdquo e-VLBI

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11

Summary

Summarybull Many European VLBI stations start to reach ~1 Gbps

streaming with the old hardware

bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP

bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID

bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12

QuestionsAnswersbull Contact information

Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi

bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi

bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642

10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12

QuestionsAnswersbull Contact information

Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi

bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi

bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642