The Mark 5B VLBI Data System

Alan Whitney for the Mark 5 teamMIT Haystack Observatory

20 September 20065th e-VLBI WorkshopHaystack Observatory

The Mark 5 team

Will Aldrich

John Ball

Peter Bolis

Roger Cappallo

Kevin Dudevoir

Brian Fanous

Dave Fields

Chet Ruszczyk

Jason SooHoo

Mike Titus

Dan Smythe

Alan Whitney

Ken Wilson

Mark 5A Data Acquisition System(introduced 2002)

Mark 5A Characteristics

• Direct plug-compatible replacement for 64-track Mark4 or VLBA tape drives

• Record/Playback at rates up to 1024 Mbps• Accept 2 independent ‘8-pack’ disk modules can be used in

‘ping-pong’ fashion for nearly continuous recording• Records 8, 16, 32 or 64 tracks from Mark4 formatter (1024

Mbps max) or VLBA formatter (512 Mbps max)

Current Mark 5A Status

• ~150 Mark 5 systems deployed; >95% of tape systems have been replaced

• >1200 Mark 5A ‘8-pack’ disk modules deployed (>2 PB of storage!); growing rapidly!

• Many stations are now entirely Mark 5A, except in some cases for data which goes to VLBA correlator

• 1 Gbps experiments are now routine for both geodesy and astronomy

• VLBA has mostly converted to Mark 5; all VLBA stations have Mark 5A, but correlator not yet fully outfitted

Mark 5B Characteristics• Full VSI-H compatibility• Same chassis as Mark 5A; uses same disk modules; requires Mark 5B I/O card• 1024 Mbps record/playback• Eliminates need for external formatters, but requires a VSI-H interface• Station Unit capabilities for connection to Mark 4 correlators is designed into

Mark 5B• Extensive built-in phase-cal extraction and state counting on both data record

and data playback• Front-panel status display – 8 tri-color LEDs• DIM and DOM capabilities are separate FPGA downloads• FPGA is programmable via software

Development support from Mark 5 development consortium – BKG, EVN, KVN, JPL, MPI, NASA, NRAO, USNO

DPSCLOCK

DPS1PPS

ROT Clock(Requested Observe

Time)

DOT Clock(Data Observe Time)

QVALID

QDATA

RBS 0

RBS 31

Control

DOM(Data Output Module)

DTS(Data Transmission System)

Figure 1: VSI-H Functional Block DiagramVSI6.DRWARW 21 Jun 2000

ALT1PPS

DPS(Data Processing System)

TransmissionMedium

R1PPS

RCLOCK

80

-pin

MD

R co

nn

ecto

r; all LV

DS

LVDSRS-232

PDATA

BS 0

BS 31

1PPS

CLOCK

DIM(Data Input Module)

DOTMON

TVR TVG

TV0

TV31

TVRTVG

TV0

TV31

32x32C

rossbar

Notes: 1. Shaded items are for illustrative purposes only. 2. PVALID is optionally transmitted from DIM to DOM. 3. PDATA is optionally transmitted from DIM to DOM. 4. Data delay in DOM is required only for storage-based systems. 5. See text for discussion of use of optional use of P/QSPARE1/2 signals. 6. If DIM/DOM in single box, ALT1PPS/DPSCLOCK/DPS1PPS share single MDR-14 connector. 7. This diagram does not show all functions and options -- see VSI-H specification for details.

PVALID

LVDS EthernetTTL

80

-pin

MD

R co

nn

ecto

r; all LV

DS

DAS(Data Acq System)

Compareand

Synchronize

High-orderTime

Delay

ROT1PPS

ROTMON

QCTRLPCTRL

PSPARE1

TTL

(TVGCTRL)

PSPARE2(DOT1PPS)

QSPARE1(DPSCLOCKX)

QSPARE2(DPS1PPSX)

VSI-H VSI-H

VSI-S

VSI-E

Mark 5A/B System Comparisons

Mk5A Mk5B

Data InterfaceEmulates

Mk4/VLBAtape transport

VSI-H(64MHz max clock rate)

Max data rate 1024 Mbps 1024 Mbps

Record modes8, 16, 32, 64

“tracks”1,2,4,8,16,32

bitstreams

Disks Mk5 “8-pack” Same

Chassis Mk5 Same

I/O card Mk5A Mk5B

SS card XF2 XF2

I/O-SS intf Modified FPDP FPDP

Mark 5B Functional Block Diagram

CLOCK

1PPS

PVALID

BS

RCLOCK

QVALID

QDATA

RBS

DPSCLOCK

DPS1PPS

Fan-out

DOTClock

Add NDRheader

Stripheader

Da

ta

FPDPBus

PDATA

32Fan-in

Databuffer(min

256MB)

WriteRead

Xbar

32

Clo

ck

TO

T1P

PS

PCAL/Statecount

PCI

Va

lidity

Readcontrol

VSIIn

VSIOut

LVDS/RS232Conversion

To MOBOSerial Port

Internal Clock

INTCLK

Int

DIM

1PP

SD

OT

1P

PS

CF

INT

TOT1PPS

TO

T1P

PS

DOT1PPS

DPS1PPSXDPSCLOCKX

ALT1PPS

DOMCLK

DOM1PPS

DIM1PPS

ALT2PPS

CFInt

Gen

DOM1PPS

DOMCLOCK

CFINT

DO

M1P

PS

BSCLOCK

DOT1PPS

PVALID

From DIM Select

CLOCK

1PPS

PVALID

BS

RCLOCK

R1PPS

QVALID

QDATA

RBS

DPSCLOCK

DPS1PPS

DIM DOM

Fan-out

DOTClock

Add NDRheader

Stripheader

Da

ta

FPDPBus

PDATA

StreamStor

3232Fan-in

Databuffer(min

256MB)

WriteRead

Delayparams &

CF hdr databuffer

XbarAdd

CF hdr(for SUmode)

32

Clo

ck

TO

T1P

PS

PCAL/Statecount

PCI

Va

lidity

Readcontrol

TVR

TVG

VSIIn

VSIOut

LVDS/RS232Conversion

To MOBOSerial Port

Internal Clock

INTCLK

Int

DIM

1PP

SD

OT

1P

PS

CF

INT

/RO

TIN

T

TOT1PPS

TO

T1P

PS

DOT1PPS

DPS1PPSXDPSCLOCKX

ALT1PPS

DOMCLK

DOM1PPS

DIM1PPS

INTCLK

ALT2PPS

CFInt

Gen

DOM1PPS

DOMCLOCK

CFINT

DO

M1P

PS

BSCLOCK

DOT1PPS

PVALID

From DIM Select

ROT1PPS (BOCF)

TVG

TVR

INTCLK

SPARE1SPARE2

Mark 5B I/O Board

FPDP VSI Output VSI Cntl/Mntr

VS

I In

put

256MB memoryPCI Connector

Mark 5B Status

• Checkout of Mark 5B is complete; first rev of DIM software has been released

• Mark 5B is in regular use at Westford antenna; also used in large mm experiment in April 2006

• Mark 5B is interfaced to Haystack Mark 4 correlator and in routine production use

• ~30 Mark 5B I/O boards have been built and tested – will be distributed to Mark 5 development consortia members in near future

• Mark 5B can be ordered from Conduant Corp in Loveland, CO

 

Reasons to adopt Mark 5B

• Eliminate need for expensive external formatters; particularly important for new stations or stations without existing Mark 4 or VLBA formatters

• DBEs and dBBCs will require VSI-H interface• With a 14-BBC Mark4, up to 1792 Mbps can be recorded with two

parallel Mark 5B systems; current systems can only generate a maximum of 1024 Mbps of formatted data due to Mk4 formatter limitation

– requires modification of Mark 4 formatter for VSI-H• Extensive phase-cal extraction and state counting capabilities for better

diagnostics and better system calibration• Replace unreliable Station Units on Mark 4 correlators; SU capability

is built into Mark 5B

Mark 5A/B Compatibility

• Mark 5B can play only Mark 5B recordings (VSI format in/out)• Upgraded Mark 5A (“Mark 5A+”) can play:

- All Mark 5A recordings- Mark 5B recordings made in almost all modes;

playback is in VLBA-track-format• Mark 5A+ design is complete and tested, but is needed only on

correlators that do not yet support Mark 5B• Existing Mark 5A systems can be upgraded to Mark 5A+ with

new Xilinx download and upgraded software

 

Mark 5B+ (2048Mbps)

• Conduant has introduced an upgraded StreamStor (dubbed “Amazon”) that supports up to ~3 Gbps on FPDP2 interface

• Mark 5B I/O card has been designed to support input VSI-H clock rate of 64MHz, as well as FPDP2 compatibility, to support max recording rate of 2048 Mbps with Amazon board

• May be desirable to record across 2 disk modules (16 disks) simultaneously• Playback is limited to 1024 Mbps• Recordings made on Amazon will be playable on standard Mark 5B system• No promises yet on date of general availability, though hope in a few

months

 

Mark 5A/B/B+ System Comparisons

Mk5A Mk5B Mk5B+

Data Interface Emulates Mk4/VLBA

tape transport

VSI-H(64MHz max clock rate)

VSI-H(64MHz max clock rate)

Max data rate 1024 Mbps 1024 Mbps 2048 Mbps

Record modes 8, 16, 32, 64“tracks”

1,2,4,8,16,32bitstreams

Same as Mk5B

Disks Mk5 “8-pack” Same Same

Chassis Mk5 Same Same

I/O card Mk5A Mk5B Mk5B

SS card XF2 XF2 Amazon

I/O-SS intf Modified FPDP FPDP FPDP2(clocks on both

edges

Mark 5 Upgrade Costs

Target

Existing

Mk5A Mk5B Mk5B+

0 $18K ~$20K ~$23K

Mk5A - ~$3K(Mk5B I/O)

~$12.5K

(Mk5B I/O plus Amazon)

Mk5B - - ~$9.5K

(Amazon)

Note: Does not include external cabling costs, typically a few hundred dollars

Mark 5B Software

• Current Mark 5B DIM software is built into same program as Mark 5A, but is renamed Mark5B

• New program called dimino is being written by Kevin Dudevoir exclusively for Mark 5B DIM support– includes advanced network interface management and VSI-E support

– includes phase-cal processing support

• Mark 5B DOM support on Haystack/MPI/USNO correlators is separate program called domino

Disk-Media Status

• Hard disk price vs capacity/performance continues to drop- Now at or below ~$0.50/GB and continues to drop

(Mark 4/VLBA tape was ~$2.00/GB)

• 400 GB disks are becoming more affordable – 8-pack of 400GB disks comparable to ~11 VLBA/Mark 4 tapes

• 750 GB disks are now available – 8-pack of 750GB disks comparable to ~21 VLBA/Mark 4 tapes;

~26 hours @ 1 Gbps unattended!• 1 TB disks will soon be available

-2

-1

0

1

2

3

4

5

6

7

1980 1985 1990 1995 2000 2005 2010

Year

log($/GB)

Disc Drive Street Prices

Computer Tape Media

1998 IBMDisc Projection

Mark IV/VLBA/K4 Media

S2, S3 Media

Disc industryProjections

1998 NSICDisc Projection

LTO Media Projection

c

Tape vs. Disc Price Comparison

Disk-Media Reliability

• Based on statistics collected at Haystack, average disk drive failure rate is ~0.5% per year

• Failure rate of Hitachi 250GB has been higher than average, but may now have been fixed

• Conduant is shipping these disks in assembled Mark 5 disk modules

• Disk reliability at high was investigated in Mark 5A tests on Mauna Kea in early 2006:

Tested disk drive types were:Maxtor 300-GB Model 7L300R0Seagate 300-GB Model ST3300831AWestern Digital 320-GB Model WD3200SB-01KMA0Hitachi 250-GB Model HDS722525VLAT80

→Only the Hitachi’s sustained good performance at 14000ft(However, all disks recovered performance when returned to low altitude)

Further development of Mark 5

• Considering direct 10GigE interface into StreamStor board to support recording up to 4Gbps

• Considering direct 10GigE data interface from DBE at 4Gbps

Mark 4 Station Unit emulation

 

Station delay model

Applydelaymodel

Phase-calextraction

(16 tones/chn)

Statecounting

Playback

Phase, delay model(per channel)

To correlator(16 channels)

Insert

Station unit function

Disks

Mark 4 Station Unit