ku high energy physics rice project - phys.hawaii.eduidlab/project_files/salt/docs/rice_daq… ·...

RICE 2 Specification 2/21/2005 1 of 37

RICE DAQ 2

KU High Energy Physics RICE Project

Instrumentation Design Laboratory University of Kansas

John Ledford Ken Ratzlaff

February 2005


Revision History Revision Date Author Comments

x 2/21/05 JPL Yet another revision of the base initial document.

Table of Contents 1. Overview & Introduction...................................................................................................... 5

1.1 Purpose of This Document .......................................................................................... 5 1.2 Quick Overview............................................................................................................ 5 1.3 Design Constraints ...................................................................................................... 5 1.4 Collected Useful Facts and Information ....................................................................... 6

1.4.1 General Information.............................................................................................. 6 1.4.2 If We Use Ice Cube Holes .................................................................................... 6

2. Whole System Diagram & Functional Description .............................................................. 7 3. Clusters & Antennas (The Outdoor Part) ............................................................................ 9 4. RF Signal Transport from Antennas to DAQ..................................................................... 12

4.1 Optical Fiber Modulator / Demodulator ...................................................................... 12 4.2 RF Signal Conditioning Transmit Side....................................................................... 12 4.3 RF Signal Conditioning Receive Side........................................................................ 13

5. Digitization ........................................................................................................................ 14 6. L0 Triggering System – Discrimination per Antenna......................................................... 16 7. L1 Trigger System – Cluster Level Triggering................................................................... 18

7.1 Packaging.................................................................................................................. 18 7.2 Cluster Trigger Board Overview................................................................................. 19 7.3 Sample Clock & Divided Clock .................................................................................. 20 7.4 Buffering and Distribution of Signals.......................................................................... 20 7.5 Triggering Functions.................................................................................................. 20 7.6 FPGA Program Storage and Provisioning ................................................................. 20

8. L2 Trigger / HW Trigger System ....................................................................................... 21 9. GHz Source & Sync Pulse - Generation & Distribution..................................................... 23 10. GPS Timing System...................................................................................................... 27 11. Computers in the System.............................................................................................. 29

11.1 SW Trigger Computer System................................................................................... 29 11.2 Event Storage Computer System .............................................................................. 29 11.3 Control Computer ...................................................................................................... 29

12. Power System............................................................................................................... 30 12.1 Rack View.................................................................................................................. 30 12.2 Control Card .............................................................................................................. 31 12.3 Distribution Card ........................................................................................................ 32

13. In Ice Power & Monitoring System ................................................................................ 34 14. Science-y Things........................................................................................................... 36

14.1 Unbiased Trigger ....................................................................................................... 36 14.2 Others ?..................................................................................................................... 36

15. Board ID Table .............................................................................................................. 36


16. Acknowledgements ....................................................................................................... 37


Table of Figures Figure 1: Overall System Diagram............................................................................................. 7 Figure 2: Outdoor Overview of RICE2 ....................................................................................... 9 Figure 3: Antenna Clusters in Ice Bore Holes .......................................................................... 10 Figure 4: RF Signal Conditioning on Transmit Side ................................................................. 12 Figure 5: RF Signal Conditioning on Receive Side .................................................................. 13 Figure 6: One Digitizer / Antenna Channel Block Diagram ...................................................... 14 Figure 7: L0 Discrimination Diagram........................................................................................ 16 Figure 8: L0 Discriminator Detailed Diagram ........................................................................... 17 Figure 9: Cluster Trigger Processing Subsystem..................................................................... 18 Figure 10: Cluster Trigger Processing Unit Block Diagram...................................................... 19 Figure 11: HW Trigger Block Diagram ..................................................................................... 21 Figure 12: ADC Sync Pulse ..................................................................................................... 23 Figure 13: Overview of Timing Generation / Distribution Packaging ........................................ 24 Figure 14: Rear View of Timing Unit ........................................................................................ 24 Figure 15: Detailed Diagram of Generator Source Card .......................................................... 25 Figure 16: Distribution Card Diagram....................................................................................... 26 Figure 17: RICE GPS Rack Mount System.............................................................................. 27 Figure 18: GPS Timing System Block Diagram ....................................................................... 28 Figure 19: Power System Rack View....................................................................................... 31 Figure 20: Power System Control Card Block Diagram ........................................................... 32 Figure 21: Power System Distribution Card Block Diagram..................................................... 33 Figure 22: In Ice Electronics Block Diagram ............................................................................ 34 Figure 23: In-Ice Voltage / Current Measurements & Control Switch....................................... 35


1. Overview & Introduction

1.1 Purpose of This Document

Document the design for the RICE 2 DAQ hardware. This document contains all hardware related documentation, including some of the gruesome gory details. This is this week’s design. If this document is more than 1 week old, be sure to check for changes. Too bad this joke isn’t as funny as it should be.

1.2 Quick Overview

Utilizing an Analog RF transmission over fiber optics, we plan to put full bandwidth signals from the ice to digitizers in the central DAQ. The use of these RF modulators will allow for vast system complexity reductions over past versions of this system design. In the ice will be an antenna, amplifiers, and a modulator. The in ice system will transmit over single mode fiber to the surface and into the counting house. There digitizers will continuous sample the signal into a memory buffer and a hardware trigger will look for coincidences. When a trigger is formed the memory buffers will be read out and processed / stored. The intention is that there will never be a need to actually stop the digitizers and the system is being designed to handle some high trigger rates.

1.3 Design Constraints

The ice is cold, around -50oC. Thus anything in the ice will need to operate at -55oC. If we use IceCube holes, we will have to be able to stand re-freeze which is a tough situation. We need to be able to afford it. The timing resolution for channel-channel event reconstruction needs to be < 5ns. The system will be flexible and expandable to include any reasonable (and finite) number of clusters. The current design plan is for a maximum of 400 measurement points (clusters). This is a soft maximum and with further testing / research and a LOT more money could likely be raised.


1.4 Collected Useful Facts and Information

1.4.1 General Information

• The propagation delay for RF signals through ice is 4.7us / km or around 4.7ns/meter

1.4.2 If We Use Ice Cube Holes

• Drill is drilling 2450m deep • Drill holes are > 45cm 36 hours after turned over to deployment team. • Drill vertical is within 50 feet (15.2m) from center of hole. • Everything has to sink in cold water. • The temperature in the sub-surface trenches is around –70oC.


2. Whole System Diagram & Functional Description A

nt C

lust

er (

N)

Power System(1 Port / Cluster)

Status data& control

PC Win 2K

SoftwareTrigger PCor Server

Farm

EventStorage PCLinux (Big

Hard Drive)

x 4 Antennas

RF SignalConditioning(1 / Antenna)

In Ice StatusMonitoring ?

(1 Port / Cluster)

Digitization &Cluster Triggering

(1 / Cluster andhas 4 digitizers) E

ther

net S

witc

h

HW TriggerSystem

x all clusters

1 GHz Source &Sync Pulse

GPS TimingSystem

for

N c

lust

ers

Eve

nt ID

There are twoethernet

systems. Onefor trigger

packets fromL1 to SW

trigger and onefrom readout to

L1 to getsample data.

Figure 1: Overall System Diagram

Figure 1 shows an overall system diagram. There are a number of clusters that have 4 antennas in each cluster (section 3). Clusters are intended to be used for directional vetoes and local coincidence to beat down thermal noise.


The signals are brought to the surface over fiber and are demodulated, filtered, amplified, and attenuated as needed (section 4). These are fed into the digitizer which will also be the discriminator (see section 5). The digitizer feeds the discriminator outputs to the L1 trigger board in the cluster (section 7). This board looks for cluster wide patterns and makes a decision in the cluster about whether to pass a trigger pulse upstream. In the HW trigger system (section 8), the L1 trigger board will pulse a connection to the L2 HW trigger board. If an L2 trigger is found, all digitizers read out a section of data that are collected in the L3 SW trigger system. This system does physics cuts and makes decisions whether to keep it. If it’s decided to keep the event, it is forwarded to the event storage PC to keep in deep storage until it is sent home to look at further (section 11.2). If an L2 trigger is found, a GPS unit (section 10) timestamps that global trigger with UTC time and then forwards the information via ethernet to the L3 trigger system. There are 1 GHz low phase noise clock source & distribution buffers in another unit (section 9). The timing unit also generates a sync pulse that is used to line up the digitizers to a common time. This system has enough copies for every cluster digitizers & cluster trigger system. There are provisions for the cluster triggering unit to send L1 trigger information via ethernet to a software trigger system if we end up needing it (section 11.1). It is not intended as the primary trigger at this time but is used as the L3 trigger. This trigger path will be separate from the readout path as to minimize latency and not have readout stopping the trigger system. There is a PC that is used to control the system and provision information as needed in the system (section 11.3). It collects status information and is the primary human interface. Log files will be created, and system performance measured. Ethernet is the primary communications system in this detector. Gigabit will be most likely used since everything is local and can be done over copper. Copper Gigabit switches are dirt cheap and falling in price daily.


3. Clusters & Antennas (The Outdoor Part)

Single Mode Fiber for RF Transmission

In-Ice Electronics &RF Modulators &

Low Noise Amps &Antennas, in pressure

housings

CountingHouse /

DAQ

240 VAC Power Cables

Expected to have 3 to 5clusters per borehole &

4 antennas / cluster

Extra Single Mode FiberJunction Box

Figure 2: Outdoor Overview of RICE2

A single cluster’s worth of the outdoor part of the DAQ is shown in Figure 2. The main DAQ is in the counting house. From the counting house cables are run in a trench to a junction box. Another set of cables are run from the junction box down the antenna borehole to the electronics. The antennas in the cluster are all about 5m apart to give enough separation to give direction-of-travel veto capability. These are likely to be separate housings joined to the main electronics housing via coax. This is shown conceptually in Figure 3. This figure also shows an example of 4 antenna clusters all 100m apart.


Fib

er

Cop

per

BoreHole

Clu

ster

Clu

ster

Clu

ster

Clu

ster

100m

100m

100m

100m

Cluster Closeup Surface

This drawingassumes 4clusters per

borehole with100m spacing.

To DAQ

RICE1 antenna modules & LNA

Mod

ulat

ors,

Pow

er S

uppl

y,an

d B

ias

Ts

New pressure housing

Clu

ster

Very good coax

Figure 3: Antenna Clusters in Ice Bore Holes

Each cluster is supplied a power feed and an optical cable. The power is 240VAC so that there is very little power loss in the cabling. The power cable is intended to be between 12 to 18 gauge. There will be a simple linear power supply which will provide the power necessary for the RF modulator and Low Noise Amplifier (LNA). The LNA draws about 2W and the modulator draws about 3W. This gives a power draw of around 6W / antenna (due to using a linear regulator) or 25W / cluster. Assuming 75% efficiency, this is around 33W total input power. At 240VAC this gives a current draw of around 138mA. 14 Gauge wire has 8.3 ohms / km. In a 2km deployment there is 4km of conductor. 33 ohms at 140mA = only a 4.5V drop. We may use 18 gauge which would have only a 12V drop. We plan to have a circuit on the power supply card that controls and measures the power to each device and reports over the unused fiber the voltage, current, pressure, and temperature.


Each cluster is supplied a power feed and a twin fiber optical cable. The power is supplied by the power distribution system (section 12) which is able to switch power on and off, and measure the voltage / current being supplied to the cluster. The unused fiber is a good way to get voltage / current / pressure / temperature measurements from the cluster. If we can manage a bi-directional link over one fiber then we can also control the power interfaces. We plan to have a simple PIC processor and sensors to report status data and provide some rudimentary control over the system in case something goes wrong and we have to troubleshoot or shut down part of the system.


4. RF Signal Transport from Antennas to DAQ

4.1 Optical Fiber Modulator / Demodulator

There are a number of vendors offering RF-Over-Fiber solutions. The three we are currently looking at are Fiber Span (http://www.anacomsystems.com/), Miteq (http://www.miteq.com/), and Photonuum (www.photonuum.com). Each vendor claims to be able to transport around 2-3GHz of analog bandwidth with an impressive amount of dynamic range. Tests are underway with test systems and multi kilometers of fiber to see how well they perform. Initial tests indicate that RF over fiber should work really well.

4.2 RF Signal Conditioning Transmit Side

PowerLimiter

BandPassFilter

LowNoiseAmps

RFModulator

WD

M F

ilter

s

DipoleAntenna

PowerLimiter

BandPassFilter

LowNoiseAmps

RFModulator

DipoleAntenna

... ... x4 ... ...

Clu

ster

Fib

er

Figure 4: RF Signal Conditioning on Transmit Side

The antenna feeds a power limiter. The purpose of this is to limit the maximum amplitude of the output signal so we do not damage components downstream. The output of this limiter feeds a band-pass filter which limits the band of interest to the same band as our amplifier’s bandwidth. Upstream we can further band limit as desired, but this one is intended to cut off below 200MHz and above whatever the amplifier can handle. The low noise amplifier is needed to get above the noise floor of the RF modulator. The modulator can take pure thermal noise of around -174dBm and the output of the modulator is around -135dBm. This noise floor is inherent in the transmission equipment and requires us to get our signals well above this output level. By adding 50-60dB of gain, our signals are about 20dB above the demodulator noise floor and thus the modulators should not add any significant noise to our signal.


The modulators have a tremendous dynamic range and we hope to sit in the low end of their transmission capability, but well above their noise floor. A test modulator / demodulator have been ordered by Gary Varner at University of Hawaii and characterization should happen in early Feb 2005.

4.3 RF Signal Conditioning Receive Side

RFDemodulator

InlineAttenuatorsAmp

Clu

ster

Fib

er RFDemodulator


RFDemodulator


RFDemodulator


WD

M F

ilter

s

To

Dig

itize

r In

puts

BandpassFilters

BandpassFilters

BandpassFilters

BandpassFilters

Figure 5: RF Signal Conditioning on Receive Side

The RF signals for each of the four antennas are multiplexed over the one fiber. There is one modulator and demodulator per antenna in the cluster. These systems have different wavelengths and are sent over one fiber. At the surface there is a WDM splitter / filter. The demodulators to give us coaxial outputs for the RF signal of each antenna. The signal coming in is very tiny and is amplified again with a large amplifier. Then inline attenuators are added to tune the noise level to the input range of the A/D. The A/D has a full scale input of 600mVpp. (All RF numbers are whacked and need Dave Besson’s input to correct). In rough numbers, the incoming signal is around -170dBm. If we want to feed 4 bits of noise into the A/D, then the noise level needs to be around 40mV into 50 ohms or around -15dBm. Since the RF fiber transport has a nominal gain of 0dB, we need around 155dB of gain total, somewhere in the system. We have around 60dB in the ice, so we need approximately another 95 dB in the signal conditioning for the receiver.


The resulting massaged signals are fed into the digitizers.

5. Digitization

The digitizers are closely tied together with the cluster trigger system discussed in 6. See that section for a lot more detail on triggering aspects of the cluster level processing.

Discriminator

2GSa/SecADC

FPGA

Buffer Memory

Global Trigger InputSample Data To Cluster Trigger Card

Discriminator Trigger Outputs

Analog Input

1GHz Sample ClockDigitizer Sync Signal

Config from Cluster Trigger Card

Figure 6: One Digitizer / Antenna Channel Block Diagram

There are two primary functions on the digitizer board. The first is discrimination. There are programmable discriminators that turn the RF signal into a trigger pulses. See section 6 and Figure 8 for more details. The second major function is the digitization itself. The chip we are planning to use is the ADC08D1000 from National Semiconductor. This is a dual channel 1 Gsa/sec A/D converter. The chip has a built in ability to use both channels on one input and get an effective 2 Gsa/sec A/D converter. This data will come into the FPGA which will do short term buffering and put the data into buffer memory. The current plan is to use a DDR400 SODIMM laptop memory DIMM. The plan is to use a 1 Gbyte module which would provide us with 500ms of sample storage.


The FPGA also looks for incoming global triggers. If a global trigger is found, then the FPGA arranges to pull the samples out of memory (without stopping the writing to memory) and sends them to the cluster processing FPGA which will send them to the main DAQ via gigabit ethernet. Once data is read out, a time correction will be made in software to take out the transit time of the fiber so that the system can then do event reconstruction. The time correction values will be measured on install with OTDRs and will also be calibrated through pinging the array. For SW triggers the correction is likely to be made in the Cluster Trigger Processing Unit. For HW triggers the corrections will be made by the event PC.


6. L0 Triggering System – Discrimination per Antenna

The current plan is to build the L0 Discriminator into the same board as the digitizer as shown in Figure 6. This would involve using Gary Varner’s RF triggering systems knowledge and circuitry developed for ANITA. The plan would be to power split the signal as shown below in Figure 7 between the discriminator circuitry and the ADC. The discriminator portion of the signal is split again for trigger banding.

SMAInput

Pow

er S

plitt

er

Pow

er S

plitt

er

ADCInput

FrequencyBanding

FrequencyBanding

Discriminator (DAC &High Speed Comparator)


To

L1 T

rigge

r P

roce

ssin

g C

ard


Figure 7: L0 Discrimination Diagram

A little more detail is shown in Figure 8. Each discriminator section is made up of a programmable reference (through a D/A converter) and a high speed comparator. First is banding the trigger into two different energy level detectors. This will allow us to say if the signal has low and high frequency energy. For the high frequency detector, we need to use a tunnel diode to help shape the pulse into a signal we can more easily detect. Also included is a “very high level” detector that will allow us to see when a very high signal is present. This indicates that there is a high background source such as a satellite or some other piece of equipment that is killing us. Each of the discriminator outputs feed into the central trigger unit processing FPGA for making L1 trigger decisions.


D/A

SignalInput

HP LP

Tunnel Diode DetectorProvisioned By System

L0 Trigger Output Hi

D/A

SignalInput

HP LP

Provisioned By System

L0 Trigger Output Low

D/A Provisioned By System

L0 Trigger Super High

Figure 8: L0 Discriminator Detailed Diagram


7. L1 Trigger System – Cluster Level Triggering

7.1 Packaging

The digitizers need to be packaged with a trigger card to make cluster level trigger decisions. This packaging is show below in Figure 9. Since we are planning 4 antennas / cluster and a high level / low level digitizer per antenna, we will have 8 digitizers in a group. The cluster trigger processing unit talks via ethernet to the rest of the system.

(8) Digitizers

(1) Cluster Trigger Processing Unit

Analog Input SMA Connectors

SW L1 Trigger Out& Provisioning Input

Digitizer Sync Pulse Input(LVPECL over SMA)

GHZ Sample Clock Input(LVPECL over SMA)

Global Trigger Input, HW L1Trigger Out, and Cluster IDnumber.

Ch

3 H

igh

Ch

4 H

igh

Ch

2 H

igh

Ch

1 Lo

w

Ch

1 H

igh

Ch

2 Lo

w

Ch

3 Lo

w

Ch

4 Lo

w

10 conductor twistedpair ribbon cable

Sample Data Readout

Figure 9: Cluster Trigger Processing Subsystem


7.2 Cluster Trigger Board Overview

L1 Trig Out

Global Trig In

Sync In

GHZ Clock In

Ethernet

L1TriggerFPGA

GBeMAC/PHY

Div

8

Reset125 MHz

Clk

Buf

fer

to D

igiti

zers

to D

igiti

zers

to D

igiti

zers

Discriminator Outputsfrom Digitizers

Clk

Buf

fer

Clk

Buf

fer

to D

igiti

zers

Clk

Buf

fer

Program and provisioneach of the FPGAs on thedigtitizer boards

EthernetGBeMAC/PHY

Figure 10: Cluster Trigger Processing Unit Block Diagram


7.3 Sample Clock & Divided Clock

The main clock in the system will be the 1GHz sample clock. This clock will be used for the digitizers directly and will be divided down to 125MHz and will provide an 8ns “timestamp” for the software trigger function if implemented. Also, the divided clock will be used to run logic inside the FPGA. It will be divided down on the cluster trigger processing subsystem. The sample clock is buffered and sent to the digitizers for the cluster.

7.4 Buffering and Distribution of Signals

Some signals are very sensitive to propagation delay and skew. Care will be taken in all buffering and distribution to keep all signal propagation delays the same so every digitizer will get the signal at the same time. The global trigger input signal, the sync input, the 1 GHz sample clock input, and the divided clock will all be buffered and distributed.

7.5 Triggering Functions

The FPGA on the CTPU will provide local coincidence triggering based on coincidence levels and direction of travel information. Also a provision will be made that when a cluster trigger decision is made, a packet could be sent out via ethernet for a SW trigger system.

7.6 FPGA Program Storage and Provisioning

The CTPU provides an excellent place to store the FPGA code for the digitizers in this group. So, there will be some form of flash memory such as a USB thumb drive or compact flash card, or just regular flash that can be written by the software to store the programming information. The CTPU software will monitor the FPGA status and when needed will initiate and reprogram the FPGAs in the digitizer cards. The CTPU software will also provision the digitizer cards as needed.


8. L2 Trigger / HW Trigger System

Conceptually the Level 2 HW trigger system is very simple. The block diagram is shown in Figure 11.

Global Trig Out to Clusters

Ethernet

L2 Trigger FPGA

RABBIT10Mbit

L1 T

rigge

r In

puts

Fro

m C

lust

ers

Event ID# Out

Figure 11: HW Trigger Block Diagram

There are a pair of signals that every cluster needs. First is the global trigger decision, which is made on the L2 trigger board, and the other is the trigger output from each cluster which need to get to the L2 trigger board. One problem is that cabling and connectors are expensive. Consensus was sought on how to best cable this and it seems the best answer is to use twisted pair ribbon cables. On the HW trigger board there are twenty 50 pair ribbon cables. Each 50 pair cable is split into five 10 pair cables. Each of the 10 pair cables goes to a cluster. Thus, this board is good for 100 clusters. The FPGA has differential LVPECL drivers and termination built into the chip. In order to save costs, we are planning to use these drivers for signals over the ribbon cable. We have hopes this will work well since physics folks use differential ECL over ribbon all the time. The ribbon cable will have the global trigger as one pair and the L1 trigger as another pair. This leaves 6 conductors left over. There are two conductors that are not connected and not used at this time. One conductor is going to be used as a “presence detect”. There is a pull up resistor on the HW trigger board and if a cluster is plugged in, the software can note that and act accordingly. There are three conductors with 8 voltage levels possible on them. An A/D converter on the cluster board will help the software decode the cluster ID number. Three octal bits can convey 512 unique states, which is more than we are currently planning on the “big” version.


There is a 10 bit event ID number output so that other systems can tag the trigger with the associated data. Meaning the ID number will be updated before the trigger is sent out. The boards that get the trigger will save the event ID number and ship that with the corresponding readout data (GPS time or sample data). This is carried over a standard 20 pin ribbon cable and can jump from unit to unit to unit. The hardware is simple, but the trigger logic in the FPGA will not be. The intention is to use a huge FPGA (the 2 million gate Xilinx Virtex 2 XC2V2000-5FG896C) and allow very complex trigger algorithms to be evaluated. The intention is for one of the physics folks to write this logic. Unlike the RICE 1 HSV board, this system will have plenty of time to make a decision and thus more options are available. For the science needs, there will also be an unbiased trigger logic system in the FPGA. This will be provisionable through the GUI for 1Hz, 10Hz, 100Hz, etc. as well as a “trigger now” button. An ethernet jack is on the board for provisioning of the FPGA and to report when a trigger occurs. The main L3 software trigger computer needs to know when an event has occurred. This is accomplished by a broadcast packet out of the trigger system. This way everyone knows when a trigger has occurred. This might give us an extra layer of error checking (i.e. an announcement comes in but someone did not get the high speed version and we can flag this as an error).


9. GHz Source & Sync Pulse - Generation & Distribution

One of the key simplifying aspects of this system is that all the digitizers can be synchronized. This means that sample XYZ in every channel represents the same time. This is accomplished using the sync pulse feature of the A/D converter. The chip requires the sync pulse to look similar to the one shown below in Figure 12. The actual requirements are listed in the datasheet for the chip, but what is shown will work well.

GHz Sample Clock

Sync Pulse

8 Clks

Figure 12: ADC Sync Pulse

The system as currently envisioned will fit into a single 19 inch rack mount unit. This is shown below in Figure 13. A rear view of the system is shown in Figure 14. A detailed block diagram of the timing generator port is shown in Figure 15. A detailed block diagram of the distribution card is shown in Figure 16. To start we need a 1 GHz source. It is difficult to find a 1 GHz plain Jane oscillator. Most oscillators are space grade or have insane options we don’t need. A search continues for a good oscillator, though one was found for 1K$ that we may buy. The primary buffer has two inputs with an internal multiplexor and we plan to have two options available. The first is a fixed frequency oscillator that takes a 10MHz reference clock and multiplies it by 100 to get the 1 GHz. This plan calls for an RF input which is then put through a comparator to make into a PECL clock. The threshold is programmable through a DAC, but will most likely be AC coupled on the RF input and the reference tied to ground. The second option is to take clock inputs directly from a clock generator using ECL or PECL. These inputs will also be AC coupled and thus either will work. Stanford Research makes a very nice looking clock generator, but it costs a lot more. The hope is to find some better / cheaper / faster clock source for under 1K$. The primary 1-20 splitters have many more outputs than are needed for a 120 output system. One of the other GHz outputs will actually be routed around back to a clock divider through the FPGA to make the sync pulse. The sync pulse is put through a delay line so we can line it up perfectly with the GHz clock. The delay value is measured one time and set forever.


Generator Board1 GHz

Oscillator

DistributionCards

DistributionCards

DistributionCards

SMA & Coax Cabling

19 Inch Rack Mount Unit

SMA Connectors out of the back.

Figure 13: Overview of Timing Generation / Distribution Packaging

CLKSYNC

CLKSYNC

CLKSYNC

CLKSYNC

19 Inch Rack Mount Rear View (4 rows shown, could be up to 6 rows)

Figure 14: Rear View of Timing Unit

The rabbit module is part of a standard setup we plan to use for all “slow” systems. The EEPROM will hold the module’s IP address. The power supply on the board will supply 3.3V for the generator card and through a ribbon cable to all of the distribution cards.


Due to performance issues, we do not want to put SMA jacks on the generator card outputs. Since this is all self contained within one box, we intend to hang “pigtails” off of the board. This means we will take a 2 foot cable, cut it in half, and solder down the cut ends to pads on the board. This makes routing the board easier, the board smaller, and saves a small fortune in connectors.

10MHzOvenized

StableReference

ClockFPGA

to GHz OscillatorEthernet RABBIT

10Mbit

From GHz Oscillator

1-20 Clock Buffer 1-20 Clock Buffer

DelayLine External Oscillator

Option (ECL or PECL)

PECLClock

Divider

To Distribution Cards To Distribution Cards

Sync Pulse Out

EEPROM

D/AProvisioned By SystemDelay Value

Sou

rce

Sel

ect

PowerSupply

Figure 15: Detailed Diagram of Generator Source Card

The downside is that there will be a bunch of pigtail pairs unused in the box. As another cost saving measure, we plan to add output board as needed, in 10 output sets. Initially, we plan to install 30 output sets. The rest are easily field-installable as more money becomes available. The maximum intended for this box is intended to be 120, 150, or 180 sets depending on how things look and how much rack space we want to take up.


The distribution card in Figure 16 is incredibly simple. Bring in the signal differentially over coax / SMA. Terminate at chip. Put connectors on outputs of chip. Provide power with a ribbon cable input. Done. Oh yeah, throw in some bypass caps and mounting holes ☺.

SMA

1-10 Output Buffer Chip

SMA SMA SMA SMA SMA SMA SMA SMA SMA

SMA SMA Differential inside box,single ended to clusters.Power

Ribbon Cable Coax

Figure 16: Distribution Card Diagram


10. GPS Timing System

GPS Time is only used to cross correlate with IceCube. This subsystem is fairly basic. An assembly view is shown in Figure 17 and a block diagram for the RICE GPS Board is shown in Figure 18. The intention is to timestamp each trigger event with as close to UTC as possible. When the L2 Trigger is found, this board takes the current UTC time and saves it and the event ID together. Then this information is sent via the rabbit module to the L3 software system which correlates all the data. The basic theory of operation is to count time from the PPS input. The PPS resets time in that second and a high speed clock is used to count off the number of nanoseconds since the PPS pulse happened. Since we have a very nice stable clock running around, the intention is to bring in the GHz clock and sync and use our “standard” divider to get either 125 or 250 MHz to count UTC time with.

RICE GPS Board

19 Inch Rack Mount Unit

GPSReceiverModule

To Antenna

PowerSerial Data

PPS10MHz Clock

Figure 17: RICE GPS Rack Mount System

This will provide either 4 or 8 ns of counting resolution. Absolute error to UTC will be strictly defined by the GPS receiver accuracy plus the counting resolution we have. The new Motorola M12+ GPS modules claim a resolution of < 10 ns using a 1 sigma average and < 60 ns using 6 sigma average. This unit also has a granularity messaging system that if implemented correctly they claim a resolution of < 2 ns using a 1 sigma average and < 12 ns using 6 sigma average. The plan is to use this card and try to implement their granularity message system to get the best accuracy possible. If this unit doesn’t work we can always get one of the multi K$ units to feed our card.


Int GPS Serial Data

RabbitEthernetModule

FPGA

Int GPS PPS

Global Trigger

Ethernet toControl System

Spare Trigger Inputs

Event ID

GHz ClockSync

ClockDivider

Int GPS 10 MHzE

EP

RO

M

Ext GPS Serial DataExt GPS PPSExt GPS 10 MHz

Figure 18: GPS Timing System Block Diagram


11. Computers in the System

11.1 SW Trigger Computer System

A SW triggering computer is an option we will keep. This computer would get the corrected timestamps from the CTPUs and would apply veto algorithms and other physics mandated cuts to the data. When a SW trigger is found, a dumbed down version of the HW trigger system gets a message via ethernet and fires the global trigger line. This dumbed down version would maintain the ability to do unbiased triggers, etc.

11.2 Event Storage Computer System

This is in essence a great big hard drive. It’s function can easily be located in the SW trigger computer.

11.3 Control Computer

A GUI will be written to control the system. This GUI will allow easy upgrades of FPGA code as well as setting operating parameters, viewing different forms of status information, generating log files, and running of the system. The control computer will also have alarm conditions. If one is identified it can send an email to the system operator back in the states who can tell what is going on.


12. Power System

The system needs to be powered up. Each cluster will have an output port. That port is switchable via software and the voltage / current will be monitored. Voltage & Currents will be sent to the control computer in a timely manner. This system will be expandable to have the same number of ports as we have clusters.

12.1 Rack View

Figure 19 shows how the rack might look with one shelf of the distribution system. The shelf would be sized to comfortably feed however many ports naturally fit in the shelf. This will be determined by sensor sizes, which connector we pick, etc. The basic concept is illustrated below. We can have as many distribution shelves as we need for the system. They can be expanded near indefinitely.


Status LEDs

19" Rack

Ethernet Connection forSystem Control

Front View

Status Display ?

Crate Controller CardPower Distribution Cards

Power Connector (Type TBD)

19" Rack

Rear View

Low Voltage DC for Logic PowerHigh Voltage AC for Cluster Power

Communications Bus via DINconnectors

Figure 19: Power System Rack View

12.2 Control Card

Figure 20 shows a general block diagram of the distribution system. The Rabbit Semiconductor ethernet module (RCM3200) will talk via the backplane to the distribution cards and via 10Mbit ethernet to the control system. If we find a decent status display, we may include it in the design. A temperature sensor is included just so we can monitor the temperature in the shelf.


Bac

kpla

ne C

onne

ctor

Data & Control

ControlModule(Rabbit)

EthernetInterface

Status LEDs

Status Display ?

LogicPower

TempSensor

Figure 20: Power System Control Card Block Diagram

12.3 Distribution Card

Figure 21 shows a block diagram of a distribution card. 240VAC comes in via the backplane and is filtered & fused. This power is then switched with a solid state switch. The output side has the current and voltage monitored by PIC A/D channels and the output them goes to a connector. The PIC reports current / voltages to the central monitoring system through the distribution control card. If an over-current condition happens, the solid state switch can be opened and the event reported to the control system as an alarm condition.


Bac

kpla

ne C

onne

ctor

Data & Control

ControlProcessor

(PIC)

One Output Shown

PowerFilters &

Fuse

I

V

TempSensor

Figure 21: Power System Distribution Card Block Diagram


13. In Ice Power & Monitoring System

One part of this system that will be developed / deployed is the power supply. It will provide multiple voltages for the different logical functions. The transformer will be designed and built to operate at -55C and will provide just enough voltage to power the LNAs and MODs with some margin. This system may be added if we can find a feasible way of getting voltage / currents / temperatures out of the ice system. There is a spare fiber and it’s hoped we can simply put a PIC processor on the power supply card in the ice. That processor would be able to control power to each LNA and modulator separately so we could turn off part of a cluster that was misbehaving. It is our hope we can do half duplex communication with just plain old serial ports for sending data at around 115K baud. If we can do communication over the one spare fiber, we will likely tie control into the control GUI for each output of the power supply (4 LNAs and 4 modulators). It is likely that we would feed current / voltage / temperatures back to the status system as well.

ControlProcessor

(PIC)

TempSensor

FiberTranceiver

220VACInput

FilterLinearPowerSupply

15V for LNAs

8-12V for MODs

5V

SpareFiber

Measure V & I, HavePower Switches

8 Sets

PressureSensor

Figure 22: In Ice Electronics Block Diagram


ControlProcessor

(PIC)

One Control ShownThere are 4 of thesefor the LNAs and 4

for Modulators

I

V

To LNA orModulator

LNA or ModulatorPower Supply

FiberTranceiver

Serial Port

Figure 23: In-Ice Voltage / Current Measurements & Control Switch


14. Science-y Things

14.1 Unbiased Trigger

The system shall have the ability in the main trigger system to auto trigger at a predefined rates (i.e. 1Hz, 10Hz, 100Hz, or whenever someone hits the “trigger now” button in the GUI”.

14.2 Others ?

Adding scopes into the system as a sanity check. Livetimes / channel, i.e. singles rate.

15. Board ID Table

Board ID # Function Comment 0x00 Reserved 0x01 L2 HW Trigger 0x02 0x03 0x04 0x05 0x06


16. Acknowledgements

Gary Varner at University of Hawaii has been an amazing help in this effort. His ideas, his ability to be a sounding board, and his support are much appreciated. He of course couldn’t have spent anywhere near the amount of attention he has without Peter Gorham’s project SalSA. Xilinx has really taken excellent care of us. They have donated an amazing amount of IP and products. We would be a lot poorer today without their help. AMANDA / IceCube. We wouldn’t be here without them.

ku high energy physics rice project - phys.hawaii.eduidlab/project_files/salt/docs/rice_daq… ·...

Documents