hadron engineering datasheet

Copyright 2020 FLIR Systems Inc. This datasheet is subject to change without notice.

EAR Controlled: EAR99 These commodities, technology or software are subject to the U.S. Export Administration Regulations (EAR). Diversion contrary to U.S. law is prohibited.

Hadron Engineering

Datasheet

2020-11-06

Document Number: 102-2035-40

Revision 240

Created by: Per Elmfors, Sr. Systems Engineer

EAR Controlled – See Cover Page

1 INTRODUCTION ............................................................................................................................................. 3

1.1 REFERENCES ..................................................................................................................................................... 3

1.2 ABBREVIATIONS AND DEFINITIONS ........................................................................................................................ 3

1.3 REVISION HISTORY ............................................................................................................................................. 3

2 SYSTEM OVERVIEW ....................................................................................................................................... 4

2.1 DATASHEET SUMMARY ....................................................................................................................................... 4

9Hz Hadron .............................................................................................................................................. 5

2.2 PRODUCT ARCHITECTURE .................................................................................................................................... 5

2.3 IR CAMERA ...................................................................................................................................................... 6

2.4 EO CAMERA ..................................................................................................................................................... 6

2.5 EEPROM ....................................................................................................................................................... 6

Rev FLIR0001 ............................................................................................................................................ 6

3 MECHANICAL ................................................................................................................................................. 7

3.1 HADRON MECHANICAL DIMENSIONS ..................................................................................................................... 7

3.2 IMU LOCATION ................................................................................................................................................ 9

3.3 EXAMPLE OF GIMBAL INTEGRATION ..................................................................................................................... 10

4 ELECTRICAL .................................................................................................................................................. 10

4.1 OVERVIEW ..................................................................................................................................................... 10

4.2 IR / EO SYNCHRONIZATION ............................................................................................................................... 11

4.3 HADRON EXTERNAL SIGNALS .............................................................................................................................. 11

Data buses ............................................................................................................................................. 13

4.4 HADRON TEST KIT AND SETUP ............................................................................................................................ 14

5 SOFTWARE .................................................................................................................................................. 17

5.1 OVERVIEW ..................................................................................................................................................... 17

5.2 STREAMING TO HDMI ..................................................................................................................................... 17

1.1.1 IR ............................................................................................................................................................ 17

1.1.2 EO ........................................................................................................................................................... 18

5.3 WEB INTERFACE TO TX2 ................................................................................................................................... 18

5.4 IR CAMERA .................................................................................................................................................... 19

5.5 EO CAMERA ................................................................................................................................................... 19

5.6 SYNCHRONIZATION .......................................................................................................................................... 20

5.7 IMU............................................................................................................................................................. 20


1 Introduction This document is a detailed datasheet for Hadron, a dual camera IR+EO core payload intended to be

mounted on a small UAS gimbal. There are separate interfaces to the two cameras: the EO camera has

raw 4-lane MIPI interface and the IR camera has USB 3.0 interface. Hadron also has a built in IMU for

gimbal stabilization. Video processing (compression, recording etc.) is performed outside the Hadron.

1.1 References Document Location

Boson datasheet https://www.flir.com/globalassets/imported-assets/document/boson-engineering-datasheet.pdf

Boson resources https://www.flir.com/support/products/boson#Resources

IMX412 datasheet Flyer: https://www.sony-semicon.co.jp/products_en/IS/sensor0/img/product/cmos/IMX412-AACK_Flyer03.pdf Datasheet available from Sony

ICM20602 datasheet https://www.invensense.com/wp-content/uploads/2016/10/DS-000176-ICM-20602-v1.0.pdf

Hadron mechanical IDD 70320A034-6C12080-19 Hadron Camera Assembly - Rev C.PDF

1.2 Abbreviations and definitions Abbreviation Definition

CCI Command and Control Interface

EFL Effective Focal Length

EO Electro-Optical

(H)FOV (Horizontal) Field of View

IDD Interface Design Document

IMU Inertial Measurement Unit

IR Infrared

UAS Unmanned Aircraft System

UVC USB Video Class

1.3 Revision history Revision Date Comment

100 2019-03-20 First revision.

101 2019-03-22 Added information about IR and EO sensor size, optics and video in section 2.1.

102 2019-04-11 Removed T sensor from EO sensor board. Rearranged and renamed I2C buses. Reduced to 2 I2C buses.

110 2019-04-23 Down-selected IMU to ICM20602 Added protective housing The EO and IR camera swapped places to be consistent with other FLIR UAS products (IR to the right, EO to the left) Weight of core without housing updated (now 30g) Updated max power consumption to 2820mW

https://www/

https://www/

https://www/

https://www/


Added pin-out on external connector Added pin-out on Hadron test connector to NVIDIA Jetson TX2 Added location of the IMU

120 2019-04-26 Revised the pin numbering on the main connector (mirrored from r110) and clarified location of pin 1.

130 2019-05-17 Revised dimensions with protective housing Added information about how to select SPI/I2C Added Table 3 with details about voltage levels Updated location of IMU Updated mechanical IDD Indicated coordinate system of IMU in Figure 6

140 2019-07-17 Added part number for Boson 320. Provided assembly drawing of Hadron Main Board Added INT line from the IMU to pinout of TX2 connector and Main Connector.

150 2019-08-15 Added flash maps for EO and IMU EEPROM (rev FLIR0000)

160 2019-08-22 Added part number for main external connector Extended information in EO and IMU EEPROM (rev FLIR0001) Updated weight to measured value (42.8g) and updated center of gravity and moment of inertia accordingly

170 2019-08-27 Added I2C addresses for each device.

180 2019-09-30 Information about the TX2 Software Package added. Updated export marking to EAR.

190 2019-10-23 Hadron part number updated (dash removed) Updated typical power to <1400mW Updated size and weight for rev B

200 2020-03-03 Added recommendation for air path to back of Hadron

210 2020-03-05 Clarify the electrical interfaces in introduction

230 2020-10-23 Added ECCN and HTS codes Added details about Hadron test kit Added appendix with MIPI trace length information Updated weight for rev B lens Removed old flash map rev FLIR0000 Added 9Hz version Added details about synchronization of video streams Updated lens model to SYD1201B-400

240 2020-11-06 Updated export marking to EAR99

2 System overview

2.1 Datasheet summary All dimensions and weights are preliminary and nominal design values. They may be updated later with

measured values.

Part number 60Hz: 70320A034-6C12080 9Hz: 70320A034-9C12080


Size 24 x 45 x 35 mm including protective housing

Weight 41g including protective housing

Power Typical < 1400mW, Max < 2820mW

Mechanical interface Screw mount to back plate

Electrical interface Hadron connector: Hirose DF40C-50DP-0.4V(51) Example of mating connector: DF40HC(2.5)-50DS-0.4V(51)

IR camera sensor Boson 320x256 pixels, 12m pitch, USB video and CCI

IR camera optics EFL 6.3mm, 34° HFOV, F/# 1.0

IR camera video Full resolution @ 60Hz or 30Hz

EO camera sensor Sony IMX412, 4056x3040 pixels, 1.55m pitch, 4-lane MIPI

EO camera optics Sunny SYD1201B-400, EFL 3.7mm, 80° HFOV, F/# 2.8

EO camera video Full resolution @ 60Hz See IMX412 datasheet for more options

IMU ICM20602, I2C or SPI (selectable)

Operational and storage temperature -20°C to +60°C

Tested EMI performance FCC part 15 Class B

Product classification:

Part Part number ECCN HTS

Hadron 60Hz 70320A034-6C12080 6A003.b.4.b 9013.80.9000

Hadron 9Hz 70320A034-9C12080 6A993.a 9013.80.9000

Hadron test kit 481-0081-00 EAR99 9013.90.7000

9Hz Hadron

The 9Hz version of Hadron is identical to the 60Hz version except that the Boson is a 9Hz Boson

(20320A034-9CAAX). All functionality is the same except that the IR and EO video streams cannot be

synchronized.

2.2 Product architecture The Hadron is designed to be a UAS dual IR+EO camera payload with an integrated IMU for gimbal

control. It provides raw (uncompressed) IR and EO video for further processing in the airframe.

Mechanically, the back of the Hadron can be mounted against the yoke of the gimbal. The electrical

interface is a 50pin connector for video and CCI for the two cameras as well as the IMU. As a part of the

delivery, software reference code for NVIDIA Jetson TX2 is provided. The gimbal controller is not part of

the delivery (hardware and software).


2.3 IR camera The IR camera is a Boson 320 with 34° HFOV (PN 20320A034-6CAAX 60Hz, or 20320A034-9CAAX 9Hz). It

has a built-in shutter for non-uniformity corrections.

The hyper-focal distance is given by the table below where c is the circle of confusion (2x the pixel

pitch), f is the focal length, N is the F# number and H (H = f2/Nc + f) is the hyper-focal distance.

Camera c [mm] f [mm] N = F/# H [mm]

IR 2*0.012 6.3 1.0 1660

EO 2*0.00155 3.68 2.8 1564

2.4 EO camera The EO camera is using a Sony IMX412 12MP sensor and a Sunny SYD1201A 80° HFOV lens.

2.5 EEPROM The EEPROM on the EO sensor board and the main board can be read and written by the user. FLIR

reserves the first 2048 bytes on each EEPROM for production data. Notice that the EEPROMs are not

write protected – the user must take care to not overwrite the first 2048 bytes!

Rev FLIR0001

EO EEPROM flash map

Byte number Example data Type Comment

0 – 7 FLIR0001 ASCII Revision of EO EEPROM flash map

8 – 15 H0000123 ASCII EO sensor serial number (same as Hadron SW unless the sensor is replaced)

16 – 19 0000 ASCII EO sensor hardware revision (same as Hadron revision unless it is replaced)

20 – 36 2019-08-15_142154 ASCII Date and time of production

IMU EEPROM flash map

Byte number Example data Type Comment

0 – 7 FLIR0001 ASCII Revision of IMU EEPROM flash map

Airframe Hadron

IR

EO

IMU

Gimbal

Gimbal Control

Video pipeline

MIPI, USB, SPI, I2C


8 – 15 H0000123 ASCII Hadron serial number

16 – 33 70320A034-6C-12080 ASCII Hadron part number

34 – 37 0000 ASCII Hadron hardware revision

38 – 47 0000047721 ASCII Boson serial number

48 – 62 20320A034-6CAAX ASCII Boson part number

63 – 80 2019-08-15_142154 ASCII Date and time of production

3 Mechanical The Hadron has and IR and EO camera intended to be mounted horizontally. The backplane serves as an

external mechanical interface. A gimbal yoke can be screwed onto he back and host a cable harness

leading from the Hadron to the airframe. That way, ingression protection from the back can be handled

by the gimbal design. The Hadron itself is IP53 rated as long as the back is sealed. The thermal heat path

from the Hadron is through convection off the surface and the through the metal back plate to which

the gimbal is mounted.

The Hadron has an external protective housing that secures ingression protection and protects it from

impact. It also helps against image non-uniformities in the IR image caused by wind cooling the lens

holder.

For a mechanical specification of the interface, see drawing in the mechanical IDD (section 1.1).

3.1 Hadron mechanical dimensions

Figure 1. 3D rendition of Hadron with protective housing.


Figure 2. Hadron external dimensions [mm] with protective housing.


Table 1. Estimated mechanical properties of the Hadron module with protective housing. This will be affected by the addition of the gimbal mechanics.

3.2 IMU location The IMU is located on the main board close to the center of the gravity and in the Hadron x and y

direction. See Figure 6 for the IMU intrinsic coordinate system (different from the Hadron coordinate

system).


Figure 3. Approximate location of the IMU relative to the back plane. The indicated coordinate system is for Hadron (see mechanical IDD). The IMU intrinsic coordinate system is different (see Figure 6). See Mechanical IDD for exact nominal location of IMU.

3.3 Example of gimbal integration The gimbal is not part of Hadron, but the design is intended to be integrated with a gimbal.

Figure 4. Example of how Hadron can be mounted to a gimbal yoke.

When integrating with a gimbal, the back of Hadron has to be sealed in a water and dust proof way in

order for the system to meet IP53. In doing so, a waterproof airpath must be kept open to equilibrate

pressure as temperature changes.

4 Electrical

4.1 Overview Hadron has 3 main internal components:

Boson IR camera: The self-contained IR camera.

EO camera: An IMX412 sensor with optics, an EEPROM for EO sensor calibration data.

Y

Z

Y

Y

X


Hadron main board: The board that connects the Boson and EO camera. It also holds the IMU

and an EEPROM with IMU calibration parameters.

The input power is 5 ± 0.25V on 4 pins. Each pin can sustain 0.3A. Internal 3.3V and 1.8V power rails are

generated on the main board.

Typical power consumption while streaming: <1400mW

Maximum power consumption (peak when Boson shutter is activated): <2820mW

Figure 5. Hadron internal and external signals.

4.2 IR / EO synchronization

4.3 Hadron external signals Table 2 below lists all external signals. The buses from the components to the main connector are

described below. The main connector on Hadron is DF40C-50DP-0.4V(51) and an example of a mating

connector is DF40HC(2.5)-50DS-0.4V(51).

pin Name pin Name

1 BOSON_USB_VBUS 2 5V

3 BOSON_USB_ID 4 5V


5 GND 6 5V

7 BOSON_USB_DN 8 5V

9 BOSON_USB_DP 10 RESET_N

11 GND 12 GND

13 BOSON_USB_TX- 14 IMX412_MIPI_D4+

15 BOSON_USB_TX+ 16 IMX412_MIPI_D4-

17 GND 18 GND

19 BOSON_USB_RX- 20 IMX412_MIPI_D2+

21 BOSON_USB_RX+ 22 IMX412_MIPI_D2-

23 GND 24 GND

25 BOSON_MIPI_D1- 26 IMX412_MIPI_CLK+

27 BOSON_MIPI_D1+ 28 IMX412_MIPI_CLK-

29 GND 30 GND

31 BOSON_MIPI_D0- 32 IMX412_MIPI_D1+

33 BOSON_MIPI_D0+ 34 IMX412_MIPI_D1-

35 GND 36 GND

37 BOSON_MIPI_CLK- 38 IMX412_MIPI_D3+

39 BOSON_MIPI_CLK+ 40 IMX412_MIPI_D3-

41 GND 42 GND

43 Not Connected 44 IMU_CS_N

45 IMU_INT 46 IMU_SCL-SCK

47 I2C_CAM_SCL 48 IMU_SDA-SDI

49 I2C_CAM_SDA 50 IMU_SDO Table 2. Pin-out on the main Hadron connector.

Pin / Signal Voltage

Power 5V 5V ± 0.25V

1.8V logic (I2C and SPI) V high: 1.26 to 2.1V V low: -0.3 to 0.54V

USB According to USB2 and USB3 standards

MIPI According to CSI-2 version 1.2 and MIPI D-PHY version 1.2 Table 3. Voltage levels for power and signals.


Figure 6. Assembly drawing of the Hadron main board showing the external connector P7 and the location of pin 1 in the bottom right corner. Pin 2 is in the bottom left corner. The IMU is located close to the center of gravity (U9). The intrinsic coordinates of the IMU are indicated in the figure. U37 and U38 are power supplies. DS2 is an LED indicating that 3.3V and 1.8V are on.

Data buses

IR_USB

o Boson has a USB interface for video (UVC) and CCI. It supports both USB 2.0 and USB

3.0. The video format options are described in the Boson datasheet (see section 1.1).

IR_MIPI

o Future releases of Boson will have support for 2-lane MIPI. Hadron exposes pins for

future compatibility. The associated CSI-2 I2C interface (IR_I2C) is shared with the EO

camera, the EO and IMU EEPROM on I2C bus I2C_CAM.

EO_MIPI

o The IMX412 EO sensor has a 4-lane MIPI interface.

o The EO MIPI trace lengths from the sensor to the external connector are 38 ± 0.5 mm.

IMU_I2C (1.8V)

o The IMU has an optional I2C or SPI interface. The I2C bus is separate from the other

devices on I2C_CAM.

o IMU I2C address: 1101001

I2C_CAM (1.8V)

o EO_I2C

The EO sensor I2C bus with registers for image control.

EO sensor I2C address: 0010000

o EO_EE_I2C

The EO EEPROM I2C bus.

EO EEPROM I2C address: 1010000

o EO_IMU_I2C

The IMU EEPROM I2C bus.

IMU EEPROM I2C address: 1010001

y

x


o I2C_IR

Future Boson CCI part of CSI-2.

Boson CCI I2C address: 1101010

IMU_SPI

o The IMU has an optional SPI interface that can be used instead of I2C. The part defaults

to I2C mode. To use the part in SPI Mode, pull the CS line low after waiting 2ms for the

startup time for register read/write. Then write the “I2C_IF_DIS” register to keep the

part from going into I2C mode.

RESET_N

o The Boson and the IMX412 can be reset by holding RESET_N low.

4.4 Hadron test kit and setup

Figure 7. Hadron test kit with the TX2 test board at the top and the Main test board below.

A test kit to connect Hadron to an NVIDIA Jetson TX2 is available (PN 421-0081-00). Boson USB is

connected using a standard USB2 or USB3 cable while all other signals are connected to the TX2 Camera

Expansion Connector (Samtec QTH-060-01-F-D-A-K-TR). Power to Hadron can be switched on/off using a

power switch on the main test board.


Figure 8. Block diagram of Hadron – TX2 test setup.

TX2 pin Hadron Name Tx2 Name TX2 pin Hadron Name Tx2 Name

95 RESET_N CAM0_RST_L 66 IMU_CS_N SPI_CS

1 EO_MIPI_D1+ CSI_A_D0_P 62 IMU_SCL-SCK SPI_SCK

3 EO_MIPI_D1- CSI_A_D0_N 68 IMU_SDA-SDI SPI_MOSI

13 EO_MIPI_D2+ CSI_A_D1_P 64 IMU_SDO SPI_MISO

15 EO_MIPI_D2- CSI_A_D1_N 19 IR_MIPI_D1+ CSI_C_D0_P

2 EO_MIPI_D3+ CSI_B_D0_P 21 IR_MIPI_D1- CSI_C_D0_N

4 EO_MIPI_D3- CSI_B_D0_N 31 IR_MIPI_D2+ CSI_C_D1_P

8 EO_MIPI_D4+ CSI_B_D1_P 33 IR_MIPI_D2- CSI_C_D1_N

10 EO_MIPI_D4- CSI_B_D1_N 25 IR_MIPI_CLK+ CSI_C_CLK_P

7 EO_MIPI_CLK+ CSI_A_CLK_P 27 IR_MIPI_CLK- CSI_C_CLK_N

9 EO_MIPI_CLK- CSI_A_CLK_N 75 CAM_I2C_SCL CAM_I2C_SCL

105 IMU_SCL-SCK I2C_GP_CLK 77 CAM_I2C_SDA CAM_I2C_DAT

107 IMU_SDA-SDI I2C_GP_DAT ------ ------

112 IMU_INT MOTION_INT_L

118 5V 5V 120 5V 5V

5 GND GND 6 GND GND

11 GND GND 12 GND GND












115 GND GND 116 GND GND Table 4. Pin-out on TX2 camera connector.

Figure 9. The main Hadron test board with the USB connector (J1) at the top and the micro-coax connector (J31) at the bottom.

Figure 10. IMU i2c/SPI can be selected on Hadron main test board by (de)populating 0-ohm resistors according to the schematic above.


Figure 11. Assembly drawing of the Hadron main test board showing the location of the connector to the Hadron main board (P8) and the location of resistors R327 - R330 used for selecting IMU i2c/SPI.

5 Software NVIDIA Jetson TX2 is used as a reference design and test platform. Please contact FLIR to get access to

the SW package.

5.1 Overview The TX2 reference SW is delivered as a package that installs a complete image on the TX2. Any previous

data on the TX2 is deleted in the process. Instructions for how to use the install package can be found in

the file README.hadron-software inside the package. The install package at the time these instructions

were written was:

102-2035-70_hadron-image-jetson-tx2-v1.1.0-0-g37a856e.tegraflash.zip

A package with source code for the reference software is also available. It contains a \source-

release\README file which describes the content and how to build the install package. The source

package at the time these instructions were written was:

102-2035-71_source-release-v1.1.0-0-g37a856e.tar.tar.gz

A more detailed description of what is included in the source code can be found in the software package

under 102-2035-71_source-release-v1.1.0-0-g37a856e\source-release\README. As an example, the

video synchronization method is described in section 5.6.

5.2 Streaming to HDMI

1.1.1 IR Use gstreamer pipeline to display Boson video on HDMI:


DISPLAY=:0 gst-launch-1.0 -v v4l2src device=/dev/boson-usb ! "video/x-

raw,width=640,height=512,format=NV12,framerate=60/1" ! videoconvert !

xvimagesink

1.1.2 EO Use gstreamer pipeline to display EO video on HDMI:

gst-launch-1.0 -v nvcamerasrc ! 'video/x-

raw(memory:NVMM),width=1352,height=762,format=I420,framerate=60/1' !

nvoverlaysink

If gstreamer format is not compatible with display it will show a red screen. Edit the file xorg.conf and/or

stop xservier:

Edit /etc/X11/xorg.conf

o Add the following line before the last "EndSection":

Section "Device"

Identifier "Tegra0"

Driver "nvidia"

Option "AllowEmptyInitialConfiguration" "true"

Option "TegraReserveDisplayBandwith" "false"

EndSection

Or, stop the X server with the following command before running the gstreamer pipeline:

o systemctl stop xserver-nodm

5.3 Web interface to TX2 The reference SW contains a web service that can be used to test each component of Hadron. It can be

reached over Ethernet on http://192.168.44.1:5000/ (or over USB on http://192.168.55.1:5000/). For

video streaming use Firefox (other browsers do not work as reliably).

http://192.168.44.1:5000/

http://192.168.55.1:5000/


Start streaming by going to “Visual camera” / “Boson”, then click “Start stream” and then “Visual

camera” / “Boson” again.

The full http protocol is described under the “API” link.

5.4 IR camera Please refer to https://www.flir.com/support/products/boson#Resources for the Boson SW interface.

5.5 EO camera A reference implementation of driver for video capture over MIPI and CCI, as well as the use of the EO

EEPROM data, for NVIDIA Jetson TX2, can be found in the source-release package (see section 5.1).

Register settings for IMX412 can be obtained from FLIR on request.

https://www.flir.com/support/products/boson#Resources


5.6 Synchronization The video streams from the EO and IR sensor are synchronized when started from the tab “Video sync”

in the web interface (see section 5.3). To synchronize the cameras, follow the code example in the

source release as described below, and make appropriate modifications for the intended platform if

different from NVIDIA XT2.

The top-level script is \source-release\downloads\hadron-testing-git-r0\git\bin\sync_capture_raw,

which wraps camera_sync_mode.py and sync_capture. It also uses standard v4l2-ctl utility to set up

attributes like framesize before the capture. This script sets the sync for both cameras:

camera_sync_mode.py --boson_sync master --eo_sync slave

The function camera_sync_mode.py sets the Bosons into master mode and sets appropriate registers in

the IMX412:

## for setting boson to master sync = sdk.EE.FLR_BOSON_EXT_SYNC_MODE_E.FLR_BOSON_EXT_SYNC_MASTER_MODE boson.bosonSetExtSyncMode(sync)

## for setting EO to slave 0x3f0b -> 1 # MC_MODE 0x3041 -> 0 # MASTER_SLAVE_SEL 0x3040 -> 0 # XVS_IO_CTRL 0x4b81 -> 0 # EXT_OUT_EN 0x0350 -> 1 # FRM_LENGTH_CTL

Then, sync_capture (a compiled V4L2 capture app) is used to capture the frames::

sync_capture --boson $BOSON_DEV --visual $VISUAL_DEV --count $FRAME_COUNT --delay

$VISUAL_FRAME_DELAY --output $FILE_NAME.partia

See source code in \source-release\downloads\hadron-testing-git-r0\git\src\sync_capture.c.

The Boson camera starts producing frames first, and after 6 or 7 Boson frames the visual camera starts

producing frames as well. After that startup, the frames come out as pairs with boson then visual

following shortly after. The initial condition is thus to throw away frames while only Boson data is being

received. The visual camera frame has higher capture latency so they are matched up with

$VISUAL_FRAME_DELAY number of boson frames later. This is the 'framedelay' parameter in the API.

5.7 IMU The IMU registers are defined in the IMU datasheet, see section 1.1.

hadron engineering datasheet

Documents