"Screenshots taken from a version you do not own"
The Pixie-Net is a multi-channel radiation detector readout electronics module. Designed for high-precision coincidence gamma-ray spectroscopy using HPGe detectors, scintillators, or silicon detectors, the Pixie-Net offers not only detector signal digitization and waveform capture but also pulse height measurements, on-board energy histograms, time stamping and constant fraction timing, pileup inspection, external gating and online pulse shape analysis. Besides nuclear spectroscopy, the Pixie-Net can be used for neutron/gamma discrimination, time-of-flight measurements, and coincidence/anti-coincidence measurements.
The Pixie-Net is based on the Avnet MicroZed SOM, using a Xilinx Zynq as the key processor, and running a variant of Ubuntu Linux. Those are fairly popular devices and software, and there are lots of blogs and forums providing ideas, tricks, and tools for improvement. While the Pixie-Net comes with a working set of software for data acquisition, we continue to explore improvements. Some of them are listed below.
Disclaimer: As the subtitle of this blog is already a quote , so also the ideas below are largely based on what other people wrote and did before. Credit is given as far as possible, but no guarantee is given that the code modifications work reliably. These new functions are not (all/yet) part of the Pixie-Net firmware/software, but can be added in future versions and/or implemented by interested users.
 T. Pratchett, "Only you can save mankind"; disclaimer on a fictitious video game
- PMOD GPS
- Upgrade to Ubuntu 15
- Login via Remote Desktop
- Using ROOT on Pixie-Net
- IEEE 1588 Precision Timing with a Pixie-Net
- MATLAB Interface
- Webpage Operations
- Spectrometer Performance
- IEEE 1588 related utilities
- Scientific Papers
- New Pixie-Net XL Prototype Under Development
- Added triggered, averaged waveform capture to Pixie-Net
- MicroZed Carrier Card for Trigger I/O in Pixie-16 chassis
- Upgrade to Ubuntu 18
For the pulse shape analysis (PSA) variant of the Pixie-Net, results should be displayed as 2D histograms on a web page, plotting the detected pulses' PSA value vs Energy colored by intensity. The PSA computes two sums over characteristic regions of the detector pulse, and calculates the ratio (PSA value) of the two sums. The PSA value is then a condensed characteristic of the pulse shape, and can be used with suitable detectors to distinguish events or particle types, for example gammas and neutrons or gammas and alphas. A typical result, as the above measurement with an alpha/neutron source and a crystal stilbene detector, has distinguishable branches for different particle types. Neutron/gamma measurements with CLYC or liquid scintillators show similar results.
 NIM A 809 (2016), 132: http://dx.doi.org/10.1016/j.nima.2015.08.023
The MicroZed has a "PMOD compatible" I/O connector, which is brought out as external I/O in the Pixie-Net. With some reconfiguration of the firmware and device tree , it is possibly to link the PMOD pins to the Zynq ARM's UART controller. On can then connect a UART GPS module to the PMOD port, here the Digilent PMOD GPS -- though unfortunately, due to pinout mismatch, it can not be plugged in directly but needs cabling to swap TX and RX pins. By default, the PMOD GPS module operates at a baud rate of 9600 baud, with 8 data bits, no parity, and with a single stop bit. Luckily, this is the default setting for the UART0 serial port. So reading the GPS module's output is as simple as reading from the corresponding /dev via the command line.
Use dmesg | grep tty to check which serial ports are configured.
ttyPS0 is the USB/UART console, so use cat /dev/ttyPS1 to print out the GPS messages.
In my tests, the GPS found the time ok in the office, but for position lock I had to move things outside. The first "GPGGA" message indicates that the time is 16:57:59.090 UTC with "0" for position fix and "0" for the number of satellites used. The second "GPGGA" message indicates that the time is 19:23:22.000 UTC, with position 37.36... degrees N and 122.03... degrees W, "1" for position fix and "4" for the number of satellites used.
When locked, the PMOD GPS module issues a Pulse Per Second (PPS) signal with a rise time of ~10ns, which can be fed into the Pixie-Net for external time stamping of the detector signals. Precision relative to the internal clock is about 1.5 microseconds. This is measured as the deviation from the average increment -- nominally 125,000,000 of the Pixie-Net 125 MHz clock ticks between consecutive PPS time stamps, but in reality 125,000,000 +/- ~250. The ~250 clock ticks jitter corresponds to deviation of up to ~2000ns with a more or less Gaussian distribution with 1.585 microsecond full width at half maximum (FWHM).
The question is of course, is the PPS signal jittering, or is it the Pixie-Net’s clock. Checking the PPS signals from 2 GPS modules powered by the same PN on an oscilloscope, the PPS rising edges are in sync (always see 2 edges together on the oscilloscope) but the phase shift ranged between ~20ns and 200ns between scope captures in the ~ 10min observed, maybe shifting 10-20ns between subsequent captures.
The Pixie-Net (at this point), is shipped with Xillinux, a version of Ubuntu 12. This is a nice, easy way to get started with the Zynq/MicroZed and free to use with the Xillybus Lite kit. (After all, we are using the Zynq because we want to integrate FPGA and ARM/Linux, and almost everybody else makes it ridiculously hard to share data between them, if they mention it at all.) But for some of the more advanced functions below, it's useful to upgrade to something newer. This involves such fun [man made] problems as rebuilding the kernel, cross compiling, remaking device trees and so on.
I list here and here some references I found useful for that. The procedure is quite cumbersome to set up the first time, but mostly following recipes. Contact XIA support if you want to try a copy of the Pixie-Net Ubuntu 15 OS variant.
Normally, login to the Pixie-Net's Linux system is via a terminal, either via USB/UART or via SSH. There are Zynq modules that have video outputs, but not MicroZed used here, and also use of the video may require licenses for video drivers. However, as pointed out here, it is perfectly possible to run a graphical desktop internally on the Zynq, and log on to it via Remote Desktop Connection. So what if the board has no connector for video output -- just view it over the network. (Note: The Pixie-Net's micro HDMI connector is used for GPIO, not video)
So instead of using this ....
... you can use this. (Note the same IP number on the window frame.)
Feel free to read your emails, write reports,
and play games right on your Pixie-Net. You can even open a terminal window and start typing the same things you would type into the SSH terminal ;-).
I here chose xfce as the graphical desktop, but xrdp or Ubuntu Desktop would probably work just the same. Probably requires something newer than the Xillinux / Ubuntu 12 shipped by default with the Pixie-Net -- I used Ubuntu 15, see above.
Here is the procedure:
In the Linux terminal on the Pixie-Net, install as follows (Ubuntu Desktop still provides some basic apps):
sudo apt-get install --no-install-recommends ubuntu-desktop
sudo apt-get install xfce4
echo xfce4-session >~/.xsession
In the Linux terminal, after each boot, find the Pixie-Net's IP address:
On a remote Windows PC:
- Run Remote Desktop Connection.
- Specify the Pixie-Net’s IP address
- Log in
On a remote Linux PC:
- use X11 forwarding by typing ssh -Y firstname.lastname@example.org (or equivalent) in a terminal
With the graphical desktop, one of the programs one can now run is ROOT. It's available as an Ubuntu package. Just install
- apt-get install root-system-bin
If you really want, you can view (and even analyze) the Pixie-Net data right on its own processor (screenshots coming soon).
Do I recommend it? Hmmm, not really. But it can be done. And maybe someone will someday make a graphical user interface for the Pixie-Net data acquisition that way ... to click buttons instead of typing ./startdaq.
Update August 2017
A ROOT demo interface now has been developed. Still very basic, it can configure the Pixie-Net (./progfippi etc), view ADC traces, start data acquisitions (./startdaq), and display the resulting spectra. The code is contained in a ROOT file pixieNet.C (available on request). To use it, simply start ROOT (in a terminal, type root) and then start the GUI (type .x pixieNet.C).
The Zynq used on the MicroZed used on the Pixie-Net comes with some basic built in IEEE 1588 Precision Timing Protocol (PTP) functions. That can be used to synchronize the data acquisition between multiple Pixie-Net units to nanosecond precision. (Depending on the type of implementation, that's hundreds or tens of nanoseconds.)
While the Zynq provides HW timestamping functions for PTP, the clock adjustments have to be controlled by software. Open source programs for that purpose include ptpd and LinuxPTP. While ptpd is an Ubuntu package (even in Ubuntu 12), LinuxPTP has to be downloaded and installed (at least in Ubuntu 15). But it seems to have better hardware timestamping support (at time of writing) so the tests below were done with LinuxPTP.
For Ubuntu 15, the following kernel options were required for PTP with LinuxPTP:
and the (older) xemacps network driver has to be specified in the device tree. This is the one used used in Xilinx' PTP "tech tip" example. (With the new Cadence driver, PTP is not supported for the MicroZed's Zynq, only for the UltraScale.)
After installing LinuxPTP, it can be run with 2 Pixie-Net units connected through the network :
and the software reports time offsets between master and slave of on average ~17ns, with a FWHM distribution of ~1300ns:
Not bad for something that is included in HW and SW for free, and with some stretch, "1300ns" is already "hundreds of nanoseconds" (13 of them). But the performance is not really yet up to the level of nuclear physics timing needs. And so far only the Linux clocks are synchronized, not the clocks used by ADC and FPGA to capture and process detector signals.
But improvements are in progress ...
Since PTP timestamping is best performed as close to the hardware as possible, a new hardware revision of the Pixie-Net has an optional secondary Ethernet PHY that can generate clocks and triggers from the PTP clock. The new Pixie-Net still uses the MicroZed with its default PHY, so the Zynq's MII interface was rerouted through the FPGA fabrics to the new PHY similar to the description here. The PHY is supported by LinuxPTP and the registers controlling clock and trigger outputs can be accessed with mii-tool. It can even turn on synchronous Ethernet. Using the PHY output clock for the ADC and the FPGA processing logic, detector pulses can be captured synchronous to the rest of the network. Precision reaches between a few hundred nanoseconds and a few hundred picoseconds depending on network capabilities and signal source.
The plot below shows the time difference measured for 2 coincident gamma rays captured in two Pixie-Net modules synchronized with PTP. One can see the time difference drift due to small clock frequency differences, then reverse due to the periodic adjustments by LinuxPTP.
This poster ( NSS_NTSposter.pdf ) summarizes the performance measurements in a number of configuration, including a comparison to clocking the Pixie-Net modules with a network clock from a White Rabbit demo system.
(for those who asked)
A few days experimenting with MATLAB resulted in the demo interfaces for a host PC, as shown above. It implements
- Operate Pixie-Net via serial port (click button in MALAB rather than typing command in terminal)
- Read resulting data files
- Display data from files.
While in principle the Pixie-Net's Zynq firmware can be developed with MATLAB tools as well, this is currently not supported by XIA.
As an alternative to the terminal entry and execution of the C programs to set up and perform data acquisition (e.g. ./progfippi and ./startdaq), there is a [moderately] secure web page that allows execution of the C programs as a cgi script from a webpage. The Web Operation page can be reached via a link in the “CGI Current Data” section of the Pixie-Net Home page. It essentially lists the C functions that would normally be typed into the terminal; clicking on the function name executes the function. Data is created in a separate subdirectory (/var/www/webops) with Linux permissions for data write access for the webserver. The Web Operation page shows the equivalent links from the Pixie-Net home page to download and display the data.
As this webpage allows generation and execution of files from any remote user, the webserver should also be modified to require authentication to access this webpage. See example here.
Below are a few plots showing how the Pixie-Net compares to the other Pixie spectrometers in some typical nuclear physics setups
The DP83640 PTP Ethernet PHY used above can be programmed to generate triggers, clock outputs, and PPS signals. While the Linux drivers for the DP83640 define some of the relevant registers, I could not find a utility to program them. (LinuxPTP has some code to initialize some of them, but not to e.g. turn on a PPS signal or issue a trigger at a user defined time). However, there is a Linux utility "mii-tool" which I modified to read/write the DP83640 registers and perform a few basic tasks like
- report PTP time and System time
- turn on Synchronous Ethernet mode
- issue a trigger at time TON and time TON+DUR
This modified utility is here released as open source (see legal and disclaimers in the zip file. It should run on any Linux without the need for installing drivers etc, but I tested it only on Ubuntu 15, Kernel 4.x, for Zynq (ARM).
Please contact XIA for copies of posters, conference records, etc.
Network time synchronization for detector data acquisition electronics
Wolfgang Hennig, XIA LLC
Large scale nuclear physics experiments often use arrays of radiation detectors, which can be physically separated, for example in separate rooms or along a beamline. Time synchronization of the detector readout electronics is essential to detect related events, which is often accomplished by sharing clock, trigger and reset signals within and between racks of digitizing electronics. This works well over short distances, but requires dedicated cabling and/or modules and becomes cumbersome for widely separated arrays. As the detector readout electronics is in many cases operated by computers linked over standard data networks, an alternative to dedicated clock distribution trees is the synchronization of clocks over the network. Network time synchronization protocols like IEEE 1588 or White Rabbit are reported to achieve low/sub nanosecond timing resolution, but these results generally refer to clock offsets computed in software or to once-per-second reference pulses, not to the capture of detector pulses with amplitudes and waveforms.
The work reported here thus focuses on integration of network time synchronization techniques with digital data acquisition for radiation detectors. Detector waveforms are captured with analog to digital converters that are synchronized to the network master clock and/or are tagged with time stamps related to the network time. Key to this integration is the handling of network clock and time in the same FPGA firmware that processes the analog to digital converter’s data stream. We will describe implementations based on FPGA firmware and on network hardware, and report timing resolutions obtained for two synchronized detectors capturing coincident radiation.
Network Time Synchronization of the Readout Electronics for a New Radioactive Gas Detection System
W. Hennig1, S. Hoover1, V. Thomas2, O. Delaune2
1XIA LLC, 31057 Genstar Rd., Hayward, CA 94555, USA
2CEA, DAM, DIF, F-91297 Arpajon, France.
In systems with multiple radiation detectors, time synchronization of the data collected from different detectors is essential to reconstruct multi-detector events such as scattering and coincidences. In cases where the number of detectors exceeds the readout channels in a single data acquisition electronics module, multiple modules have to be synchronized, which is traditionally accomplished by distributing clocks and triggers via dedicated connections.
To eliminate this added cabling complexity in the case of a new radioactive gas detection system prototype under development at the French Atomic Energy Commission (CEA), we implemented time synchronization between multiple XIA Pixie-Net detector readout modules through the existing Ethernet network, based on the IEEE 1588 precision time protocol. The detector system is dedicated to the measurement of radioactive gases at low activity and consists of eight large silicon pixels and two NaI(Tl) detectors, instrumented with a total of three 4-channel Pixie-Net modules. Detecting NaI (Tl)/silicon coincidences will make it possible to identify each radioisotope present in the sample. To allow these identifications at low activities, the Pixie-Net modules must be synchronized to a precision well below the targeted coincidence window of 500-1000 ns. Being equipped with a 1588 compatible Ethernet PHY that outputs a locally generated but system-wide synchronized clock, the Pixie-Net can operate its analog to digital converters and digital processing circuitry with that clock and match time stamps for captured data across the three modules. Depending on the network configuration, the implementation is capable to achieve timing precisions between 300 ns and 200 ps.
Peer reviewed article at https://doi.org/10.1109/TNS.2018.2885488
Recently received the first prototype of a larger/faster version of the Pixie-Net, working title Pixie-Net XL
It has the same pulse processing structure as the Pixie-Net, but adds
- Two Kintex 7 FPGAs for pulse processing
- Dedicated ADC daughtercard for each Kintex 7
- Dedicated SFP Ethernet port (1/10G) for each Kintex 7, White Rabbit compatible
- Dedicated RJ45 Ethernet port (10/100M) for each Kintex 7, PTP 1588 compatible
The daughtercard can fit a variety of ADC configurations, for example
- 4 channels, 14 bit, 100-500 MHz with individual gain/offset
- 8 channels, 14bit, 100-250 MHz with fixed gain differential inputs
- 8-16 channels, 12bit, 50 MHz with fixed gain differential inputs
- multi-channel GHz ...
Sampling interval 2-64K samples (8ns - 260us), for 4K samples (total 32us to ~1s).
The Pixie-16 MZ-TrigIO is designed to route signals from the PXI backplane in a Pixie-16 chassis (rear connectors) to the front panel (front connectors) and make logical combinations between them in FPGA fabric. It has the following features and capabilities:
- Ethernet programmable trigger/coincidence control module for the Pixie-16
- 48+ Pixie-16 backplane trigger connections to local Zynq processor
- 48 front panel LVDS connections to local Zynq processor
- MicroZed Zynq processor with embedded Linux, acting as a standalone PC with built-in SD card drive, USB host, 10/100 Ethernet, webserver, etc
- 1588 PTP and SyncE clock synchronization
- Open source user access to software and firmware
- Use as standalone desktop unit or in 6U PXI chassis
- Custom I/O standards via daughtercards
Update July 2019: For documentation partially in Chinese and customization at PKU see here
Similar to the Ubuntu 15 upgrade, the OS on the Pixie-Net and the Pixie-16 MZ-TrigIO can be upgraded to Ubuntu 18.04 LTS. The SD card image can be downloaded from
- The SD card image includes the Pixie-Net control functions. For the Pixie-16 MZ-TrigIO replace the contents of /var/www with the Pixie-16 MZ-TrigIO specific functions.
Kernel version is "Xilinx 17.4" from https://github.com/Xilinx/linux-xlnx/releases
- still includes the old EMAC_PS network driver
- includes a bug fix for the network transmission of large files
The following kernel driver options were enabled in addition to the defaults:
- The file system is Ubuntu 18.04 LTS from https://rcn-ee.com/rootfs/eewiki/minfs/
- The FUSE_FS kernel driver and installed Ubuntu packages ntsf-3g and ntsf-config allow mounting of ntsf files systems (e.g USB drives)
- The ROOT package and remote desktop login were not installed or configured
- The USB/UART login now requires a password