TrakEM2 Software for Neural Circuit Reconstruction

1There is a growing consensus that detailed volumetric reconstructions of thousands of neurons in millimeter-scale blocks of tissue are necessary for understanding neuronal circuits,. Modern electron microscopes (EM) with automatic image acquisition are able to deliver very large collections of image tiles –. Unfortunately, the problems of acquiring the data have so far been easier to solve than that of interpreting it,. Increasingly, neuroscience laboratories require automated tools for managing these vast EM data sets using affordable consumer desktop computers.

2Here, we present such a tool. It is an open source software package, named TrakEM2, that is optimised for neural circuit reconstruction from tera-scale serial section EM image data sets. The software handles all the required steps: rapid entry, organization, and navigation through tera-scale EM image collections. Semi- and automatic image registration is easily perfomed within and across sections. Efficient tools enable manipulating, visualizing, reconstructing, annotating, and measuring neuronal components embedded in the data. An ontology-controlled tree structure is used to assemble hierarchical groupings of reconstructed components in terms of biologically meaningful entities such as neurons, synapses, tracts and tissues. TrakEM2 allows millions of reconstructed entities to be manipulated in nested groups that encapsulate the desired abstract level of analysis, such as “neuron”, “compartment” or “neuronal lineage”. The end products are 3D morphological reconstructions, measurements, and neural circuits specified in NeuroML and other formats for functional analysis elsewhere.

3TrakEM2 has been used successfully for the reconstruction of targeted EM microvolumes of Drosophila larval central nervous system, for array tomography, for the reconstruction and automatic recognition of neural lineages in LSM stacks, for the reconstruction of thalamo-cortical connections in the cat visual cortex and for the reconstruction of the inhibitory network relating selective-orientation interneurons in a 10 Terabyte EM image data set of the mouse visual cortex, amongst others.

4An EM volume large enough to encapsulate significant fractions of neuronal tissue and with a resolution high enough to discern synapses presents numerous challenges for visualization, processing and annotation. The data generally consists of collections of 2d image tiles acquired from serial tissue sections (Figure 1;, ) or from the trimmed block face (Block-face Serial EM or SBEM,,; focused ion beam scanning EM or FIBSEM, ) that are collectively far larger than Random Access Memory (RAM) of common lab computers and must be loaded and unloaded on demand from file storage systems. Additional experiments on the same data sample may have generated light-microscopical image volumes that must then be overlaid on the EM images, such as in array tomography, or correlative calcium imaging,. TrakEM2 makes browsing and annotating mixed, overlaid types of images (Figure S1) over terabyte-sized volumes fast (Text S1, section “Browsing large serial EM image sets”) while enabling the independent manipulation of every single image both from a point-and-click graphical user interface (GUI; Figure 1e, S2, S3, S4) and by automatic means (Text S1, section “Image adjustment”).

5The images acquired with the EM microscope represent views of tissue that has been deformed by the sectioning process, by the heat of the electron beam, by charging effects, and by the magnetic lenses. For serial sections, part of the section may be hidden away by a section fold or support-film fold (Figure S5), and counterstaining with heavy metals further increases the difficulty of the task by occluding parts of the section with accidental precipitates (Figure S5). All images require illumination adjustments (Figure S5, S6).

6TrakEM2 recovers the original sample present in the resin block from the images with a robust automatic multi-step image registration approach. First images are corrected for distortions induced by the EM magnetic lenses. Then, image tiles belonging to individual sections are montaged combining a linear alignment established from invariant image features (SIFT; ) and an elastic alignment that compensates for the remaining non-linear distortion.

7Similarly, the section series are aligned by firstly using invariant features to estimate a linear transformation followed by elastic alignment to compensate for non-linear distortion. Alternatively to an immediate elastic alignment of the series of montages, feature correspondences can be used to estimate each image tile’s globally optimal pose with respect to overlapping tiles within the same section and in adjacent sections. This method enables the reconstruction of section series from section montages that cover only a few regions of interest disconnected in the section plane but related across sections (e.g. sparse images of different branches of a neuron). The methods implemented for montaging, global tile pose estimation and elastic alignment calculate global alignments for groups of images while explicitly minimizing the local deformation applied to each single image. Only by that constraint, very large montages or series of montages can be aligned without accumulating artificial deformation.

8In combination, TrakEM2’s alignment and deformation correction tools both manual and automatic allow high quality volume reconstruction from very large section series. Complex imaging arrangements are supported, including low-resolution images of large fields of view that were then complemented with high-resolution images for areas of interest, or different tilts of the same section. Tens of thousands of images are registered with an off-the-shelf computer in a few days.

9Both linear and non-linear transformations are expressed with a system that brings pixels from the original image space to the transformed space in one single computational step, concatenating all transformations and expressing the final transformation in the precomputed mipmap images (Figure 1d; Text S1, section “Browsing large serial EM image sets”). Additionally, the TrakEM2 GUI enables direct point-and-click manipulation of the transformation of any image in the volume, before or after the automatic registration without significant cost in data storage (relative to the dimensions of the image) or image quality (Text S1, section “Assembling the volume with automatic registration of image tiles” and “Manually correcting automatic image registration with affine and non-linear transformations”; Figure S2, S3).

10The second step in neuronal circuit reconstruction consists in identifying and labeling the neurons and synapses in the image volume. The current gold standard is computer-assisted manual labeling, either by brushing 2d areas (,; not practical for large volumes) or by marking skeletons,,. Automated methods for neuronal reconstruction are currently the focus of intensive research in Computer Vision (for review see ). TrakEM2 offers manual and semi-automatic methods for image segmentation (Figure S7) and for sketching structures with spheres and tubes (Text S1, section “Stick-and-ball models”; Figure S8), and interfaces with automatic image segmentation programs (Text S1, section “Image segmentation for 3d object reconstruction”).

11Manual skeletonization of a neuronal arbor requires continuous recognition operations that are not always done with full confidence given ambiguity in the image data. In our experience an all-or-nothing approach (edge or no edge, that is, to connect two parts of a neuronal arbor or not) does not sufficiently express all the information available to the human operator. Therefore TrakEM2’s skeleton data types are composed of nodes and directional edges that express parent/child relationships between nodes with a confidence value that captures the degree of certainty in the continuity of the skeleton at that edge (Figure 2). Edge confidence values are particularly useful to restrict ulterior circuit analysis to the most trustable subsets of the skeletons. Additionally each node holds a list of text annotations (“tags”) to highlight structures of interest or to label nodes as places to branch out later (e.g. with a TODO tag), and also a radius value (treeline skeleton subtype) or a 2d area (areatree skeleton subtype) to render 3d skeletons as stick models or volumes, respectively (Figure 2; Text S1, section “Image segmentation for 3d object reconstruction”). To correct mistakes skeletons are cut or joined at any node. Node edges accept any color (e.g. to label a branch), or follow a color code that expresses betweeness-centrality (computed as in ) relative to other nodes, branches or synapses.

12Given the unreliability of human-based skeletonization (tracing) of neurons, TrakEM2 facilitates the revision of skeleton nodes. An interactive GUI table lists all skeleton nodes and sorts them by location, edge confidence or tags, allowing quick targeted review of interesting or problematic parts of the skeleton (Figure 2). To systematically review complete neuronal arbors, TrakEM2 generates sequences of images centered at each node (fly-throughs) for each skeleton branch (Figure 2) that exploit the human ability to detect small changes in optic flow: missassignments across sections are readily identified as sudden shifts in the field of view. This review method aids as well in locating unlabeled synapses and untraced branches.

13TrakEM2 expresses synapses with connector elements that relate areas or skeleton nodes with other areas or nodes. Each connector consists of an origin and a number of targets, each assigned a confidence value, to express from monadic to diadic and polyadic synapses (Figure 2h). To aid the systematic reconstruction of all upstream and downstream neuron partners of a specific neuron, TrakEM2 presents an interactive table that lists all the incoming and outgoing connectors of a skeleton, and who they connect to. Incomplete synaptic partners are then visited one at a time and reconstructed. All tables are dynamically updated as nodes and connectors are added to or removed from the skeletons. The resulting neuronal circuit is then exported in various formats including NeuroML.

14The reconstruction of one or a few neuronal arbors is very different to the reconstruction of a complete neuronal processing module. The main difference is the scale: the latter is generally composed of dozens or thousands of neuronal arbors. While a human operator tracks the identities of a small collection of elements with ease, the task becomes very time consuming and error prone for large collections of neurons. In our experience the cut off is at about 50 elements.

15Nesting arbitrary groupings of reconstructed elements collapses a collection of arbitrary reconstructions into a meaningful entity such as a neuron. For example, a neuron may be represented with a nucleus (represented by a sphere), an arbor (represented by an areatree) and a list of synapses (each represented by a connector). Large collections of neurons are grouped by modality (“sensory neurons” versus “motor neurons” or “interneurons”), or by lineage (such as “BLD5”, “DALcl2”, etc. in the fly larval brain), or by experimental condition (“GFP-labeled”, “RFP-labeled”), or by any desirable arbitrary grouping or nested groupings. Hierarchical grouping effectively reduces the complexity in the management of large collections of objects by collapsing them into high-level entities meaningful for the human researcher. These groups are application-specific and in TrakEM2 are constrained by a controlled vocabulary with the required hierarchical groups (Figure 3). With hierarchical data organization and a search tool that supports regular-expressions, TrakEM2 enables the location, manipulation, measurement (Text S1 “Measurements”; Figure S9) and visualization of entities at the desired level of abstraction, be it fragments of neurons, individual neurons, a lineage of neurons, neuronal circuits, or arbitrary compartments or areas of the brain.

16We have described the key properties of TrakEM2, an open source software that is optimized for neural circuit reconstruction from serial section EM image data sets. TrakEM2 answers the quickly growing demand for a flexible and robust application for implementing at tera-scale the workflows typical of current connectomics projects that require volumetric reconstruction, visualization, and analysis of objects observed through 2D images. In this way, TrakEM2 supports the quest of neuroscientists to obtain a complete picture of the circuits embedded in the densely connected neurons of nervous systems. Indeed, ever since Schwann’s theory of the cell and Cajal’s neuron doctrine, neuroscientists have struggled to describe the diversity of neurons in the brain and their synaptic contacts that define the neuronal circuitry underlying brain functions.

17The turning point in this quest occurred in 1986, when Sydney Brenner and collaborators published their monumental work, the complete wiring diagram of the nematode Caenorhabditis elegans, with only 302 neurons. The choice of organism was key to their success, given the technological means of the time. However, a quarter of a century later, no other central nervous system has been reconstructed in full.

18Brenner’s reconstruction of the C. elegans nervous system was performed largely without the assistance of a computer. The work consisted in photographing (with film) serial 50 nanometer sections of the nematode worm, and annotating neurons and synapses on paper prints. An early computer-based system was used for three-dimensional reconstruction of a few very small volumes. The introduction of personal computers in the mid-eighties opened the way for the development of the first computer-assisted reconstruction systems such as TRAKA and three years later Neurolucida (; MicroBrightField), bringing feasibility to computer-assisted neuronal reconstruction. Both these systems were oriented towards the reconstruction of labeled neurons at the optical level. They solved the data storage problem of the time, that very large fields of view were far too large for computerized storage, by operating on microscope stage coordinates rather than pixel coordinates in a digitized image. Meanwhile, the results of Moore’s Law, and improving electronic camera technology, have opened opportunities for storing and manipulating very large datasets of images. For large-scale serial section electron microscopy (EM) in its many variants (serial section electron tomography or SSET,; serial section transmission EM or ssTEM; block-face EM or SBEM; focused ion beam scanning EM or FIBSEM, ), coupling live imaging with neuronal reconstruction would result in damage to, and eventually disruption of, the nanometer-thick sections, or it is not possible (such as in block-face EM or FIBSEM). Acquiring images first and then performing the analysis offline is necessary.

19The software IMOD revolutionized EM image volume analysis with tools for visualizing and aligning the sections of image stacks, and for manually counting, measuring and modeling objects in the 3d volume. The software Reconstruct catered to the special needs of neuronal reconstruction from EM, namely tools for manual and semi-automated image registration within a section (montaging, for large fields of view) and across serial sections, and tools for volumetric reconstruction and measurement of neuronal structures. The software package ir-tools made new developments of the computer vision field accessible for serial EM reconstructions, including automated image montaging and contrast limited adaptive histogram equalization for image enhancement (CLAHE; ), among others. All these softwares evolved considerably since their publication dates and complement each other to various degrees. Originally, each was designed with specific technological problems and scientific questions in mind.

20TrakEM2 is deployed along with all the necessary image processing libraries with Fiji, an open source image processing application. Fiji provides automatic deployment of software updates and comprehensive documentation via a publicly accessible wiki (http://pacific.mpi-cbg.de). Fiji supports a variety of scripting languages useful for the programmatic manipulation of data structures in TrakEM2. The functionality and batch-processing capabilities of TrakEM2 are extensible at will.

21TrakEM2 has already been employed in a variety of applications. While originally designed for reconstructing neural circuits in anisotropic serial section EM (for example, see,, ), researchers have found TrakEM2 useful for other EM modalities, for example for registering series of images from FIBSEM and annotating synapses by hand. The segmentation tools have been used for generating a gold standard segmentation of brain tissue to compare with the output of automatic segmentation algorithms on EM images, and for reconstructing neuronal lineages and organs in laser-scanning microscopy data sets.

22TrakEM2 must evolve as new imaging methods deliver higher-resolution data sets of ever increasing volumes. The open source nature of TrakEM2 allows any researcher to modify the program to suit specialized needs, and to incorporate implementations for novel algorithms from the computer vision and image processing fields. For example, TrakEM2 currenty exploits the anisotropic nature of serial section EM data, in which the X and Y dimensions have about 10 times higher resolution than Z (which is limited by the thickness of the section). Now, novel algorithms for tomographic reconstruction of serial sections and more isotropic EM imaging with BFSSEM and FIBSEM suggest that the approach, which limits the manipulation of image data to the XY plane will need to evolve to meet this challenge. General improvements in data storage and computing capacity will be very helpful for handling the coming new kind of large isotropic high-resolution EM data sets.

23TrakEM2 source code is under a distributed version control system (git) that encourages forking the source code base, while retaining the capability of contributing back to the main development branch. TrakEM2 has been publicly available as open source since day one. The many contributions of interested users and developers have, and will, greatly enhance the utility of TrakEM2, for the benefit of all.

24TrakEM2 has been written using the Java programming language and uses numerous image processing libraries including ImageJ (Wayne Rasband), mpicbg (Stephan Saalfeld), LOCI bio-formats, ImgLib (Stephan Preibisch, Stephan Saalfeld, Tobias Pietzsch and others), ImageJ 3D Viewer, Stitching, bUnwarpJ, JaMa (Mathworks and NIST), postgresql-jdbc, JFreeChart (jfree.org), edu_mines_jtk (Dave Hale), Level Sets (Erwin Frise) and Simple Neurite Tracer, among others. The source code is released under the General Public License and is under version control with git at http://repo.or.cz/w/TrakEM2.git. Binaries are distributed with Fiji (Schindelin et al, submitted to Nature Methods) via the automatic plugin updater.

25The EM data used here to exemplify the use of TrakEM2 corresponds to the abdominal neuropil of the first instar larva of Drosophila, and will be made available in full elsewhere.