Open access 3D electron microscopy datasets of brains

One of the coolest technical developments in neuroscience during the last decade has been driven by 3D electron microscopy (3D EM). This allowed to cut large junks of small brains (or small junks of big brains) into 8-50 nm thick slices, which are then imaged with nanometer resolution, resulting in 3D stacks of imaged tissue. Here, I want to highlight some of those datasets which are easily accessible in the internet but, at least from my impression, under-used by other researchers.

Apart from that, also the technical concepts and breakthroughs underlying this development are very interesting. The three main approaches, serial block-face electron microscopy (SBEM), serial section transmission or scanning electron microscopy (ssTEM or ssSEM) and focused ion beam SEM (FIB-SEM) have been very nicely reviewed by a colleague of mine, Benjamin Titze, including some very beautiful and instructive figures (special recommendation for Fig. 4). Of course, this is only a part of the challenge: First, the brain tissue must be stained with heavy metals to be visible for electrons. Second, after the acquisition, human annotators or machine learning have to extract neuronal morphologies or synapse distributions from the huge datasets.

However, I find also the raw 3D EM data very interesting. Those datasets are still rare, but I think that many people do not know that some of them are easily accessible to anyone with an internet connection. And it is a true pleasure to have the full screen filled with the overwhelming clutter of neuronal dendrites and to follow them in 3D just by scrolling with the mouse. is probably the best place to start. After a simple registration, one can directly access some of those EM datasets in the browser:, or through other tools. Not all of the datasets are of the highest quality (and it is not always easy to judge data quality for a lay person), but most of them offer highly interesting views into the complexity of the brain (scroll wheel for going through the slice, Ctrl + scroll wheel for zooming). Here I want to highlight a few of them. They can be accessed by clicking on the neurodata/ndwebtools link above.

The following excerpt by Lee et al. (2016) shows a small zoom-in into the somato-sensory cortex in mouse. A thick dendrite (red arrows) is passing vertically through the image. In this ssSEM datasets, synapses look really nice (yellow arrow, with a beautiful vesicle cloud below), but they look even nicer in 3D, so you should have a look at the 3D data yourself.


The following picture from a dataset of the Cardona lab shows a small zoom-in of the drosophila brain. (I assume that the scale bar generated for this dataset is a bit off; the 100 nm shown here probably correspond to 500 nm in reality.) The red arrows highlights a filament of the cytoskeleton, probably a microtubule in charge of transport along the dendrite. The pink arrow indicates one of the many mitochondria with its cristae. I wonder where in dendrites the mitochondria are more likely to occur… The yellow arrow indicates a local darkening at the contact site between two neurites, and I have no idea what this is. A gap junction? A strange synapse? A precipitate, i.e., an artifact of the staining procedure?


In hippocampus CA1, things look very similar, in a ssTEM dataset used by Bloss et al. (2018). This study focuses on clustering of synapses from single axons. Axons can easily be recognized by their dark and thick myelin sheath (red arrows). If you have a lot of time, you can scroll through the dataset and try to find a node of Ranvier. – As in almost all datasets, there are planes or entire regions with low quality staining or low signal to noise imaging or something else that went wrong. Sometimes this is very local, just a blurring of boundaries (yellow arrows) that is difficult to interpret.


And here is a zoomed-out view of a single plane of a dataset by Wanner et al. (2016) of the olfactory bulb of larval zebrafish. Here, the large roundish shapes are not cross-sections of dendrites, but neuronal somata:


I just want to encourage people to browse through these datasets. Browsing in 3D is much more interesting than watching these still images. – Or if you are teaching students about neuroscience, why not send them a link such that they can discover neurons themselves by scrolling and zooming through the brains? I haven’t seen many people who were not fascinated when first encountering 3D EM data and not overwhelmed by the sheer amount of dendritic arborizations.

(And this is a bit funny, if we keep in mind that electron microscopy does not see much of the more complex level of cells, the crowded microenvironment, which is a chaos of competing, interacting, diffusing little protein machines.)

As an alternative to that is accessible even without any registration, a couple of test datasets are available with neuroglancer, a rendering software developed by Google. Check out the dataset from Takemura et al. (2015) by following this link. It is an isotropically resolved dataset (8 nm in x, y and z). You can use the scroll wheel and Ctrl to browse through the stack or to zoom in and out. The software includes three EM viewports and an additional rendering of a number of selected neurons.

Another way to explore 3D EM data is to go to, where one can discover 3D EM datasets of neurons (retina, based on Briggman et al., 2011) within the framework of a game – which is fun. Over the last couple of years, the user interface has become very pleasant. The downside compared to the other options is that one cannot discover freely in a big dataset; plus, there is no labeling of the inner organelles or vesicles of the neurons, which is part of the fun for the other datasets.

To understand more details within these EM images, I found it interesting to go through the first chapter of the book Dendrites (“Dendritic structure”), which can accessed almost to its full extent via Google Books.

Full disclosure: my current host lab is working on 3D EM data in zebrafish. My own projects do not involve electron microscopy directly.

Posted in Data analysis, machine learning, Microscopy, Network analysis, zebrafish | Tagged , , , | Leave a comment

How well do CNNs for spike detection generalize to unseen datasets?

Some time ago, Stephan Gerhard and I have used a convolutional neural network (CNN) to detect neuronal spikes from calcium imaging data. (I have mentioned this before, here, here, and on Github.)

This method is covered by the spikefinder paper that was recently published (Berens et al., 2018), based on a competition that featured ground truth for a training and a test dataset. This was a great and useful competition. But there are some important caveats (which are mentioned in the discussion of the main paper). Here, I will discuss one of the caveats.

For the competition, the test set consisted of five datasets; neurons not included in the test set, but from the same original datasets with similar conditions and signal-to-noise etc. had been included in the training datasets. Therefore, the competition does not allow to understand how well the networks would generalize to neurons from datasets that have not been used for training. In theory, there would be one algorithm to learn everything and to predict everything (case 1):


But of course it was possible to e.g. train one method for each dataset separately (case 2), without violating the conditions of the competition (the left column symbolizes the training datasets, the right column the test datasets, colors indicate the same original datasets):


And of course there is a smooth transition between the first and the second possibility, such that it is not easy to judge how closely an algorithm has been fine-tuned for the respective datasets.

Fine-tuning would allow to perform very well in the competition with an algorithm similar to case 2, but there would be no means to easily generalize the method to unseen datasets – and that’s not really desirable.

It is obvious why the competition necessarily had this ‘design flaw’: To circumvent this problem, the training set would have to consist of maybe 15-20 datasets of a couple of neurons each, and the test dataset of ~10 independent datasets (this would be my own rough estimate). Currently, nobody has this amount of ground truth data.

However, I’m interested in applying the prediction method that we developed to new calcium imaging, so I wanted to know how well the network generalizes. Partially, this has already been investigated by Pachitariu et al. (bioRxiv, 2017), but this study did not consider the test datasets directly (which were not public at this point in time).

So I basically trained the CNN algorithm (this one) on all datasets, except on the one I wanted to use as test dataset. Something like this (case 3):


Then I compared the result with the predictions from the normal training (‘case 1’) for the test data. Here are the results:


Each black dot corresponds to the prediction performance for one neuron (the greater the value, the better). The left-hand datapoints for each of the five datasets are results from a network that has been trained on all datasets (case 1), whereas right-hand datapoints are results from a network that has not been trained with other neurons from the same original dataset (case 3). Blue dots indicate the mean across neurons for each dataset. The difference indicated at the top is the pseudomedian difference (‘case 1’ minus ‘case 3’), the test used is a non-parametric one.

Judging from this plot, I would say that the predictions are maybe slightly worse, but not much for ‘case 3’ compared to ‘case 1’ (no statistical differences found, but I think there is a tendency).  This is pretty reassuring with respect to the power of the algorithm/network to generalize to unseen datasets – it works.

Posted in Calcium Imaging, Data analysis, machine learning, Neuronal activity | Tagged , , , , , , | Leave a comment

A list of cognitive biases

There are a handful of cognitive biases that are well-known to most scientists: confirmation bias, the Dunning-Kruger effect, the hindsight bias, the recency effect, the planning fallacyloss aversion, etc.. Although they should not be taken as universal laws (for example, recently there was some criticism of generalizations of the loss aversion concept), but it is still important to understand which – probably unconscious – biases might shape our behavior, both as humans and as scientists.

I think that some of thoses biases can be useful to think over if one wants to become better at planning (for example of scientific experiments) or better at understanding data and one’s own (biased) interpretation of it. An unusual resource on cognitive biases that I can recommend is the first third of HPMOR, a Harry Potter fanfiction that discusses cognitive biases in the context of an entertaining narrative. And wikipedia offers a more or less comprehensive list of such biases: List of cognitive biases.

I found it very interesting to read through this list. Although some of it is kind of common sense, having a name for a phenomenon can make a difference – similarly, if I know the names of all the trees and plants, wandering through a forest is different from before, because I start to see things not only with my eyes.

Posted in Data analysis, Links | Tagged | 2 Comments

Springtime for two-photon microscopy

Today, the fields and forests around Basel are full of flowers that try to disseminate their pollen. Fixed pollen are, apart from sub-diffraction beads and the convallaria rhizome, one of the most commonly used test/reference samples for fluorescence microscopy. This is both due to their fine, spiky structures and their strong autofluorescence. The scientific study of pollen (and other small things), palynology, provides us with elaborate protocols on how to collect, clean, stain and fix pollen (example 1example 2) with glycerol jelly between two glass slides.

For two-photon microscopy, these protocols are not ideal since the objectives typically have no correction for glass cover slips between the sample and the objective. Therefore I tested whether it would be possible to look at pollen with a much simpler protocol, using a two-photon microscope. Continue reading

Posted in Imaging, Microscopy | Tagged , , , | Leave a comment

Layer-wise decorrelation in deep-layered artificial neuronal networks

The most commonly used deep networks are purely feed-forward nets. The input is passed to layers 1, 2, 3, then at some point to the final layer (which can be 10, 100 or even 1000 layers away from the input).  Each of the layers contains neurons that are activated differently by different inputs. Whereas activation patterns in earlier layers might reflect the similarity of the inputs, activation patterns in later layers mirror the similarity of the outputs. For example, a picture of an orange and a picture of a yellowish desert are similar in the input space, but very different with respect to the output of the network. But I want to know what happens in-between. How does the transition look like? And how can this transition be quantified?

To answer this question, I’ve performed a very simple analysis by comparing the activation patterns of each layer for a large set of different inputs. To compare the activations, I simply used the correlation coefficient between activations for each pair of inputs. Continue reading

Posted in Data analysis, machine learning, Neuronal activity | Tagged , , , , , | Leave a comment

Understanding style transfer

‘Style transfer’ is a method based on deep networks which extracts the style of a painting or picture in order to transfer it to a second picture. For example, the style of a butterfly image (left) is transferred to the picture of a forest (middle; pictures by myself, style transfer with


Early on I was intrigued by these results: How is it possible to clearly separate ‘style’ and ‘content’ and mix them together as if they were independent channels? The seminal paper by Gatys et al., 2015 (link) referred to a mathematically defined optimization loss which was, however, not really self-explanatory. In this blog post, I will try to convey the intuitive step-by-step understanding that I was missing in the paper myself. Continue reading

Posted in Data analysis, machine learning | Tagged , , , | Leave a comment

Can two-photon scanning be too fast?

The following back-of-the-envelope calculations do not lead to any useful result, but you might be interesting in reading through them if you want to get a better understanding of what happens during two-photon excitation microscopy.

The basic idea of two-photon microscopy is to direct so many photons onto a single confined location in the sample that two photons interact with a fluorophore roughly at the same time, leading to fluorescence. The confinement in time seems to be given by the duration of the laser pulse (ca. 50-500 fs). The confinement in space is in the best case given by the resolution limit (let’s say ca. 0.3 μm in xy and 1 μm in z).

However, since the laser beam is moving around, I wondered whether this may influence the excitation efficiency (spoiler: not really). I thought that his would be the case if the scanning speed in the sample is so high that the fs-pulse is stretched out so much that it spreads over a distance that is greater than the lateral beam size (0.3 μm FWHM).

For normal 8 kHz resonant scanning, the maximum speed (at the center of the FOV) times the temporal pulse width is, assuming a large FOV (1 mm) and a laser pulse that is strongly dispersed through optics and tissue (FWHM = 500 fs):

Δx1 = vmax × Δt = 1 mm × π × 8 kHz × 500 fs = 0.01 nm

This is clearly below the critical limits. Is there anything faster? AOD scanning can run at 100 kHz (reference), although it can not really scan a 1 mm FOV.  TAG lenses are used as scanning devices for two-photon point scanning (reference) and for two-photon light sheet microscopes (reference). They run at up to 1000 kHz sinusoidal. This is performed in the low-resolution direction (z) and usually covers only few hundred microns, but even if it were to cover 1 mm, the spatial spread of the laser pulse would be

Δx1 = 1 mm × π × 1000 kHz × 500 fs = 1 nm

This is already in the range of the size of a typical genetically expressed fluorophor (ca. 2 nm or a bit more for GFP), but clearly less than the resolution limit.

However, even if the infrared pulse was smeared over a couple of micrometers, excitation efficiency would still not be decreased in reality. Why is this so? It can be explained by the requirement that the two photons arriving at the fluorophor have to be absorbed almost ‘simultaneously’. I was unable to find a lot of data on ‘how simultaneous’ this must be, but this interaction window in time seems to be something like Δt < 1 fs (reference). What does this mean? It reduces the true Δx to a fraction of the above results:

Δx2 = 1 mm × π × 1000 kHz × 1 fs = 0.003 nm

Therefore, smearing the physical laser pulses (Δx1) does not really matter. What matters, is the smearing of the temporal interaction window Δt over a spatial distance larger than the resolution limit (Δx2). This, however, would require a line scanning frequency in the GHz range – which will never, ever happen. The scan rate must always be significantly higher than the repetition rate of pulsed excitation. The repetition rate, however,  is limited to <500 MHz due to fluorescence lifetimes of >1-3 ns. Case closed.

Posted in Imaging, Microscopy | Tagged , , , , | 8 Comments