We compare foundation models (SAM and DINO variants) with classical features from ilastik and supervised learning. The FM features outperform classical approaches. Attentive probing is better than RF, but too slow for interactive use. See example results in the fig.

We use embeddings from foundation models as features, either for a RF or different attentive probing variants. The features are averaged over masks for object classification. See the image for a method overview.

Segmenting cells in microscopy is much easier these days thanks to foundation models. Can we use these models for other tasks, e.g. cell classification? See our latest work! We find big improvements for object and pixel classification compared to classical approaches.

How can we use foundation models such as (micro)SAM to improve electron microscopy segmentation? Check out our preprint! We found substantial improvements for nucleus, mito, and neurite-segmentation using initialization and semi-supervised learning with foundation models.

Looking for a PhD position at the intersection of AI, imaging, and gene therapy? Apply for this position in my lab: tinyurl.com/2a2v6tvx

Part of sfb1690.uni-goettingen.de that studies hearing, vision, and more.

Plus, you can create pretty pictures like the one below :).

We comprehensively evaluate our method and use it to analyze intact mouse and gerbil cochleae, including SGN sub-types, and to validate (opto-)genetic therapies preclinically (see screenshot). CochleaNet is also applicable to lower-resolution data from commercial systems.

Cochleae were cleared, stained and imaged with a high-resolution light-sheet microscope (doi.org/10.1038/s41587…). After preprocessing,we segment and analyze the data with CochleaNet, using the three dedicated networks trained on newly annotated data.

Preprint alert! CochleaNet, our framework for analyzing cochlear light-sheet data. It consists of three networks to segment spiral ganglion neurons, inner hair cells, and detect synapses. Rendering of a full cochlea below, find the preprint at doi.org/10.1101/2025.1…

3. The model selection now supports two additional models (Histopathology, Medical Imaging) and uses human readable names, see the screenshot below.
For more on these models check out: computational-cell-analytics.github.io/micro-sam/micr…

2. We added preliminary support for automatic tracking via trackastra, developed by @martweig. See the video for an example result and check out computational-cell-analytics.github.io/micro-sam/micr… for details.

Our next micro_sam release is here! We have a new model for light microscopy, that massively improves for automatic segmentation! See the qualitative and quantitative comparison in the images, v2 is our previous version, v3 is the new one.

Our first deposition of synaptic vesicles segmentations is now in the Cryo ET Portal! We segmented vesicles in over 50 tomograms to enable analysis of membrane proteins and more.
cryoetdataportal.czscience.com/depositions/10…

Our models form the basis of micro_sam, our napari plugin for interactive and automatic segmentation. It can segment data in 2D, 3D and across time. You can find all the details at github.com/computational-…

To achieve these improvements, we finetune SAM on a LM dataset (ca. 50k images, 1.2M annotated cells) and on a EM dataset (ca. 5k images, 90k annotated nuclei and mitos). We also add a new decoder for instance segmentation, which provides efficient automatic segmentation.

After a long journey, Segment Anything for Microscopy is now published in Nature Methods! We significantly improve SAM for interactive and automatic segmentation in light and electron microscopy and build a user-friendly tool. nature.com/articles/s4159…

PathoSAM improves SAM forinteractive segmentation of nuclei and adds automatic and semantic segmentation. According to our experiments, it is SOTA for instance segmentation (see ) and performs well for semantic segmentation.

Announcing PathoSAM, our foundation model for nucleus segmentation in histopathology. PathoSAM supports interactive and automatic nucleus segmentation (instance and semantic). See segmentation on a WSI from openslide, check out arxiv.org/abs/2502.00408 or read on for details.

Our latest preprint PEFT-SAM is on arxiv! We study parameter efficient finetuning to adapt SAM to biomedical images and introduce a new workflow for adaptation based on only two labeled images. See our workflow and improvements, check out arxiv.org/abs/2502.00418 for more.

Our work, MedicoSAM, is now on arXiv! We study finetuning SAM for medical images. MedicoSAM improves over SAM and other models, especially for interactive segmentation, see examples in the figure. Preprint at arxiv.org/abs/2501.11734. Read on for a summary.
@AnwaiArchit

There was quite some interest in SynapseNet, our tool for synapse analysis in electron micrographs, since we announced it. If you want to use it yourself you can now find a video tutorial that shows how to use its napari plugin.
youtu.be/7n8Oq1uAByE

First, to answer the question, according to SynapseNet the electron tomogram of the mossy fibre synapse contains 9,061 vesicles, see the reconstruction of all vesicles rendered in orange. It can also identify active zones (blue), mitochondria (red, cyan) and other structures.

Want to know how many vesicles there are in a mossy fibre synapse, without counting thousands of vesicles by hand in electron micrographs? Our new tool SynapseNet has you covered and is now available as a preprint biorxiv.org/content/10.110…. Read on for a short overview.

I have reactivated the bsky account and be more active there in the future. Follow for updates on our groups work. (Will still post here for the time being.)

Tried molmo.allenai.org for the first time. Cool to have an open source VLM, but still hilariously bad for microscopy and gaslights you about what it does :D.

Is GPT 4o a better microscopy image analyst than earlier versions? Doesn't seem so, at least not when it comes down to identifying mitochondria in EM

Announcing our new and stable release of Segment Anything for Microscopy!
We introduce new features and improve the tool to segment microscopy data with a few clicks, like the mitochondria in the video, first segmented automatically and then corrected interactively.

E.g. gets count for this plate from AGAR dataset right (123 colonies). However, when prompted to explain approach is very vague, and also can't/wont give json with exact colony locations. (Have not tried to circumvent this yet).

Automatic 3d segmentation works in 2 steps:
First segment all objects in a given slice, then segment those objects in the volume.

Release 0.3.0 for micro_sam is there!
- Support for polygon and ellipse prompts
- Automatic instance segmentation in 3d
- Faster fine-tuning
See the advantage of polygon over box annotation for a cell with complex shape below!

Everything at a glance:
- preprint: biorxiv.org/content/10.110…
- software: github.com/computational-…
- documentation: computational-cell-analytics.github.io/micro-sam/micr…
- tutorials: youtube.com/watch?v=ket7bD…

... and the video tutorial for the tracking annotation tool: youtu.be/Xi5pRWMO6_w.

... the video tutorial for the 3d annotation tool: youtu.be/PEy9-rTCdS4, ...

... in the video tutorial for the 2d annotation tool: youtu.be/ket7bDUP9tI, ...

One of our main finding: finetuning the SAM model significantly improves performance for light microscopy datasets, including image settings **we have not trained on**. Italic font datasets in the image were used for training (results on a test split), normal font not!

Our micro_sam preprint is out! We finetune Segment Anything for Microscopy and build interactive napari annotation tools based on it. See preprint at biorxiv.org/content/10.110…, software at github.com/computational-… and more info in this thread.

Similar improvements for cells, neurons and nuclei with the EM model!

Another big micro_sam update! We have released the first batch of finetuned models for microscopy and the code for training/finetuning your own models. See the improvements due to the LM model in the figure! (cyan=prompt, yellow=correct segmentation, red=SAM prediction).

@stevejohnryan Segmenting neurites in EM works quite good, see the example video for our example data from github.com/computational-…. However, it has a limitation for branching structures, where it doesn't work well.

And we integrate the first fine-tuned models from @AnwaiArchit. See how they improve segmentation compared to the standard segment anything below! (Standard on the left, fine-tuned on the right)

We improved the automatic instance segmentation to enable instance rerun when changing parameters, which makes it much easier to find good settings:

New release alert! We published a big update for segment anything for microscopy github.com/computational-…. Now supporting tiled prediction, better instance segmentation and initial fine-tuned models.
See how smooth interactive annotation runs now for a large image!

I am looking for a PhD student to work on applications of vision foundation models in biology:
Make segmentation and tracking for biology effortless! (See the gif for our initial attempts). Apply at uni-goettingen.de/de/644546.html…. Please RT!

The tracking annotator allows automatically tracking cells over time, and has support for dividing cells.

The 2d annotator also support using the automatic segmentation functionality of #SegmentAnything to initialize the segmentation (see gif below), and bulk segmentation from bounding boxes (see the gif in the first tweet).

We support interactive 2d, 3d segmentation and tracking from bounding box and point annotations. See for example segmenting a mitochondrion in 3d from point annotations in a single slice:

Introducing #SegmentAnything for microscopy, our napari based tools for interactive microscopy annotation: github.com/computational-…

We also study different adaptation strategies (joint vs. two-stage training ) and demonstrate our approach for three applications: cell segmentation in phase-contrast microscopy, mitochondria segmentation in EM and lung segmentation in X-Ray.

We build on current self-training methods for semi-supervised learning and domain adaptation (FixMatch, AdaMatch, MeanTeacher) and comine them with the probabilistic UNet. We use samples from the PUNet to estimate pseudo-label masks and show how this improves domain adaptation.

If you need to visualize, analyze and share large and multi-modal microscopy data read on:
MoBIE has now been published in Nature Methods: nature.com/articles/s4159…
Particular thanks to @Sci_Wanderlust and @tischitischer who co-developed the core MoBIE functionality.

Glad I am not applying for the Walter-Benjamin-Program! (CV needs exact school dates from the first grade...)