Ludwig Maximilian University Develops Deep Learning System to Measure Ice Thickness in Cryo-EM Samples
“MathWorks provided us with professional tools and support to create a standalone deep learning application that saves us enormous time and costs in screening the quality of cryo-EM samples.”
Challenge
Design a deep learning system that can automatically check the ice thickness of cryo-EM samples
Solution
Use Deep Learning Toolbox to develop image segmentation and classification models to detect sample grids and classify them based on ice thickness
Results
- Quality screening time reduced
- Screening costs reduced
- Researcher time saved
The widefield light microscope used for the fast grid screening.
The Gene Center at Ludwig Maximilian University of Munich (LMU) uses cryogenic electron microscopy (cryo-EM) to examine the structure of proteins and biological complexes. This process involves carefully screening the ice thickness—a measure of sample quality—of frozen grid samples, which is a very slow and expensive task when done with an electron microscope.
Dr. Christophe Jung, a microscopy specialist, and Dr. Markus Hohle, a math and physics lecturer from the LMU Gene Center, recently developed a deep learning tool in MATLAB® that can determine the thickness of frozen grid samples by analyzing images captured with a color camera. The tool has enabled LMU researchers to save time and money by first screening frozen samples with deep learning before identifying and sending optimal samples to the electron microscope for closer examination.
“The deep learning tool we built with MATLAB has been a game-changer in our laboratory research work,” says Dr. Hohle. “Strong professional support and educational material made the difference in choosing MATLAB over other alternatives.”
Challenge
An electron microscope facility costs $3–5 million to set up and $4,000–5,000 per day to run, which puts it beyond the reach of many individual research groups. Additionally, with many samples needing examination, the microscope is often overbooked.
Screening sample quality with the electron microscope is also slow. Each grid tile must be imaged separately and assessed for ice thickness quality. The visual assessment of ice thickness from the EM images requires significant experience and results may vary depending on the individual researcher. Even with automated sample loading and acquisition, an electron microscope cannot screen more than 12 grids per day, which is well below the dozens of samples that require examination.
There have been previous attempts to automate the screening process, including efforts that involved using deep learning algorithms. However, they all relied on images acquired from the electron microscope, which does not accelerate the screening process.
The researchers at LMU Gene Center were interested in creating a screening method that could determine the quality of samples from images captured by color cameras. This would enable them to complete some of the screening process on less expensive hardware and hence reduce the load and operating costs of the electron microscope.
Solution
The LMU team captured 650 images of cryo-EM samples containing a total of 4,000 tiles using an interferometric microscope and a color camera. They used Image Processing Toolbox™ to import their images into MATLAB and perform preprocessing on them.
The LMU team used Image Labeler to manually annotate the images. To improve deep learning models, the researchers developed custom MATLAB scripts that augmented training examples by performing various transformations, including rotating them randomly and cutting out different regions.
With Deep Learning Toolbox™, they trained two neural networks for image segmentation and classification. The first network, an image segmentation model based on Inception-ResNet-v2, recognizes single squares in each sample grid tile. The second neural network, based on the DarkNet-19 model, classifies the ice-thickness quality of each detected square. The researchers used pretrained deep learning models available in Deep Learning Toolbox and fine-tuned them on their own samples.
With Parallel Computing Toolbox™, the LMU team was able to configure their neural networks to take advantage of graphics processing units to speed up the training and inference process.
Finally, the researchers used MATLAB Compiler™ to package the application in an executable that can run on any Windows computer, even if it doesn’t have MATLAB. They named the application “Artificial Neural Network–Based Image Categorization and Analysis Software (ANNICAS).” After installation, ANNICAS runs in the background on the computer. It automatically monitors and analyzes new images added to a source folder and stores the results in a target folder.
LMU open-sourced the project and published it in addition to the scientific publication on GitHub to help other researchers build on and improve their work.
Results
- Quality screening time reduced. “With ANNICAS, we can screen 20 samples in 5 minutes,” said Dr. Jung. “Previously, the same batch required a full day’s time of work with the electron microscope.”
- Screening costs reduced. “Given the cost of running the electron microscope, screening each batch would cost us $4,000–5,000,” Dr. Hohle said. “By automating the process, we’re making enormous cost savings and also freeing up the microscope’s time for other important work.”
- Researcher time saved. “The manual process required students and researchers to spend hours staring at samples and evaluating their quality,” Dr. Hohle said. “With that out of the way, they can spend their time doing more meaningful work.”
