BitFlow frame grabber technology has been incorporated into a prototype Augmented Reality Microscope (ARM) platform that researchers at Google AI Healthcare (Mountain View, CA) believe will accelerate the adoption of deep learning tools for pathologists around the world in the critical task of visually examining both biological and physical samples at sub-millimeter scales.
The application driving the ARM platform runs on a standard off-the-shelf computer with a BitFlow Cyton CoaXPress (CXP) 4-channel frame grabber (CYT-PC2-CXP4)
connected to an Adimec S25A80 25-megapixel CXP camera for live image capture, along with an NVidia Titan Xp GPU for running deep learning algorithms. Using Artificial Intelligence (AI), the platform enables real-time image analysis and presentation of the results of machine learning algorithms directly into the field of view.
Importantly, the ARM can be retrofitted into existing light microscopes found in hospitals and clinics around the world using low-cost, readily-available components, such as the BitFlow Cyton frame grabber, and without the need for whole slide digital versions of the tissue being analyzed. This innovation comes as welcome news: despite significant advances in AI research, integration of deep-learning tools into real-world diagnosis workflows remains challenging because of the costs of image digitization and difficulties in deploying AI solutions in microscopic analysis. Besides being economical, the ARM platform is application-agnostic and can be utilized in most microscopy applications.
According to Google researchers, opto-mechanical component selection were driven by final performance requirements, specifically for effective cell and gland level feature representation. The Adimec camera's 5120×5120 pixel color sensor features high sensitivity and global shutter capable of capturing images at up to 80 frames/sec, while the BitFlow Cyton CXP-4 has a universal PCI-E interface to the computer that simplifies set-up. The eMagin SXGA096,1292×1036 pixel microdisplay mounted on the side of the microscope includes an HDMI interface for receiving images from the computer. This opto-mechanical design can be easily retrofitted into most standard bright field microscopes. Including the computer, the overall cost of the ARM system is at least an order of magnitude lower than conventional whole-slide scanners, without incurring the workflow changes and delays associated with digitization.
The basic ARM pipeline consists of a set of threads that continuously grab an image frame from the camera, debayer it to convert the raw sensor output into an RGB color image, prepare the data, run the deep learning algorithm, process the results, and finally display the output.
Google researchers believe that the ARM has potential for a large impact on global health, particularly for the diagnosis of infectious diseases, including tuberculosis and malaria, in developing countries. Furthermore, even in hospitals that will adopt a digital pathology workflow in the near future, ARM could be used in combination with the digital workflow where scanners still face major challenges or where rapid turnaround is required as is the case with cytology, fluorescent imaging, or intra-operative frozen sections.
Since light microscopes have proven useful in many industries other than pathology, the ARM can be adapted for a broad range of applications across healthcare, life sciences research, and material science. Beyond the life science, the ARM can potentially be applied to other microscopy applications such as material characterization in metallurgy 12 and defect detection in electronics manufacturing.