Cassandra-Based Image Processing: Two Case Studies KERRY KOITZSCH‘IMAGE AS BIG DATA SYSTEMS LLC’ (IABDS.COM)

Who Am I?

Image processing enthusiast Big data technology enthusiast Robotics enthusiast NoSQL database enthusiast Microscopy enthusiast Software engineer Author

Overview of this Presentation

This quick overview of some of our ongoing projects describes why Cassandra is a key part of our ongoing research, development, and client support activities.

The presentation highlights to areas of research which involve Cassandra technologies in the “images as big data” arena: an automated microscope slide application prototype, and a stereo ”smart sensor” mobile robot prototype.

Both of the use cases describe rely heavily on Cassandra and related “helper libraries” to provide data storage capabilities for the two software prototypes.

Throughout the presentation we discuss how the flexibility, high performance, and ability to “play well with” other components makes Cassandra an essential part of the applications described here.

Some Context About Our Use Cases

The automated microscopy case study is based upon our current POCs for a distributed microscope slide analytics application. In it, we use standard optical microscopes fitted with color digital cameras and an automatic slide-loading cartridge system to process “decks” of glass or plastic microscope slides, usually of human cell tissue of various kinds, the staining process depending on what the nature of the tissue is, and, in the case of hematology slides (human blood) no staining at all. Different resolutions of magnification, slide quality, and metadata must be processed.

The stereo vision robotics platform case study is based upon our current POCs for a stereo vision mobile robotics application. The application uses Hadoop, Spark, Flink, Solr, and a number of additional machine learning libraries to analyze stereo image pairs, build two- and three-dimensional models of the environment, and uses Bayesian sensor fusion to enhance the 3D models with additional environmental information.

Our Objectives and How Cassandra Helps us Achieve Them

In our use cases, we have to integrate mechanical control systems, complex ‘image as big data’ analytics, and a variety of data formats and metadata in a common representation and with efficient implementation.

Cassandra and associated ”helper libraries” such as Spring Data Cassandra, Solandra, and Katta allow us to maintain a flexible technology stack while using off-the-shelf proven technologies which “play well” together.

Cassandra has well-defined parameters for optimization, scalability, and availability. Cassandra enables us to meet our objective of maintaining flexibility while

maintaining high performance capabilities with a changeable server topology. Cassandra based technologies are flexible enough to accommodate a wide range of

different application domains, including microscopy (medical imagery of all kinds) and mobile robotics applications (stereo vision ‘image as big data’ technology)

Our little software secret! :D

We are not always software engineers…. Sometimes, we are software ARCHAEOLOGISTS! But what is software archaeology? And how does it assist us build

modern high-performance distributed software systems?

We are re-inventing software archaeology!

Software Archaeology is Based Upon:

The realization that the “first wave of AI technologies” (up to the mid-1990s) introduced a host of valid ideas, concepts, strategies, and approaches which were not completely assimilated into succeeding waves of AI technology.

The realization that “looking back at our software roots” may be a good thing. Identifying “best of breed” strategies of the past and re-thinking them, using up-

to-date software stacks. Some examples of these technologies include pre-Hadoop distributed processing systems and algorithms, DeKleer’s implementations of “problem solvers” [1], geometric toolkits such as VANTAGE and Geometer [2],[3].

This re-thinking and re-implementing process has already been successfully done for POCs, products, research extensions and particularly for software patents [4]…

Migrating ‘First Wave AI’ Concepts

Quality image processing systems have been lost in the course of time (for example General Electric’s Geometer [4], the Vantage modeling system [5], and many others) … while the implementation may need updating many of the concepts may be --- and have been --- reimplemented using modern technologies (and in particular Cassandra-centric technologies)

Migrating our inspirations to modern implementation *Inspiration* does not *equal* implementation: we may choose to re-

implement in Clojure rather than Lisp, for example, and we use Cassandra-centric technology stacks for almost *all* our implementations…

Why Our Case Studies are Cassandra-Centric

Cassandra is a flexible, full-featured noSQL database Integrates well with Hadoop, Spark, and Flink-based software components A lot of development focus has already been done on different technology stacks, for

example, the SMACK stack Integrates well with the battery of Spring Framework components we use Cassandra is scalable, available, and highly-tunable “Plays well” with Lucene/Solr based technologies (via Solandra and other “glue-ware”) It’s easy to integrate Cassandra into use cases and software prototypes, allowing a

“rapid prototyping/development” style Its also easy to use Cassandra in a plugin/module based architecture style Cassandra ’plays well’ with a variety of machine-learning libraries Cassandra is flexible enough for research, solid enough for products

POJOs to Cassandra via Spring Data

Plain old java objects (POJOs) may be mapped to Cassandra using Spring Data Cassandra

We would also like to map ontology objects to Cassandra We begin with fundamental feature objects of images: regions, contours,

lines, junctions, and points, the kind of object instances generated from, for example, connected component algorithms, edge extraction operators, and the like

For our domain examples, the extracted image metadata and associated “fused” data from other sensors [ for example microscope slide data sets might have other sensor results associated with them, like spectrometry and chromatographic data ]

Images ARE Big Data

Images have volume, variety, velocity, veracity While the image volume may be relatively low, the complexity of the

individual image objects may be extremely complex There may be associated signals (such as LIDAR and geolocation data) Images may be multispectral and multiresolution in nature Images may have significant associated metadata (as with medical

imagery) Images may play only one part in a larger “big data enterprise”

Considerations About the SMACK Stack

Case Study 1: An Automated Microscopy System

We want to be able to analyze large numbers of microscope slides automatically.

The image analysis component may be complex and time consuming: we require a distributed solution

Images may require cleaning, rescaling, metadata extraction, and image format conversion

Analysis results are stored in Cassandra database with associated search mechanisms via Solandra and custom code.

The physical apparatus is a standard medical optical microscope with automated stage and lighting controls and an automated microscope slide cartridge system holding 25 microscope slides per deck.

A View of the Physical Apparatus…

… And How We See the Problem...

Samples of Input Data from a typical data set

Cassandra-Centric Infrastructure

Cassandra’s flexibility allows us to adapt data models appropriately for new experiments and ideas.

We are also able to maintain consistent ontology development by progressively evolving our table definitions and POJOs.

We can develop our data models in separate Cassandra instances for purposes of experimentation.

Metadata can be easily expressed: this is particularly important for experimental medical data, as citations, references to other data sets, parameters, and associated sensor data may be present, as well as timestamps and audit trail / handler data if information comes from a doctors office, insurance company, or hospital.

Architecture Diagram, Case Study I

Canova is now DataVec

Ellipse Feature Extraction Example

Technology Stack, Case Study I

Mechanical controls are implemented in Java and C++: these control the microscope stage controls, lighting, positioning, cartridge control, and others

Timestamps are generated by the microscope controls and are associated with analysis records

Slide images may be analyzed one or more times using different data analysis pipelines

Slide images may be retained in the file system and subjected to feature extraction and analysis

We avoid saving images as blobs in Cassandra at this time We can “ramp up” our complexity gradually as we evolve our software. For example,

the next slide shows contour extraction from a microscope slide: the contour data may be stored directly into Cassandra.

Contour Extraction Example

Conclusions, Case Study I

We learned in this case study that Cassandra’s flexible data modeling and straightforward seamless integration with other ‘image as big data’ components, including machine learning libraries such as Mahout and deeplearning4j enabled us to build high performance slide analysis prototypes which were able to store complex microscopic slide information (derived from feature extraction) and metadata (associated with the image itself, such as citations, other sensor data, authorship and auditing/handler data, and the like)

Case Study II: A Stereo Vision Robotics Platform

Using a standard dual-image synchronized stereo camera on a mobile robotics platform, we are able to obtain timestamped, geolocated/microgrid located image pairs for analysis. Image pairs may be generated periodically to produce a set of sequential image pairs.

These image pairs are subjected to feature extraction, model building, and other analytic techniques to produce scene models, which are associated with scene ontologies and scene model templates to incorporate within a navigation program.

The navigation program and its data, along with internal sensors within the robot itself, allows the mobile robot to successfully navigate scenes and perform simple tasks.

The navigation program is also able to perform qualitative navigation based on landmarks.

Goals of the Stereo Vision Platform

Use a standard stereo image pair generating dual camera for experimentation with a mobile robot

We want to investigate two software problems: qualitative navigation and integrated sensor fusion with robotic control

We want to use inspiration from Poggio’s MIT Vision Machine in particular as a starting point for our software

However, we wish to use an integrated data pipeline design with “best of breed” third-party components, including Cassandra, Hadoop/Spark, and Solr.

We model much of our stereo three-dimensional modeling on Hartley and Zisserman [5] and Faugeras [6]

Hardware of the Robotics Vision System

Architecture Diagram: Simple Stereo Vision

Images from Our Stereo Camera

Image Analysis Example I

Image Analysis Example II

Feature Extraction

We can use many of the standard feature extraction techniques in a distributed computing environment, including corner-finding, edges, junctions, t-junctions, and the like.

Connected component, canny edge extraction, deep learning, genetic algorithms are all appropriate feature analysis technologies.

Importing and exporting feature extraction data to and from Cassandra is easy and makes sustainable experimentation possible.

We can migrate from ontology to POJO to schema based design with ease as our software can handle each of these representations.

Technology Stack, Stereo Vision Platform

Data Flow in the Robotics System

Data flow in the robotics system is primarily mediated by Apache Kafka It turns out much of the extraneous ‘image information’ may be ‘thrown

away’ at least temporarily --- for purposes of specific data flows Data flows may be mediated by Hadoop, Spark, or Flink based

components Some machine learning components used in the data pipeline

processing are now technology agnostic : for example, Mahout supports Spark

All of the image processing components have been found to work extremely well and seamlessly with Cassandra

Conclusions, Case Study II

Cassandra was a key component in our robotics software system We were able to run a Cassandra “image database”, “control

database”, and “environmental database” concurrently on three different servers, and coordinate the data effectively for sensor analysis, fusion, and control purposes.

We found we were able to do “rapid prototyping” of robotics software using Cassandra as our primary data store technology

Some Observations about the Technologies

In spite of the fact that the two applications we discussed here were so different, we evolved both of the projects in parallel in order to leverage the common elements of the projects with a mind towards development an ‘image as big data’ toolkit, with which it would be possible to develop a wide range of domain applications, using the idea of ‘images as big data’ --- and the secondary technologies of sensor fusion --- as unifying themes.

Appropriate database technologies are key to the success of the POCs shown here, and Cassandra proved to be the superior choice for a number of compelling reasons.

While the idea of distributed image processing is by no means new --- a variety of implementations on different hardware configurations including Symbolics LISP machines, Connection Machine, and other parallel/multiprocessor hardware were implemented throughout the 1990s

These older distributed concepts have been replaced by Hadoop, Spark, Flink, and multicore/GPU based applications at the low level, and, to a certain extent, by semantic web/ontology/reasoning engines at the high level vision processing phase.

Conclusions and Future Work

We intend to pursue our concept of innovation + software archaeology : best of breed new implementation --- with inspiration from quality software systems of past and present

Applying it to new domain areas and POCs Particularly sensor fusion, ”smart sensors”, drone and mobile robotics

applications We intend to pursue the research aspects of “image as big data”, and

to implement POCs, products, and patents accordingly

Thank You!

kkoitzsch@kildane.com

References and Citations [1] Forbus, Kenneth D., and de Kleer, Johan. Building Problem Solvers,

Cambridge, MA: MIT Press, 1993. [2] Balakumar, P. et al. VANTAGE: A Frame-Based Geometric Modeling System,

Programmer/Users Manual V1.0. Robotics Institute, Carnagie Mellon University, 1988.

[3] Barry, Michele, Cyrluk, David, Kapur, Deepak, Mundy, Joseph, and Nguyen, Van-Duc. A Multi-Level Geometric Reasoning System for Vision. In Geometric Reasoning, Depak Kapur and Joseph Mundy, eds. Cambridge, MA: MIT Press, 1989.

[4] Patent, OCR Enabled Management of Accounts Payable and/or Accounts Receivable Auditing Data, Patent Number 20110213685

[5] Hartley, Richard, and Zisserman, Andrew. Multiple View Geometry in Computer Vision. Cambridge University Press, 2000.

[6] Faugeras, Olivier. Three Dimensional Computer Vision: A Geometric Viewpoint. Cambridge MA: MIT Press, 1993.