visual computing in medicine - vsb.czdap.vsb.cz/cass2017/img/visualcomputingmedicine.pdf ·...
TRANSCRIPT
9/7/17
1
VisualComputinginMedicine
Hans-ChristianHege
Int.SummerSchool2017onDeepLearningandVisualDataAnalysis,Ostrava,07.Sept.2017
Heiko Ramm BrittaWeberDanielBaum
’
Acknowledgements
HansLamecker StefanZachow DagmarKainmüller
9/7/17
2
VisualComputing
VisualData
Image/VideoAnalysis
ComputerVision
ComputerGraphics
ComputerAnimation
VR,AR
DataVisualization
Non-VisualData
DataProcessing
Data Acquisition
Imaging
Image/VideoProcessing
Image-Related Disciplines
9/7/17
3
Visual Computing
Visualcomputing =all computersciencedisciplineshandlingimagesand3Dmodels,i.e.computergraphics,imageprocessing,visualization,computervision,virtualandaugmentedreality,videoprocessing,butalsoincludesaspectsofpatternrecognition,humancomputerinteraction,machinelearninganddigitallibraries.
Corechallengesaretheacquisition,processing,analysisandrenderingofvisualinformation(mainlyimagesandvideo).
Applicationareasincludeindustrialqualitycontrol,medicalimageprocessingandvisualization,surveying,robotics,multimediasystems,virtualheritage,specialeffectsinmoviesandtelevision,andcomputergames.
source:Wikipedia
Images (mathematically)
• Domain:compact;2D,3D;or2D+t,3D+t⇒ video
often:
• Range:greyvalues,colorvalues,“hyperspectral”values
often:
• Practicalcomputing:domainandrangearediscretized• Domain“sampled”(pixels,voxels)• Range“quantized”,e.g.,• Function piecewiseconstantorsmoothinterpolant
Image:
9/7/17
4
Image Examples (I)
Greyvalue images
Conebeamimageindentistry
entiregreyvaluerangedepicted
⇒ poorimage contrast
medium-widthwindow
⇒ goodoverall contrast,coveringsoft-tissueandbone
small-widthwindow
⇒ highcontrastforboneandteeth
04095
025
6
Windowing
window
Image Examples (II)
2D,3D,…
X-rayprojection ElectronTomography(resolution:1,5nm)
9/7/17
5
Image Examples (III)
static
MRIofahead(resolution:<1mm)
dynamic
Real-timeMRIofahumanheart(resolution2mm/50ms)
Image Examples (IV)
Colorimages
RGBimagelayers(„colorchannels“)colorimage
Colorspace:3dimensionalPixelvalues=coordinatesincolorspace
Lightmicroscopy(inanatomicpathology)
9/7/17
6
Image Examples (IV)
Vectorimages
Flowmappingincardiology
Images (IV)
Tensorimages
Diffusiontensors(2Dslicein3D)visualizedbyellipsoids Fibertracks
9/7/17
7
VisualComputinginMedicine
3DExcite-LivingHeart–Pow
erwall(DassaultSystem
es)
Visual Computing in Medicine
Acquisition,processing,analysisandrenderingofallvisualinformation(images/videos,3Dmodels)
thatarisesduringdiagnosis,treatmentandprevention.
Requirestechniquesfrom
• Image/videoprocessing,patternrecognition,computervision,machinelearning
• Computergraphics,visualization,computervision,virtualandaugmentedreality,humancomputerinteraction
9/7/17
8
Data in Medicine
Medicine:science&practiceofthediagnosis,treatmentand preventionofdisease,whetherphysicalormental.
Amedicaltreatmentinvolvesseveralprocesses,wherethepatient'shealthstatusisalwaysthecenterofattention.
• Medicalhistory
• Directlyaccessibleparameters
• Datafromimaging• Datafromlaboratory
HealthStatus
⇒ LotsofData!E.g.incardiologynowadaysperstandardexamination:about2GB
Medicine: Overall Process, Data Processing
InitialState
(ComputerAided)Diagnosis
AnamnesisMeasurementSimulationVisualization
(ComputerAided)Therapy Planning
MeasurementSimulationVisualization
ChangingState
MonitoringSimulationVisualization
(ComputerAided)Treatment Healing
ChangingState
MonitoringVisualization
SimulationVisualizaion
PredictedState
(ComputerAided)Prediction
9/7/17
9
Simulation
Personalized Simulations in Medicine
Indiagnosis: revealinformationthatcannotbemeasured,e.g.,computetheloadingofakneejointfordifferenttypesofmovement,giventheindividualanatomyandthebodyweight.
Intherapy: planandoptimizetreatments/surgeriese.g.,enablesurgeontotrydifferentsurgerytechniquespre-operatively,giventhecurrentanatomicalandphysiologicalstate
Inprevention: makelong-termpredictionsofhealthstate,e.g.,dependingondifferentlifestyles,giventhecurrenthealthstate
9/7/17
10
Personalized Simulations in Medicine
Inchemicalspace mainlyordinarydifferentialequations(ODEs)(systemsbiology) andstochasticdifferentialequations(SDEs)
Inspace&time mainlypartialdifferentialequations(physical) ⇒ FiniteElementmethods
⇒ Anatomicalmodelsrequired
9/7/17
11
AnatomyReconstruction- anexemplarytaskofvisualcomputing
AVision
9/7/17
12
Dr. „Bones“ McCoy Spock
*) RichardA.Robb,BiomedicalImaging,Visualization,andAnalysis,Wiley-Liss,2000
PhysicianonthestarfleetspaceshipEnterprise
Let these guys inspire our imagination… *)
Averycompacthandhelddevice:
1. Pointittothebodyofthepatient;thenthecompleteanatomic,physiological,biochemicalandmetabolicstatus isinstantaneously determinedanddisplayed.
⇒ “tricorder”
2. Placeitonthediseasedorinjuredregion;thencompletecureiseffected.
Dr. McCoy‘s Ultimate Healing Device
9/7/17
13
Wearesatisfiedwith
creationofareconstructed3Dpatientmodel
suitableforplanning,optimizationandcontrolofatherapy
Our Less Ambitious Aim
è
PatientModels
9/7/17
14
PatientModels
anatomystuff.co.uk
population averagepostmortemreal model
Teran,Sifakis,Blemker,…&FedkiwCreatingandsimulatingskeletalmusclefromthevisiblehumandataset.IEEETransVisualComput Graph,2005.
patient-specificpostmortemvirtual model
Zachow,Muigg,Hildebrandt,Doleisch &HegeVisualexplorationofnasalairflow.IEEETransVisualComput Graph,2009.
patient-specificantemortem (!)virtual model
Patient-SpecificModels
anatomical è geometrical
functional è physical / mathematical / numerical(biomechanical, physiological, …)
Model
(this talk)
9/7/17
15
Krebs
©Sob
otta
ExampleApplications:SurgicalReconstruction
Lamecker,Zachow,Hege,Zockler,Haberl:SurgicalTreatmentofCraniosynostosisbasedonaStatistical3D-ShapeModel,CARS2006
Zachow,Lamecker,Elsholtz &Stiller:Reconstructionofmandibulardysplasiausingastatistical3Dshapemodel. InternationalCongressSeries(Vol.1281,pp.1238-1243).Elsevier,2006.
Howmustbonesbeshaped?
fractures
dentures
jointreplacement
Mccullochlaw
.net
ExampleApplications:ImplantDesign/Fitting/IndividualManufacturing
9/7/17
16
Tetrahedralheartmodel
ExampleApplications:FunctionalSimulation
Zhang Y,Bajaj.C.Finiteelementmeshingforcardiacanalysis.ICESTechnicalReport04-26,UniversityofTexas,Austin,2004.
Beatingheart
Anatomical & Functional Models
FunctionalModels
systemsofordinarydifferentialequations
systemsofpartialdifferentialequations
⇒ RequirementsforAnatomicalModels
goodgeometric approximationofshapes
goodnumericalapproximationoffunctions
9/7/17
17
WholebodyMRI
Pyramidalneuronfromthehippocampus,CFM
Bloodvesselsinbrain,MRI angiography (7T)
Golgiapparatusincell,ET
Anatomy Reconstruction: On All Length Scales
Trabecularstructureoftheradiusbonemicro-CT
Reconstruction Pipeline
9/7/17
18
Images• Anatomicalinformation ⇒ anatomicalmodels• Functionalinformation ⇒ functionalmodels
Anatomyreconstruction• Identificationandsegmentationofanatomicalunits• Creationof(discrete)geometricalshaperepresentations
Images ⇒ Models
ImageData
ImageSegmentation
SurfaceReconstruction
VolumetricGridGeneration
ImageFiltering
ImageRegistration
SurfaceImprovement
Anatomy Reconstruction Pipeline
9/7/17
19
Improveresultsoflaterprocessingsteps(edgedetection)• Medianfilter• Anisotropicdiffusion• Nonlocalmeans• Statisticalmethods• Machinelearning
Denoising
Image Denoising with ML
Usesparsecodingcombinedwithdeepnetworkspre-trainedwithadenoising auto-encoder
Feed-forward convolutional neural network to separate the noise from the noisy image
K.Zhang,W.Zuo,Y.Chen,D.Meng,andL.Zhang.BeyondaGaussianDenoiser:ResidualLearningofDeepCNNforImageDenoising.IEEETransImag Proc,26:7(2017),3142 - 3155
BLS-GSM =BayesLeastSquareswithaGaussianScale-Mixture
KSVD =DictionaryLearningviaSVDSSDA =StackedSparseDenoising Auto-encoder
J.Xie,L.Xu,andE.ChenImageDenoising andInpainting withDeepNeuralNetworks.InAdvancesinNeuralInformationProcessingSystems,341–349(2012).
Notyetappliedtomedicalimagedata!
9/7/17
20
Necessarywhen• Multimodalimagingisused• Acquisitionsmadeatdifferenttimes• Imagedorgansaremoving
Image Registration
Overlayof2imagesincheckerboardpattern
unregistered registered
Ingredients:• Setofallowedspatialtransformations:rigid,affine,free,…• Similaritymeasure,basedoncorrespondingfeatures• Optimizationprocedure
Image Registration
9/7/17
21
Software:• Elastix toolbox,http://elastix.isi.uu.nl• FAIR,www.mic.uni-luebeck.de/people/jan-modersitzki• ITK- Segmentation&RegistrationToolkit,https://itk.org
Image RegistrationLiterature:
JModersitzki:NumericalMethodsfor ImageRegistration,OxfordUniversityPress,2004
SHenn,KWitsch:IterativeMultigridRegularizationTechniquesFor Image Matching,SIAMJ.Sci.Comput.,23:4,(2001),1077-1093
SKlein,MStaring,KMurphy,MAViergever,JPWPluim,Elastix:atoolboxforintensity-basedmedicalimageregistrations,IEEETransMedImag 29:1,(2010)196-205
• UnsupervisedDeepLearning• ConvolutionalStackedAuto-Encoder(CSAE)• Training:3Dimagepatches(∼ 104;21x21x21)sampledfrom∼ 107 voxels
• Multilayerencoder networktransfershigh-dim3Dpatchestolow-dimfeaturerepresentations
• Decoder networkrecovers3Dimagepatchesfromthelearnedrepresentationsbyactingasfeedbacktorefineinferencesintheencodernetwork
• Learnedfeaturerepresentationssteerthecorrespondencedetectioninageneral(sophisticated)imageregistrationframework
Image Registration using MLS.Wang,M.Kim,G.Wu,D.ShenScalableHighPerformanceImageRegistrationFrameworkbyUnsupervisedDeepFeatureRepresentationsLearningIn:SKZhou,HGreenspan,DShen,DeepLearningforMedicalImageAnalysis,Elsevier,2017,pp.245-269
Appliedto7T-MRIbrainimages:Consistentlybetterresultsthanstate-of-the-artmethods!
• Manyfurtherstudiesarenecessary(otherimagemodalities,anatomicalregions,…)
• Strategies to dealwith(hyper-)parameters
9/7/17
22
Hardsegmentation
● Classificationofpixels(voxels)● Uniqueassignments
Image Segmentation
Softsegmentation
● Non-uniqueassignments● Probabilisticclassmemberships
Inordertodeterminepixel/voxellabelscorrectly,often
• variousnon-obviousimagepropertiesarenecessary,includingnon-localones
• imageinformationdoesnotsuffice;additionallypreviousknowledgeisnecessary(⇒ Bayesianmethods)
Additionally,objectdefinitionsareoftenpureconventionsandsometimestheydependonthetask.
Problem of Image Segmentation (1)
9/7/17
23
Problemofimagesegmentationsolvedexcellentlybytheevolutionofbiologicalvisualsystems.
⇒ developcomputer-basedvisioninspiredbybiologicalvision
Approaches (twoextremes):
• Constructalgorithms,whichoperatedirectlyonimagefeatures
• Developlearnablealgorithms(e.g.,artificialneuralnetworks),whichimplementprinciplesofbiologicalintelligenc
Problem of Image Segmentation (2)
• Thresholding:global,adaptive(e.g.Otsu‘smethod)
• Clusteringmethods:partitionimageintokclusters(k-means,histogram-basedclustering)
• Edgedetectionmethods:findedges&connectedgesegments
• Regiongrowingmethods:startwithaseedandgrowaccordingtosomesimilaritycriterion
• Watershedmethods: imagegradientmagnitude=topographicsurface;waterplacedatanypixelsflowsdownhill;pixelsdrainingtoacommonminimumformasegment
Image Segmentation Methods
9/7/17
24
• Graphpartitioningmethods:pixelsorgroupsofpixelsareassociatedwithnodesofagraph;edgeweightsdefinethe(dis)similaritybetweentheneighborhoodpixels;partitioningofthegraphaccordingtovariouscriteria:normalizedcuts,randomwalker,...
• PDE-basedmethods:e.g.levelsetmethod:contour=0-levelofscalarfunction;startwithaseedcontourandpropagateituntilitreachestheobjectboundary
• Variational segmentation:energyfunctionals areminimized,e.g.ofMumford-Shahtype
• Model-basedsegmentation:assumptionthatstructuresofinterest/organshavearepetitiveshape;probabilisticmodelexplainingthevariationoftheshape;usethismodelaspriorwhensegmenting
Image Segmentation Methods
• Fast,interactivealgorithms(intelligentscissors,graphcuts,…)
• Utilizehumanvisualsystem
Interactive Image Segmentation
SegmentationEditorinAmira
9/7/17
25
• Utilizationofa-prioriinformation(shapeoforgan)• Trainingdatasets(sufficientlylargenumber)• Statistical3DShapeModels(SSM)• Model-basedsegmentation
Automatic Image Segmentation
• Roughlyplaceshapetemplateintotheimagedata- usingGeneralizedHoughtransform
• Iterativelyadapttheshapemodeltotheimagedata- guidedbygreyvalueprofiles
Automatic Image Segmentation
9/7/17
26
• ‘Intensityprofiles’guidethedeformationprocess:• Ateachvertexofsurfacemesh,commonlyalongsurfacenormals,intensitiesaresampledalonglinesegments
• Oneachprofile,acostfunctionisderivedfromimagedataforanumberofequidistantsamplingpoints
• Minimumcostdetermines(locallyoptimal)newpositionfortherespectivevertex
• Patient-specificcharacteristicsnotcontainedinSSMlimitaccuracyofsegmentations
• Accuracycanbeincreasedbysubsequentfreeformdeformations
Automatic Image Segmentation
Seim H, Kainmueller D, Heller M, Lamecker H, Zachow S, Hege H-C:Automatic Segmentation of the Pelvic Bones from CT Data Based on a Statistical Shape Model.Proc VCBM, pp. 93-100, 2007
Solution: simultaneouslysegmentmultipleadjacentobjects&incorporateknowledgeabouttheirspatialrelationship
Automatic Image Segmentation: Accurate JointSegmentation
Problem:
9/7/17
27
Automatic Image Segmentation: Accurate JointSegmentation
distalfemurproximaltibia
acetabulumproximalfemur
Modelinitializationwithstatisticalshapemodels
Graphcutsoptimization
D. Kainmueller, H. Lamecker, S. Zachow, H-C. Hege.Coupling Deformable Models for Multi-objectSegmentation. ISBMS, LNCS vol. 5104, pp. 69-78.Springer, 2008.
Automatic Image Segmentation Using CNNs
• Hugeprogressduringpastyears;often superiortopreviousstate-of-the-arttechniques
• Bigadvantage:Methodsrequirenofeatureengineering;adaptflexiblytotheproblemthey aretrainedfor
• Fieldmuchtoolarge,tobepresentedhere• Currentmethodsapplicabletomedicalimages
• Dependverystronglyonimagemodality& anatomicalregion• Requireparameterstobetuned• Moregenericmethodsareneeded
Part 3: Medical Image Segmentation, In:Zhou SK, Greenspan H and Shen D (eds.)Deep Learning for Medical Image Analysis,Academic Press, 2017, pp. 177-242
9/7/17
28
E.g.forFE-simulation:surfaceandvolumemesh
• Unstructured
• Multi-material(⇒ generalizedMC)
• Locallyadaptiveresolution
• Controlonelementquality
Rinside / Routside
(Rinside / Routside )ideal
Quality =
Grid Generation
high res
simplified
optimized
Details,ComplexityandMeshQuality
Improvement of Surface Meshes
9/7/17
29
• Consistentremeshing of• Non-manifoldtrianglemeshes• Withuser-definedfeaturelines
• Resultmesh• Withhighregularityandtrianglequality• Preservedgeometry&topologyofthe
• inputmesh• featureskeleton
• Basedonlocaloperationsonly
Improvement of Surface Meshes
Zilske M, Lamecker H, Zachow S:Remeshing of non-manifold surfacesEurographics 2008, pp. 211-214
Improvement of Surface Meshes
9/7/17
30
Improvement of Surface Meshes
Remeshing ofnon-manifoldtriangulations
Rinside / Routside
(Rinside / Routside )ideal
Quality =
Improvement of Surface Meshes
9/7/17
31
Automaticcontrolofelementsizeusinga“sizingfield”
Generation of Volumetric Meshes
Lamecker H, Mansi T, Relan J, Billet F, Sermesant M, Ayache N, Delingette H:Adaptive Tetrahedral Meshing for Personalized Cardiac SimulationsProc. MICCAI Workshop on Cardiovascular Interventional Imaging and Biophysical Modelling (CI2BM),pp. 149-158, 2009.
Generation of Volumetric MeshesMeshgenerationbyadvancingfrontmethodandwithconsiderationoftheelementsize
Consistenthandlingofheterogeneousinputs,includingCADdata:
Kahnt M, Ramm H, Lamecker H, Zachow S:Feature-Preserving, Multi-Material Mesh Generation using Hierarchical Oracles.MICCAI Workshop on Mesh Processing in Medical Image Analysis, LNCS 7599, pp. 101-111, 2012.
9/7/17
32
Generation of Volumetric MeshesSinuses(forsimulationofnasalairflow):
Zachow, S ; P. Muigg ; Th. Hildebrandt ; H. Doleisch ; H.-C. Hege:Visual Analysis of Nasal Airflow. IEEE TVCG, 15:8, pp. 1407-1414 (2009)
FurtherApplications
9/7/17
33
Orthopedicsurgeryplanning:decisionsupportbasedonindividualbiomechanics
Surgery Planning
incooperationwith
Premise:validatednumericalmodel• Kneereplacement• Studywith328automaticallycement-lessimplantedtibiae
• Result:inter-patientvariabilityofbonestrainattheimplant-boneinterface(overafullgaitcycle)
Clinical Research with Virtual Patients
incooperationwith
Galloway F, Seim H, Kahnt M, Nair P, Worsley P, Taylor M:A Large Scale Finite Element Study of an OsseointegratedCementless Tibial Tray; J Bone Joint Surg Br, 2012
9/7/17
34
Biomechanical Simulation
Conclusions
9/7/17
35
• We are on the way to the digital patient
• Problem: Utilize the available data intelligently along the whole medical procedure
• Extract essential information automatically
• Visualize essential information
• Chance: Personalized medicine
• Requires patient-specific simulation
• Requires patient-specific anatomical models
Visual Computing in Medicine
• Two major problem areas in practice:• Image segmentation
• Meshing
• Within past 5 years: more progress than probably most experts expected, especially in the field of image segmentation,due to machine learning
• Current state:• “Everything” can be segmented
• But typically this requires• Design of a specific algorithm
• At least: adaptation of (possibly many) free parameters
• Large training data sets
Creation of Patient-Specific Anatomical Models
9/7/17
36
• Learnable segmentation algorithms:Find systematic ways for construction of neural networks, particularly for segmentation & registration
• Track uncertainties explicitely; deliver error bounds
• Extend anatomical models to functional models(⇒ biophysical quantitative imaging)
• Improve numerical simulation methods
• Develop simulation-based decision support systems
• Simplify, simplify, simplify…
Research Topics
TheQualcommTricorder XPRIZE:
A$10millioncompetitiontobringhealthcaretothepalmofyourhand.
300teamsparticipated,2winners
AwardceremonyonApril12,2017
Seehttps://tricorder.xprize.org
Medical Tricorder:Science Fiction becomes Reality
9/7/17
37
Textbooks
B.Preim,C.BothaVisualComputinginMedicine2nd ed.,MorganKaufman,2014812pp
I.N.Bankman (ed.)HandbookofMedicalImageProcessingAcademicPress,2009984pp
9/7/17
38
Y.J.ZhangGeometricModelingandMeshGenerationfromScannedImagesAcademicPress,2017458pp
S.K.Zhou(ed.)MedicalImageRecognition,SegmentationandParsingAcademicPress2015542pp
S.K.Zhou,H.Greenspan,D.Shen(eds.)DeepLearningforMedicalImageAnalysisAcademicPress,2017458pp
