AI Summit – Sophia Antipolis – 7 Nov 2018
AI & Healthcare: Towards a Digital Twin? Nicholas Ayache Inria, France
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
1
AI & Healthcare • AI allows to augment the cognitive capabilities of the physicians in order to better take care of their patients • For instance AI allows to analyze simultaneously multimodal & multi-dimensional medical images along with other patient data • An information flow too big and too complex to be optimally processed by a human brain.
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
2
Feb. 2017
Dermatology
Deep neural networks Classification of lesions • benign vs. cancer • Pre-training: 1.3 Million natural images • Fine Tuning • ~130.000 lesions covering 2000 pathologies • performances = confirmed dermatologists https://cs.stanford.edu/people/esteva/nature/ • •
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
3
Radiology
June 2017
Final labels: 0. healthy 1. calc benign 2. mass benign 3. calc malignant 4. mass malignant
0. healthy
1. cancer
Courtesy of Therapixel
The Digital Mammography DREAM Challenge 1 Training : ~640.000 mammographies Therapixel Inria spinoff
1.200+ participants
AI & Digital Twins for e-Medicine
N. Ayache 7 Nov 2018
4
Ophthalmology Feb. 2018
•
Training: ~300 000 patients
Déc. 2016
•
N. Ayache 7 Nov 2018
Training: ~130 000 patients
AI & Digital Twins for e-Medicine
IDx-DR FDA Approved April 2018
5
ONCOLOGY: Radiomics to Predict Immunotherapy Response Lancet Oncol 2018, August 14, http://dx.doi.org/10.1016/ A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study Institut Gustave Roussy, CentraleSupĂŠlec, Therapanacea
Hundreds of patients
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
6
AI & Digital Patient • AI is not limited to specific (and often narrow) classification tasks trained on (often huge) specific databases • By combining AI with biophysical models of life, one can build a digital twin of the patient and develop computational tools to assist diagnosis, prognosis and therapy: digital medicine • N. Ayache: L’imagerie médicale à l’heure de l’intelligence artificielle, Santé et IA, CNRS Editions, 2018 • Medical Imaging & Machine Learning: towards an artificial intelligence? Collège de France, May 2018 www.college-de-france.fr
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
7
The Personalized Digital Patient
 (Digital Twin)
Digital Medicine prognosis
clinical, biological, omics, behavior, lifestyle, ‌
Geometry Statistics Semantics Biology Physics Chemestry
Computational Models of Human Body Physiology personnalized
Multi-scale
Images + Measures
Anatomy
Learning
evolution
in silico
in vivo
diagnosis
planification simulation control therapy
AyacheTowards a Personalized Computational N N. Ayache. Patient. IMIA Yearbook of Medical Informatics, 2016 8 AI & Digital Twins for e-Medicine 7 Nov 2018
Illustrations from recent research of our Inria team EPIONE with academic, clinical & industrial partners
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
9
Digital Patient / Twin Computational Physiology Computational Anatomy
Chemestry
Physics Biology Semantics
Statistics
Algorithms
Geometry N Ayache. Towards a Personalized Computational Patient. IMIA Yearbook of Medical Informatics, 2016 N. Ayache AI & Digital Twins for e-Medicine 10 7 Nov 2018
1. Cochlear Implants Statistical Analysis ~1000 subjects
Inria IUFC Oticon
Geometrical Model
Bayesian Segmentation
Electrode Insertion
T. Demarcy, C. Vandersteen, N. Guevara, C. Raaelli, D. Gnansia, N. Ayache, H. Delingette, Automated analysis of human cochlea shape variability from seg. Ο CT images Comp. Medical Imaging & Graphics 2017
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
11
Mauna Kea Technologies
2. Endomicroscopy
• André, Vercauteren, Wallace, Buchner, Ayache. IEEE TMI 2012 • M Kohandani Tafreshi et al., Digestive Disease Week, DDW 2014
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
Colon
Smart Atlas
Database indexed by structural invariants
New image
~4500 Images with expertises
Pathology: Purely Benign 12 Semantic features: Round crypts, Medium lumen, Normal Goblet Cells
3. Multiple Sclerosis MRI
PET
predicted
Demyelination true
MRI Prediction
PET Biomarker
Training: 28 subjects
Generative Adversarial Networks
(millions of voxels)
W. Wei, E Poirion, B Bodini, S Durrleman, N Ayache, B Stankoff, O Colliot. Learning Myelin Content in Multiple Sclerosis from Multimodal MRI through Adversarial Training. MICCAI 2018
N. Ayache 7 Nov 2018
13
AI & Digital Twins for e-Medicine INSTITUTION DU CERVEAU ET DE LA MOELLE EPINIERE ICM_09_2742_LogoAn_Quad 26/08/2009 24, rue Salomon de Rothschild - 92288 Suresnes - FRANCE Tél. : +33 (0)1 57 32 87 00 / Fax : +33 (0)1 57 32 87 87 Web : www.carrenoir.com
Ce fichie r est un documen t d’exécutio n créé sur Illustrator version 10.
ÉQUIVALENCES QUADRI MAGENTA 70% JAUNE 100%
CYAN 100% MAGENTA 70% NOIR 60%
4. Brain Tumors MRI
• • •
T2
Flair
T1
T1Gd
BRATS 2017 285 patients Full or Weak supervision
segmentation
Edema Proliferating rim Necrotic core
P. Mlynarski, H Delingette, A Criminisi, and N Ayache. 3D CNNs for Tumor Segmentation using Long-Range 2D Context. ArXiv 2018 + Submitted.
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
14
Mixed Supervision segmentation Large Number of WEAK Annotations
30 FA
Dice index
Small Number of FULL Annotations
15 FA
Number of Fully Annotated images
5 FA
Number of Weakly Annotated Images P. Mlynarski, H Delingette, A Criminisi, and N Ayache. Mixed-supervised learning for Tumor Segmentation. Submitted+ ArXiV2018
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
15
Biophysics Standard Dosimetry
0. Clatz
Mass. General Hospital, Boston
Migration + proliferation
∂c = ∇.(D∇c ) + ρc(1 − c ) ∂t
tumoral cell density model
GTV : 60 Gy - T1Gd + 2cm CTV 46 Gy - T2Flair + 1.5cm Optimized Dosimetry
Fisher-Kolmogorov radiotherapy model authorized maximum dose
• M Lê, H Delingette, …, J Unkelbach, N Ayache. IEEE Tr. on Medical Imaging 2016
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
16
Thermoablation
5. Liver Tumors
Biophysical model : blood flow + heat transfer + local necrosis
Temperature
C Audigier, T Mansi, H Delingette, Michael Choti, Ali Kamen, L. Soler… D Comaniciu, N Ayache N. Ayache AI Surgery & Digital Twins for e-Medicine International J. of Computer Assisted Radiology and 2016 7 Nov 2018
Siemens Princeton Johns Hopkins IHU Strasbourg Inria
PDEs: Navier-Stokes Porous media, reaction–diffusion- advection
Local Necrosis
17
6. Cardiology Geometry of ventricles Deep Learning
UK Biobank : 4000 subjects •
Thanks to : • •
S Petersen D. Rueckert et al.
Q. Zheng, H Delingette, N Duchateau, N Ayache. 3D Consistent & Robust Segmentation of Cardiac Images by Deep Learning with Spatial Propagation. IEEE Transactions on Medical Imaging, 2018.
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
18
Learning Shape + Motion
95% accuracy on ACDC database (100 subjects)
Image-derived shape and motion phenotypes
Abnormal Right Ventricle Hypertrophic Cardiomyopathies Dilated Cardiomyopathies Myocardium Infarct Normals Q. Zheng, H Delingette, N Ayache. Explainable Cardiac Pathology Classification w. Motion Characteriz. by Semi-Supervised Learning of Apparent Flow, submitted, 2018.
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
19
Learn Deformation •  •  • 
Register ED/ES images (dieomorphic warping) Conditional Variational Auto Encoder 350 subjects
End-Diastole
M
End-Systole
F
z 144 10 2
�
Normals Hypertrophic Cardiomyopathies Dilated Cardiomyopathies Myocardium Infarct Abnormal Right Ventricle
J Krebs, T Mansi, B MailhÊ, N Ayache, H Delingette. Unsupervised Probabilistic Deformation Modeling for Robust Dieomorphic Registration . ArXiv 2018. DLMIA@MICCAI’2018
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
20
Anatomical Model Coronaries
Phrenic Nerve
Coronary Sinus
Scar Dense Scar
M. Sermesant N. Ayache 7 Nov 2018
2017 AI & Digital Twins for e-Medicine
21
ATP
Biophysical model mechanical
electrical sarcomeres
fibers
organ 8 zu2 (1 u) > > div(dM S Mru) + > ⌧ in < @t u = 8 < (1 z) > ⌧ open if u < ugate > @z = > z : t : if u > ugate ⌧ close
u ⌧ out + Jstim (t)
Chen,N. Cabrera, Relan, ..Delingette, Ayache, Sermesant, of Cardiovascular Electrophysiology, 2016. Ayache AI & Razavi: DigitalJ. Twins for e-Medicine Marchesseau, Delingette, Sermesant, Ayache. Biomechanics and Modeling in Mechanobiology, 2012 7 Nov 2018
Kc u εc σc
stiffness action potential deformation stress
22
Personalized Model by optimal filtering
Quantify contractility, stiďŹ&#x20AC;ness, relaxation ratio , contraction ratio, viscosity, compressibility, peripheral resistance, etc.
Predict
before dP/dt
after
Heart Failure S Marchesseau, H Delingette, M Sermesantâ&#x20AC;Ś, A F. Frangi, R Razavi, D Chapelle, N Ayache. Personalization of a Cardiac Electromechanical Model using Reduced Order Unscented Kalman Filtering from Regional Volumes. Medical Image Analysis, 2013
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
- 23
120 Personalized Hearts by statistical prior + multi-fidelity optimisation (0D â&#x20AC;&#x201C; 3D mechanical models)
R MollĂŠro, X Pennec, H Delingette, A Garny, N Ayache, M Sermesant. Multifidelity-CMA for eďŹ&#x192;cient personalisation of 3D cardiac electromechanical models. Biomechanics and Modeling in Mechanobiology, 2017. N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
24
New sequence
Smart Atlas Image manifold
Tetralogy of Fallot Ischemia appearance shape motion biophysics
Healthy Low function (EF) Inferior infarction Slight LV dilation Good candidate for CRT
Asynchrony
N. Ayache 7 Nov 2018
Margeta et al. 2015 Le Folgoc et al .2017
AI & Digital Twins for e-Medicine
25
Simulate new data normal
Simulated infarct
N Duchateau, M. Sermesant, H Delingette, N Ayache. IEEE Tr. on Medical Imaging, 2018. for e-Medicine N. Ayache AI & Digital Twins 7 Nov 2018
26
Learn Infarcts ground truth
septum 25
0
25
0.6
-25
0
25
0
25
-25
0
25
0.4
0.2
AUC: 0.995 (0.978/0.997) 0
0.2
0.4
0.6
0.8
0.6
0.4
0.2
0
1
0
0.6
0.4
0.2
0
AUC: 0.827 (0.763/0.871) p-value < 0.001 0
0.2
0.4
0.6
0.8
False Positive Rate = 1 - specificity
0
0.4
0.2
AUC: 0.865 (0.819/0.905) AUC: 0.909 (0.866/0.928) p-value < 0.001 0
0
0.4
0.2 0.6
0.4 0.8
0.6
1
0.8
1
0.6
0.4
0.2
0
0
0.2
0
0.8
0.5
0.4
27
AUC: 0.696 (0.584/0.802) AUC: 0.742 (0.687/0.779) p-value < 0.001 p-value < 0.001 0
0.4
0.2 0.6
0.4 0.8
0.6
1
0.8
False Positive RatePositive = 1 - specificity False Rate = 1 - specificity
0.2
AUC: 0. 0
0.2
0.4
Thresho (longitudina
1
0.6
0
1
0.4
False Positive Rate
1
0.2
0.6
0
1
1
0.8
0.8
0.5
Thresholding Thresholding (longitudinal strain) (circumferential strain)
1
0.8
AI & Digital Twins for e-Medicine
0.2
0.6
Thresho (radial st
1
0.8
False Positive RatePositive =01 - specificity 0.5 = 1 - specificity 1 False Rate
Thresholding (circumferential strain)
1
200 Real images
1
0.8
False Positive Rate = 1 - specificity 0 25
-25
True Positive Rate = sensitivity
1
True Positive Rate = sensitivity
-25
0
Thresholding Regression (radial strain)
True Positive Rate = sensitivity
25
True Positive Rate = sensitivity
0
True Positive Rate = sensitivity
1
N Duchateau, M. Sermesant, H IEEE Tr. on 25 Medical Imaging, 2018. -25 0 25 Delingette, -25 0 N Ayache. 25 -25 0 N Duchateau, P Allain, E Saloux, M Sermesant IEEE Tr. on Med Imaging, 2016
N. Ayache 7 Nov 2018
-25
True Positive Rate = sensitivity
0
True Positive Rate = sensitivity
25
-25
uncertainty
0
0.8
0
0
25
septum
predicted infarct 0.5
500 Synthetic meshes
test case #30
test case #30
0
uncertainty
1
Regression
-25
• •
0
predicted infarct
1
True Positive Rate = sensitivity
Long. strain -25
-25
uncertainty
septum
LAD territory
ground truth infarct
0.5
uncer
septum
test case #50
septum
test case #72
Circular strain long. strain (%)
circ. strain (%)
test case #72
test case #50
radial strain (%)
predicted
ground truth infarct b)strain Real cases long. (%) infarct
1
test case #30
simulated
ground Radial truth long. strain (%) infarct a) Synthetic cases radial strain (%) strain circ. strain (%)
test case #72
test case #72
test case #50
normal
circ. strain (%)
predicted infarct
radial strain (%) (%) long. strain (%) Regression incirc. a strain reduced deformation space infarct
test case #50
variable localisation, size, severity radial strain (%)
Infarct detection & localisation
•
15 normals 450 simulated infarcts :
test case #30
• •
0.8
0.6
0.4
0.2
AUC: 0. 0
0
0.2
0.4
False Positive Rate
Learn Electrophysiology •
personalization Simulated EP Simul. dipole
• Predict abnormal EP • Random Forests: texture + simulation EP intensity
LAVAs: Local Abnormal Ventricular Activation
texture
•
Sparse Bayesian Regression Simulated EP
Onset location + conduction velocity
Simul. dipole
Ablation targets
S Giffard-Roisin, Jackson, Fovargue, Lee, Delingette, Razavi, Ayache, Sermesant, IEEE TBME 2017 + 2018 R Cabrera-Lozoya, B Berte, H Cochet, P Jaïs, N Ayache, M Sermesant. IEEE TBME 2016 + 2018
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
28
Simulate Brain Atrophy simul@trophy: open source simulator
Prescribed atrophy in 34 regions
Biophysical model (creeping flow)
parenchyma
CSF
µΔu − ∇p = (µ + λ )∇a
µΔu − ∇p = 0
∇ ⋅ u = −a.
∇ ⋅ u + p = 0.
a : prescribed atrophy; u,p : displacement & virtual pressure
Validation, training B. Khanal, M. Lorenzi, N. Ayache, X. Pennec. NeuroImage 2016
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
29
Future Trends • Exploiting « holistic »data • Images & signals • Biology, genetics, metabolomics, transcriptomics, etc. • Behavior, lifestyle, etc. Databases: ADNI, Enigma, UK Biobank, Insight, EDS, etc. N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
30
Neuropsychiatry Better monitor patients with neuropsychiatric diseases
Images
Idex UCAJedi Inria, UNS, CHU, IPMC, CoBTeK
Data Science Biology/genomics
Movement/activity
L. Antelmi, N. Ayache, P. Robert, M Lorenzi. Multi-Channel Stochastic Variational Inference for the Joint Analysis of Heterogeneous Biomedical Data in Alzheimer's Disease. Machine Learning in Clinical Neuroimaging (@MICCAI 2018). C. Abi-Nader, N. Ayache, P. Robert, M Lorenzi. Alzheimer's Disease Modeling and Staging through Independent Gaussian Process Analysis of Spatio-Temporal Brain Changes. Machine Learning in Clinical Neuroimaging (@MICCAI 2018).
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
31
Imaging Genetics Alzheimer’s Disease
UCL
Genotype Phenotype
600 patients
Covariance Analysis (PLS) : MRI Phenotypes :~105 Grey matter atrophy
Genotypes SNPs: ~106
single nucleotide polymorphisms
Hippocampi, Amygdales Cingular, enthorinal, temporal cortex TRIB3 + APOE, TOMM40,… : Neurodegenerative processes Alzheimer’s disease
M Lorenzi,…,PM of brain atrophy to TRIB3 in Alzheimer's disease, evidence from functional N. Ayache Thompson, S Ourselin. Susceptibility AI & Digital Twins for e-Medicine 32 prioritization imaging genetics. PNAS 2018. 7 Nov in 2018
small n, large p
Lung Cancer Radiomics: variable selection
•
87 patients with a lung cancer
•
(NSCLC)
15/635 selected features for adenocarcinoma lesions
•
635 radiomic features based on CT images
14/635 selected features for squamous cell carcinoma lesions
•
11/635 selected features for other subtype
Prediction of cancer subtype
F Orlhac, P-A Mattei, C Bouveyron, and N Ayache. Class-specific variable selection in High-Dimensional Discriminant Analysis (HDDA) through Bayesian Sparsity. Journal of Chemometrics, 2018. N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
33
Breast Cancer: Radio - Metabolomics METABOLOMIC ANALYSIS
Resected lesions
Spectrum obtained with a mass spectrometer
Identification of triple-negative breast lesions Expression of 1500 metabolites
43 radiomic features
26 patients with a breast cancer RADIOMIC ANALYSIS
Conventional indices
Pre-treatment PET/CT
Metabolomic data
Radiomic data
Tumor lesion
F Orlhac, O Humbert, T Pourcher, L Jing, J-M Guigonis, J Darcourt, N Ayache, C Bouveyron. Statistical analysis of PET radiomic features and metabolomic data: prediction of triple-negative breast cancer. J Nucl Med,2018. N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
34
AI & Digital Twins New Computational Tools for Digital Medicine more personalized precise predictive preventive
To better heal the real patient N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
35
AI & Digital Twins Digital Medicine is developed to assist the physician, not to replace his/her: comprehension (global) compassion Natural Intelligence critical analysis curiosity consciousness (professional) N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
36
Valeria Manera
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
37
Thanks Inria Team : Epione team & partners • Academic • Clinical • Industrial Medical Image Analysis Journal 20th anniversary special issue (free access)
team.inria.fr/Epione/
N. Ayache 7 Nov 2018
www.college-de-france.fr
AI & Digital Twins for e-Medicine
38
H. Delingette X. Pennec M. Sermesant M. Lorenzi C. Bouveyron N. Duchateau H. Lombaert O. Colliot D Chapelle A Garny F Orlhac
W. Wei P. Mlynarski M. Le C Audigier Q. Zheng J Krebs S Marchesseau R. Mollero J Margeta L Le Folgoc S. Giffard R. Cabrera B Khanal L Antelmi C Abi-Nader
Thanks P. Robert A. Criminisi B. Stankov T. Mansi, D. Comaniciu N. Guevara, T. Demarcy, C. Raffaelli, J Mahé, M Kohandani P. Hofman F Lacombe, S Loiseau D. Fontaine T. Demarcy, D. Gnasia L. Soler M. Auffret, F/J Brag M Haissaguerre et al. P Jais, H. Cochet R Razavi J. Darcourt O. Humbert T. Pourcher D. Rueckert, S. Petersen
Medical Image Analysis Journal 20th anniversary special issue (free access)
team.inria.fr/Epione/
N. Ayache 7 Nov 2018
www.college-de-france.fr
AI & Digital Twins for e-Medicine
39
Marco Lorenzi Maxime Sermesant HervĂŠ Delingette
Xavier Pennec
N. Ayache 7 Nov 2018
Epione 2018 AI & Digital Twins for e-Medicine
Charles Bouveyron
40
Thanks Inria Team : Epione team & partners • Academic • Clinical • Industrial Medical Image Analysis Journal 20th anniversary special issue (free access)
team.inria.fr/Epione/
N. Ayache 7 Nov 2018
www.college-de-france.fr
AI & Digital Twins for e-Medicine
41
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
42
Medical Image Analysis Journal 20th Anniversary special issue 2016 free access • •
Editorial by N. Ayache & J. Duncan 32 visionary & historical papers with clinical & industrial transfer by: S. Wells, M. Brady, D. Rueckert, D. Comaniciu, A. Frangi, S. Ourselin, W. Niessen, J. Schnabel, A. Noble, A. Young, J. Weese, D. Hawkes,T. Peters, N. Navab, G. Szekely, L. Joskowicz, A. Criminisi, M. de Bruijne, D. Metaxas, N. Paragios, G. Gerig, R. Deriche, JF. Mangin, C. Barillot, M. Viergever, C. Davatzikos, P. Golland, K. Mori, M. Sonka, A. Madabhushi, R. Kikinis, G. Fichtinger
https://www.sciencedirect.com/journal/medical-image-analysis/vol/33/suppl/C
N. Ayache 7 Nov 2018
AI & Digital Twins for e-Medicine
43