Nicholas AYACHE — Inria, Université Côte d'Azur — AI & Healthcare: Towards a Digitak Twin?

AI Summit – Sophia Antipolis – 7 Nov 2018

AI & Healthcare: Towards a Digital Twin?

Nicholas Ayache Inria, France

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

1

AI & Healthcare • AI allows to augment the cognitive capabilities of the physicians in order to better take care of their patients • For instance AI allows to analyze simultaneously multimodal & multi-dimensional medical images along with other patient data • An information flow too big and too complex to be optimally processed by a human brain.

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

2

Feb. 2017

Dermatology

Deep neural networks Classification of lesions • benign vs. cancer • Pre-training: 1.3 Million natural images • Fine Tuning • ~130.000 lesions covering 2000 pathologies • performances = confirmed dermatologists

https://cs.stanford.edu/people/esteva/nature/ • •

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

3

Radiology

June 2017

Final labels: 0. healthy

1. calc benign

2. mass benign

3. calc malignant

4. mass malignant

0. healthy

1. cancer

Courtesy of Therapixel

The Digital Mammography DREAM Challenge 1 Training : ~640.000 mammographies Therapixel Inria spinoff

1.200+ participants

AI & Digital Twins for e-Medicine

N. Ayache 7 Nov 2018

4

Ophthalmology Feb. 2018

•

Training: ~300 000 patients

Déc. 2016

•

N. Ayache 7 Nov 2018

Training: ~130 000 patients

AI & Digital Twins for e-Medicine

IDx-DR FDA Approved April 2018

5

ONCOLOGY: Radiomics to Predict Immunotherapy Response Lancet Oncol 2018, August 14, http://dx.doi.org/10.1016/

A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study

Institut Gustave Roussy, CentraleSupĂŠlec, Therapanacea

Hundreds of patients

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

6

AI & Digital Patient • AI is not limited to specific (and often narrow) classification tasks trained on (often huge) specific databases • By combining AI with biophysical models of life, one can build a digital twin of the patient and develop computational tools to assist diagnosis, prognosis and therapy: digital medicine • N. Ayache: L’imagerie médicale à l’heure de l’intelligence artificielle, Santé et IA, CNRS Editions, 2018 • Medical Imaging & Machine Learning: towards an artificial intelligence? Collège de France, May 2018 www.college-de-france.fr

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

7

The Personalized Digital Patient
 (Digital Twin)

Digital Medicine prognosis

clinical, biological, omics, behavior, lifestyle, ‌

Geometry Statistics Semantics Biology Physics Chemestry

Computational Models of Human Body

Physiology personnalized

Multi-scale

Images + Measures

Anatomy

Learning

evolution

in silico

in vivo

diagnosis

planification simulation control therapy

AyacheTowards a Personalized Computational N N. Ayache.

Patient.

IMIA Yearbook of Medical Informatics, 2016 8 AI & Digital Twins for e-Medicine 7 Nov 2018

Illustrations from recent research of our Inria team EPIONE with academic, clinical & industrial partners

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

9

Digital Patient / Twin Computational Physiology Computational Anatomy

Chemestry

Physics Biology Semantics

Statistics

Algorithms

Geometry N Ayache. Towards a Personalized Computational Patient. IMIA Yearbook of Medical Informatics, 2016 N. Ayache AI & Digital Twins for e-Medicine 10 7 Nov 2018

1. Cochlear Implants Statistical Analysis ~1000 subjects

Inria IUFC Oticon

Geometrical Model

Bayesian Segmentation

Electrode Insertion

T. Demarcy, C. Vandersteen, N. Guevara, C. Raaelli, D. Gnansia, N. Ayache, H. Delingette, Automated analysis of human cochlea shape variability from seg. Ο CT images Comp. Medical Imaging & Graphics 2017

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

11

Mauna Kea Technologies

2. Endomicroscopy

• André, Vercauteren, Wallace, Buchner, Ayache. IEEE TMI 2012 • M Kohandani Tafreshi et al., Digestive Disease Week, DDW 2014

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

Colon

Smart Atlas

Database indexed by structural invariants

New image

~4500 Images with expertises

Pathology: Purely Benign 12 Semantic features: Round crypts, Medium lumen, Normal Goblet Cells

3. Multiple Sclerosis MRI

PET

predicted

Demyelination true

MRI Prediction

PET Biomarker

Training: 28 subjects

Generative Adversarial Networks

(millions of voxels)

W. Wei, E Poirion, B Bodini, S Durrleman, N Ayache, B Stankoff, O Colliot. Learning Myelin Content in Multiple Sclerosis from Multimodal MRI through Adversarial Training. MICCAI 2018

N. Ayache 7 Nov 2018

13

AI & Digital Twins for e-Medicine INSTITUTION DU CERVEAU ET DE LA MOELLE EPINIERE ICM_09_2742_LogoAn_Quad 26/08/2009 24, rue Salomon de Rothschild - 92288 Suresnes - FRANCE Tél. : +33 (0)1 57 32 87 00 / Fax : +33 (0)1 57 32 87 87 Web : www.carrenoir.com

Ce fichie r est un documen t d’exécutio n créé sur Illustrator version 10.

ÉQUIVALENCES QUADRI MAGENTA 70% JAUNE 100%

CYAN 100% MAGENTA 70% NOIR 60%

4. Brain Tumors MRI

• • •

T2

Flair

T1

T1Gd

BRATS 2017 285 patients Full or Weak supervision

segmentation

Edema Proliferating rim Necrotic core

P. Mlynarski, H Delingette, A Criminisi, and N Ayache. 3D CNNs for Tumor Segmentation using Long-Range 2D Context.

ArXiv 2018 + Submitted.

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

14

Mixed Supervision segmentation Large Number of WEAK Annotations

30 FA

Dice index

Small Number of FULL Annotations

15 FA

Number of Fully Annotated images

5 FA

Number of Weakly Annotated Images P. Mlynarski, H Delingette, A Criminisi, and N Ayache. Mixed-supervised learning for Tumor Segmentation. Submitted+ ArXiV2018

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

15

Biophysics Standard Dosimetry

0. Clatz

Mass. General Hospital, Boston

Migration + proliferation

∂c = ∇.(D∇c ) + ρc(1 − c ) ∂t

tumoral cell density model

GTV : 60 Gy - T1Gd + 2cm CTV 46 Gy - T2Flair + 1.5cm Optimized Dosimetry

Fisher-Kolmogorov radiotherapy model

authorized maximum dose

• M Lê, H Delingette, …, J Unkelbach, N Ayache. IEEE Tr. on Medical Imaging 2016

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

16

Thermoablation

5. Liver Tumors

Biophysical model : blood flow + heat transfer + local necrosis

Temperature

C Audigier, T Mansi, H Delingette, Michael Choti, Ali Kamen, L. Soler… D Comaniciu, N Ayache N. Ayache AI Surgery & Digital Twins for e-Medicine International J. of Computer Assisted Radiology and 2016 7 Nov 2018

Siemens Princeton Johns Hopkins IHU Strasbourg Inria

PDEs: Navier-Stokes Porous media, reaction–diffusion- advection

Local Necrosis

17

6. Cardiology Geometry of ventricles Deep Learning

UK Biobank : 4000 subjects •

Thanks to : • •

S Petersen D. Rueckert et al.

Q. Zheng, H Delingette, N Duchateau, N Ayache. 3D Consistent & Robust Segmentation of Cardiac Images by Deep Learning with Spatial Propagation. IEEE Transactions on Medical Imaging, 2018.

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

18

Learning Shape + Motion

95% accuracy on ACDC database (100 subjects)

Image-derived shape and motion phenotypes

Abnormal Right Ventricle

Hypertrophic Cardiomyopathies

Dilated Cardiomyopathies

Myocardium Infarct

Normals Q. Zheng, H Delingette, N Ayache. Explainable Cardiac Pathology Classification w. Motion Characteriz. by Semi-Supervised Learning of Apparent Flow, submitted, 2018.

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

19

Learn Deformation •  •  • 

Register ED/ES images (dieomorphic warping) Conditional Variational Auto Encoder 350 subjects

End-Diastole

M

End-Systole

F

z 144 10 2

�

Normals Hypertrophic Cardiomyopathies Dilated Cardiomyopathies Myocardium Infarct Abnormal Right Ventricle

J Krebs, T Mansi, B MailhÊ, N Ayache, H Delingette. Unsupervised Probabilistic Deformation Modeling for Robust Dieomorphic Registration .

ArXiv 2018. DLMIA@MICCAI’2018

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

20

Anatomical Model Coronaries

Phrenic Nerve

Coronary Sinus

Scar Dense Scar

M. Sermesant N. Ayache 7 Nov 2018

2017 AI & Digital Twins for e-Medicine

21

ATP

Biophysical model mechanical

electrical sarcomeres

fibers

organ 8 zu2 (1 u) > > div(dM S Mru) + > ⌧ in < @t u = 8 < (1 z) > ⌧ open if u < ugate > @z = > z : t : if u > ugate ⌧ close

u ⌧ out + Jstim (t)

Chen,N. Cabrera, Relan, ..Delingette, Ayache, Sermesant, of Cardiovascular Electrophysiology, 2016.

Ayache AI & Razavi: DigitalJ. Twins for e-Medicine Marchesseau, Delingette, Sermesant, Ayache. Biomechanics and Modeling in Mechanobiology, 2012 7 Nov 2018

Kc u εc σc

stiffness action potential deformation stress

22

Personalized Model by optimal filtering

Quantify contractility, stiďŹ&#x20AC;ness, relaxation ratio , contraction ratio, viscosity, compressibility, peripheral resistance, etc.

Predict

before dP/dt

after

Heart Failure S Marchesseau, H Delingette, M Sermesantâ&#x20AC;Ś, A F. Frangi, R Razavi, D Chapelle, N Ayache. Personalization of a Cardiac Electromechanical Model using Reduced Order Unscented Kalman Filtering from Regional Volumes. Medical Image Analysis, 2013

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

- 23

120 Personalized Hearts by statistical prior + multi-fidelity optimisation (0D â&#x20AC;&#x201C; 3D mechanical models)

R MollĂŠro, X Pennec, H Delingette, A Garny, N Ayache, M Sermesant. Multifidelity-CMA for eďŹ&#x192;cient personalisation of 3D cardiac electromechanical models. Biomechanics and Modeling in Mechanobiology, 2017.

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

24

New sequence

Smart Atlas Image manifold

Tetralogy of Fallot Ischemia appearance shape motion biophysics

Healthy Low function (EF) Inferior infarction Slight LV dilation Good candidate for CRT

Asynchrony

N. Ayache 7 Nov 2018

Margeta et al. 2015 Le Folgoc et al .2017

AI & Digital Twins for e-Medicine

25

Simulate new data normal

Simulated infarct

N Duchateau, M. Sermesant, H Delingette, N Ayache. IEEE Tr. on Medical Imaging, 2018. for e-Medicine N. Ayache AI & Digital Twins 7 Nov 2018

26

Learn Infarcts ground truth

septum 25

0

25

0.6

-25

0

25

0

25

-25

0

25

0.4

0.2

AUC: 0.995 (0.978/0.997) 0

0.2

0.4

0.6

0.8

0.6

0.4

0.2

0

1

0

0.6

0.4

0.2

0

AUC: 0.827 (0.763/0.871) p-value < 0.001 0

0.2

0.4

0.6

0.8

False Positive Rate = 1 - specificity

0

0.4

0.2

AUC: 0.865 (0.819/0.905) AUC: 0.909 (0.866/0.928) p-value < 0.001 0

0

0.4

0.2 0.6

0.4 0.8

0.6

1

0.8

1

0.6

0.4

0.2

0

0

0.2

0

0.8

0.5

0.4

27

AUC: 0.696 (0.584/0.802) AUC: 0.742 (0.687/0.779) p-value < 0.001 p-value < 0.001 0

0.4

0.2 0.6

0.4 0.8

0.6

1

0.8

False Positive RatePositive = 1 - specificity False Rate = 1 - specificity

0.2

AUC: 0.

0

0.2

0.4

Thresho (longitudina

1

0.6

0

1

0.4

False Positive Rate

1

0.2

0.6

0

1

1

0.8

0.8

0.5

Thresholding Thresholding (longitudinal strain) (circumferential strain)

1

0.8

AI & Digital Twins for e-Medicine

0.2

0.6

Thresho (radial st

1

0.8

False Positive RatePositive =01 - specificity 0.5 = 1 - specificity 1 False Rate

Thresholding (circumferential strain)

1

200 Real images

1

0.8

False Positive Rate = 1 - specificity 0 25

-25

True Positive Rate = sensitivity

1

True Positive Rate = sensitivity

-25

0

Thresholding Regression (radial strain)

True Positive Rate = sensitivity

25

True Positive Rate = sensitivity

0

True Positive Rate = sensitivity

1

N Duchateau, M. Sermesant, H IEEE Tr. on 25 Medical Imaging, 2018.

-25 0 25 Delingette, -25 0 N Ayache.

25 -25 0 N Duchateau, P Allain, E Saloux, M Sermesant IEEE Tr. on Med Imaging, 2016

N. Ayache 7 Nov 2018

-25

True Positive Rate = sensitivity

0

True Positive Rate = sensitivity

25

-25

uncertainty

0

0.8

0

0

25

septum

predicted infarct 0.5

500 Synthetic meshes

test case #30

test case #30

0

uncertainty

1

Regression

-25

• •

0

predicted infarct

1

True Positive Rate = sensitivity

Long. strain -25

-25

uncertainty

septum

LAD territory

ground truth infarct

0.5

uncer

septum

test case #50

septum

test case #72

Circular strain long. strain (%)

circ. strain (%)

test case #72

test case #50

radial strain (%)

predicted

ground truth infarct b)strain Real cases long. (%) infarct

1

test case #30

simulated

ground Radial truth long. strain (%) infarct a) Synthetic cases radial strain (%) strain circ. strain (%)

test case #72

test case #72

test case #50

normal

circ. strain (%)

predicted infarct

radial strain (%) (%) long. strain (%) Regression incirc. a strain reduced deformation space infarct

test case #50

variable localisation, size, severity radial strain (%)

Infarct detection & localisation

•

15 normals 450 simulated infarcts :

test case #30

• •

0.8

0.6

0.4

0.2

AUC: 0.

0

0

0.2

0.4

False Positive Rate

Learn Electrophysiology •

personalization Simulated EP Simul. dipole

• Predict abnormal EP • Random Forests: texture + simulation EP intensity

LAVAs: Local Abnormal Ventricular Activation

texture

•

Sparse Bayesian Regression Simulated EP

Onset location + conduction velocity

Simul. dipole

Ablation targets

S Giffard-Roisin, Jackson, Fovargue, Lee, Delingette, Razavi, Ayache, Sermesant, IEEE TBME 2017 + 2018 R Cabrera-Lozoya, B Berte, H Cochet, P Jaïs, N Ayache, M Sermesant. IEEE TBME 2016 + 2018

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

28

Simulate Brain Atrophy simul@trophy: open source simulator

Prescribed atrophy in 34 regions

Biophysical model (creeping flow)

parenchyma

CSF

µΔu − ∇p = (µ + λ )∇a

µΔu − ∇p = 0

∇ ⋅ u = −a.

∇ ⋅ u + p = 0.

a : prescribed atrophy; u,p : displacement & virtual pressure

Validation, training B. Khanal, M. Lorenzi, N. Ayache, X. Pennec. NeuroImage 2016

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

29

Future Trends • Exploiting « holistic »data • Images & signals • Biology, genetics, metabolomics, transcriptomics, etc.

• Behavior, lifestyle, etc.

Databases: ADNI, Enigma, UK Biobank, Insight, EDS, etc.

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

30

Neuropsychiatry Better monitor patients with neuropsychiatric diseases

Images

Idex UCAJedi Inria, UNS, CHU, IPMC, CoBTeK

Data Science Biology/genomics

Movement/activity

L. Antelmi, N. Ayache, P. Robert, M Lorenzi. Multi-Channel Stochastic Variational Inference for the Joint Analysis of Heterogeneous Biomedical Data in Alzheimer's Disease. Machine Learning in Clinical Neuroimaging (@MICCAI 2018).

C. Abi-Nader, N. Ayache, P. Robert, M Lorenzi. Alzheimer's Disease Modeling and Staging through Independent Gaussian Process Analysis of Spatio-Temporal Brain Changes. Machine Learning in Clinical Neuroimaging (@MICCAI 2018).

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

31

Imaging Genetics Alzheimer’s Disease

UCL

Genotype Phenotype

600 patients

Covariance Analysis (PLS) : MRI Phenotypes :~105 Grey matter atrophy

Genotypes SNPs: ~106

single nucleotide polymorphisms

Hippocampi, Amygdales Cingular, enthorinal, temporal cortex TRIB3 + APOE, TOMM40,… : Neurodegenerative processes Alzheimer’s disease

M Lorenzi,…,PM of brain atrophy to TRIB3 in Alzheimer's disease, evidence from functional N. Ayache Thompson, S Ourselin. Susceptibility AI & Digital Twins for e-Medicine 32 prioritization imaging genetics. PNAS 2018.

7 Nov in 2018

small n, large p

Lung Cancer Radiomics: variable selection

•

87 patients with a lung cancer

•

(NSCLC)

15/635 selected features for adenocarcinoma lesions

•

635 radiomic features based on CT images

14/635 selected features for squamous cell carcinoma lesions

•

11/635 selected features for other subtype

Prediction of cancer subtype

F Orlhac, P-A Mattei, C Bouveyron, and N Ayache. Class-specific variable selection in High-Dimensional Discriminant Analysis (HDDA) through Bayesian Sparsity. Journal of Chemometrics, 2018.

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

33

Breast Cancer: Radio - Metabolomics METABOLOMIC ANALYSIS

Resected lesions

Spectrum obtained with a mass spectrometer

Identification of triple-negative breast lesions Expression of 1500 metabolites

43 radiomic features

26 patients with a breast cancer RADIOMIC ANALYSIS

Conventional indices

Pre-treatment PET/CT

Metabolomic data

Radiomic data

Tumor lesion

F Orlhac, O Humbert, T Pourcher, L Jing, J-M Guigonis, J Darcourt, N Ayache, C Bouveyron. Statistical analysis of PET radiomic features and metabolomic data: prediction of triple-negative breast cancer. J Nucl Med,2018.

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

34

AI & Digital Twins New Computational Tools for Digital Medicine more personalized precise predictive preventive

To better heal the real patient N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

35

AI & Digital Twins Digital Medicine is developed to assist the physician, not to replace his/her: comprehension (global) compassion Natural Intelligence critical analysis

curiosity consciousness (professional)

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

36

Valeria Manera

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

37

Thanks Inria Team : Epione team & partners • Academic • Clinical • Industrial Medical Image Analysis Journal 20th anniversary special issue (free access)

team.inria.fr/Epione/

N. Ayache 7 Nov 2018

www.college-de-france.fr

AI & Digital Twins for e-Medicine

38

H. Delingette X. Pennec M. Sermesant M. Lorenzi C. Bouveyron N. Duchateau H. Lombaert O. Colliot D Chapelle A Garny F Orlhac

W. Wei P. Mlynarski M. Le C Audigier Q. Zheng J Krebs S Marchesseau R. Mollero J Margeta L Le Folgoc S. Giffard R. Cabrera B Khanal L Antelmi C Abi-Nader

Thanks P. Robert A. Criminisi B. Stankov T. Mansi, D. Comaniciu N. Guevara, T. Demarcy, C. Raffaelli, J Mahé, M Kohandani P. Hofman F Lacombe, S Loiseau D. Fontaine T. Demarcy, D. Gnasia L. Soler M. Auffret, F/J Brag M Haissaguerre et al.

P Jais, H. Cochet R Razavi J. Darcourt O. Humbert T. Pourcher D. Rueckert, S. Petersen

Medical Image Analysis Journal 20th anniversary special issue (free access)

team.inria.fr/Epione/

N. Ayache 7 Nov 2018

www.college-de-france.fr

AI & Digital Twins for e-Medicine

39

Marco Lorenzi Maxime Sermesant HervĂŠ Delingette

Xavier Pennec

N. Ayache 7 Nov 2018

Epione 2018 AI & Digital Twins for e-Medicine

Charles Bouveyron

40

Thanks Inria Team : Epione team & partners • Academic • Clinical • Industrial Medical Image Analysis Journal 20th anniversary special issue (free access)

team.inria.fr/Epione/

N. Ayache 7 Nov 2018

www.college-de-france.fr

AI & Digital Twins for e-Medicine

41

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

42

Medical Image Analysis Journal 20th Anniversary special issue 2016 free access • •

Editorial by N. Ayache & J. Duncan 32 visionary & historical papers with clinical & industrial transfer by: S. Wells, M. Brady, D. Rueckert, D. Comaniciu, A. Frangi, S. Ourselin, W. Niessen, J. Schnabel, A. Noble, A. Young, J. Weese, D. Hawkes,T. Peters, N. Navab, G. Szekely, L. Joskowicz, A. Criminisi, M. de Bruijne, D. Metaxas, N. Paragios, G. Gerig, R. Deriche, JF. Mangin, C. Barillot, M. Viergever, C. Davatzikos, P. Golland, K. Mori, M. Sonka, A. Madabhushi, R. Kikinis, G. Fichtinger

https://www.sciencedirect.com/journal/medical-image-analysis/vol/33/suppl/C

N. Ayache 7 Nov 2018

AI & Digital Twins for e-Medicine

43