Jonsson funk method considerations final by Nabarun Dasgupta

Methodological considerations in construct development Michele Jonsson Funk

33rd ICPE August 29th, 2017 Palais des Congress, Montreal, Canada 1

Disclosures â&#x20AC;˘ No funding specifically for this work. â&#x20AC;˘ No relevant conflicts of interest for this presentation.

Tradeoffs inherent to classification False Negatives <100% Sensitivity

Number of individuals

250

False Positives <100% Specificity

200

150

NonDiseased Diseased

100

120

115

110

105

100

Outcome Score

ROC interpretation

250 200 150 100 50 More

110

ROC interpretation

250 200 150 100 50 More

110

ROC interpretation

250 200 150 100 50 More

110

ROC interpretation

250 200 150 100 50 More

110

Roles and bias due to misclassification

Bias due to misclassified outcome • Toward the null if both are true: – Non-differential wrt exposure – Outcome is binary

• Unbiased in special cases 1. Perfect Sp + non-differential Se <100% → unbiased Relative Risk • If outcome is rare in all exposure groups, OR unbiased • If person-time is only impacted minimally, rate ratio unbiased

Effect of interest • Rate / risk in population of interest – Biased upward by poor Sp – Biased downward by poor Se

• Relative effect of exposure (RR) – Most algorithms maximize specificity because of special case

Bias due to misclassified exposure • Toward the null if all are true: – Non-differential misclassification wrt outcome – Error unrelated to other variables – No other sources of systematic error (selection bias, uncontrolled confounding) – Exposure is binary

Exposure misclassification in CER A1 vs A0 • Sn: 0.85 • Sp: 0.95 • True RR=2.00 • Obs RR=1.72 B1 vs B0 • Sn: 0.90 • Sp: 0.98 • True RR=2.00 • Obs RR=1.47 Jonsson Funk M, Landi SN. Curr Epidemiol Report 2014. (c) 2017 Michele Jonsson Funk

Exposure misclassification in CER A1 vs B1 • •

True RR=1.0 Obs RR=1.2

Impact of misclassified incl/excl • Inclusion – False negatives: Smaller n – False positives: • Individuals included who are not indicated for treatment

– Generalizability: Popn for inference is sicker, more medicalized

• Exclusion – False positive: Smaller n – False negatives: • Patients ineligible for tx (e.g. CKD) • Patients with prior events (e.g. cancer) which will appear to be incident (impacts outcome classification)

Bias due to misclassified confounder • Greenland’s (1980): “Partial control” finding – Adjustment for misclassified confounder still yields less biased estimate than not adjusting Good classification

Bad classification Slide courtesy of Mitchell Conover

Truth

Adjusted

Crude

Adjusted Crude

Bias due to misclassified confounder • Toward the crude (aka partial control) if all are true: – Nondifferential wrt exposure and outcome – No qualitative interaction between exposure and confounder – Independent of errors for study variables – Binary confounder effect on X are monotonic within Y; effect on Y are monotonic within X – Continuous covariate: misclassification independent of true value – No other sources of error

Likelihood of misclassification is related to true value

DEPEND

INDEPEND

NON‐ DIFF

DIFF

Effects within strata Qualitative modification

Slide courtesy of Mitchell Conover

Assessing algorithm performance • Gold standard vs alloyed • Study specific vs general purpose – Performance within key groups – exp x outcome; cov x exp; cov x outcome

• Over time – Availability of informative unstructured data sources (e.g. newly integrated radiology reports) – Changes in billing that influence practice (e.g. requirements for certain types of documentation)

Use of BMI to identify obesity â&#x20AC;¢ https://www.ncbi.nlm.nih.gov/pmc/articles/PM C5500540/

Shah NR, Braverman ER (2012) Measuring Adiposity in Patients: The Utility of Body Mass Index (BMI), Percent Body Fat, and Leptin. PLoS one 7(4): e33308. doi:10.1371/journal.pone.0033308

Subgroup differences: BMI>24 •

Sensitivity – M: 98% – F: 79%

•

Specificity – M: 35% – F: 87%

Subgroup differences: BMI >26 •

Sensitivity – M: 96% – F: 62%

•

Specificity – M: 63% – F: 90%

Major caveat on expected bias • Assumptions about direction of bias are true on average • Not necessarily in any single study • Plan on conducting quantitative bias analysis

Quantitative bias analysis • • • •

Simple bias analysis Probabilistic bias analysis (PBA) Bayesian bias analysis Multiple imputation for measurement error (MIME) • Propensity score calibration

PBA of construct measured with error • • • • •

Iterative form of simple bias analysis Adjusts treatment effect point estimate & CI Simultaneously adjust for other measured confounders Moves individuals between levels of one construct while holding others constant Requires estimates and distribution for: – Sensitivity and specificity OR – Positive and negative predictive values

Applying validation results • Near perfect specificity. – All classified as obese will stay in this category

• Poor sensitivity. – Move some individuals classified as non-obese into obese category

Non-obese (C*)

Obese (C*) X=1

X=0

Y=1

38 (A1)

20 (B1)

Y=0

461 (C1)

285 (D1)

499 (M1)

305 (N1)

X=1

X=0

Y=1

196 (A0)

114 (B0)

Y=0

4287 (C0)

3772 (D0)

4483 (M0)

3886 (N0)

Camelo Castillo 2015 JAMA Peds

Quantitative bias analysis (QBA) results Sensitivity

NaĂŻve

NaĂŻve:

Specificity

M i n

M o d e 1

M o d e 2

m a x

1.0

M i n

M o d e 1

M o d e 2

m a x

1.0

X-Y assn OR (2.5th, 97.5th)

1.46 (1.17, 1.81)

Assumes perfect Se and Sp (no misclassified obesity)

Quantitative bias analysis (QBA) results Sensitivity

Specificity

M i n

M o d e 1

M o d e 2

m a x

NaĂŻve

1.0

QBA 1

.11

.15

.32

NaĂŻve: QBA1:

X-Y assn OR (2.5th, 97.5th)

M i n

M o d e 1

M o d e 2

m a x

1.0

1.46 (1.17, 1.81)

.35

1.0

1.45 (1.16, 1.78)

Assumes perfect Se and Sp (no misclassified obesity) Trapezoidal distribution for Se, non-differential misclassification

Quantitative bias analysis (QBA) results Sensitivity

Specificity

M i n

M o d e 1

M o d e 2

m a x

NaĂŻve

1.0

QBA 1

.11

.15

QBA 2

.11

.15

NaĂŻve: QBA1: QBA2:

X-Y assn OR (2.5th, 97.5th)

M i n

M o d e 1

M o d e 2

m a x

1.0

1.46 (1.17, 1.81)

.32

.35

1.0

1.45 (1.16, 1.78)

.32

.35

.90

.95

.99

1.0

1.45 (1.19, 1.84)

Assumes perfect Se and Sp (no misclassified obesity) Trapezoidal distribution for Se, non-differential misclassification Trapezoidal distribution for Se and Sp, non-differential misclassification

Quantitative bias analysis (QBA) results Sensitivity

Specificity

M i n

M o d e 1

M o d e 2

m a x

Naïve

1.0

QBA 1

.11

.15

QBA 2

.11

QBA 3: Y=1 Y=0

Naïve: QBA1: QBA2: QBA3:

X–Y assn OR (2.5th, 97.5th)

M i n

M o d e 1

M o d e 2

m a x

1.0

1.46 (1.17, 1.81)

.32

.35

1.0

1.45 (1.16, 1.78)

.15

.32

.35

.90

.95

.99

1.0

1.45 (1.19, 1.84)

1.0

1.50 (1.24, 1.81)

.11

.15

.32

.35

1.0

Assumes perfect Se and Sp (no misclassified obesity) Trapezoidal distribution for Se, non-differential misclassification Trapezoidal distribution for Se and Sp, non-differential misclassification Differential misclassification (better in those Y=1 than Y=0)

Key considerations • Construct’s role in the study and the implications for bias due to misclassification / measurement error • Assessing performance – Related to exposure, outcome, subgroups – Validation may still be needed for existing algorithms

• Evaluate impact on effect estimates and confidence intervals – Quantitative bias analysis

Michele Jonsson Funk Âˇ mfunk@unc.edu 31