Evidence Based Practice: What do we really know? Joshua D Feder MD NFAR Mom’s Meeting July 20, 2016
3
Joshua D Feder MD Child and Family Psychiatrist Associate Clinical Professor, Department of Psychiatry, University of California at San Diego School of Medicine Adjunct Professor, Fielding Graduate University
Disclosures
Clinical Practice Teaching, Research, & Advocacy Early Years EU-UN BRIDGE & ACHIEVE UCSD SymPlay LLC
Disclosures
Clinical Practice Teaching, Research, & Advocacy Early Years EU-UN BRIDGE & ACHIEVE UCSD SymPlay LLC
Disclosures
Clinical Practice Teaching, Research, & Advocacy Early Years EU-UN BRIDGE & ACHIEVE UCSD SymPlay LLC
Disclosures
Clinical Practice Teaching, Research, & Advocacy Early Years EU-UN BRIDGE & ACHIEVE UCSD SymPlay LLC
Disclosures
Clinical Practice Teaching, Research, & Advocacy Early Years EU-UN BRIDGE & ACHIEVE UCSD SymPlay LLC
Disclosures
Clinical Practice Teaching, Research, & Advocacy Early Years EU-UN BRIDGE & ACHIEVE UCSD SymPlay LLC
Disclosures
Clinical Practice Teaching, Research, & Advocacy Early Years EU-UN BRIDGE & ACHIEVE UCSD SymPlay LLC
Disclosures
Clinical Practice Teaching, Research, & Advocacy Early Years EU-UN BRIDGE & ACHIEVE UCSD SymPlay LLC
Disclosures
Outline • Tell me something terrible • Nature is bigger than science, and science is shaky • If it gets better, did the treatment work? • 1+1 = truth : the importance of 2 good studies • Evidence Based Practice • Kim, Tearful : a graphic novella of evidence based practice • The best study to date
Tell me something terrible
Tell me something terrible… • Anxiety • Activity • Sleep • Perseveration • Aggression • Communication • Seizures • Gastrointestinal problems • Motor planning • Sensory problems • Cognition • • •
Tell me something terrible Dysregulation (poor self-regulation)
Tell me something terrible HALT - Hungry
Tell me something terrible Angry
Tell me something terrible Lonely
Tell me something terrible tired
Tell me something terrible Feder version of Bruce Perry’s Charts
State: Brain
Calm Mesocortex
Time
Extended future
Memory
STM/LTM
Thinking Reward
Abstract Beliefs
Over-arousal Rest Dissociation
Rest
Alert Limbic/ subcoritical Days/ hours Active WM/ STM Concrete Relational Flock/ vigilance Avoidance
Alarm Limbic/ midbrain Hours/ minutes
Fear Midbrain/ brainstem Minutes/ seconds
Terror Brainstem/ autonomic Loss of Sense of time
Emotional Sweet Salt Sex Fat Drugs Resistance
Reactive Relief of distress Defiance
Reflexive Rocking/ self stimulation Aggression
Compliance*
Dissociation Fainting
*see ‘prisons’
Tell me something terrible Targets
Gridding Out Target Symptoms
A ct iv it y
A tt e n ti o n
A n x i e t y
C o g n i t i o n
D e p r e s si o n
Moo d Inst abili ty “ag gres sion ”
Stimulants
+/-
+/-
-
+/ -
-
-
SSRIs
-
-
+/ -
-/ +
+?
-/+
Neuroleptics
+?
-?
+
-/ +
+?
++?
AEDs
+?
-/+
+
+?
Steroids
-?
-?
+?
-/ +? +?
Central Alpha
+?
+?
+?
-/
M ot or Pl an ni ng
+/-
O/ C, rig idi ty Pe rs ev er ati ve
Re cip ro cal int er ac tio n
Se ns or y Se nsi tiv ity
-
T i c s
-
S l e e p
-
+?
-
+?
+?
-
+?
+ +??
+?
+
+
++?
-?
+?
+?
+?
-/+
-?
+?
-?
++?
-?
+/ -?
+/-
1/+?
-/+?
+?
+?
+?
+ ? + ? +
-/ +
+
E t c …
Co mm ent s
Wt Ht tics Wt, Ht Sz Wt. Sz TD NMS Mult. SE… Mult SE… Sleep
Nature is bigger than science (and science is shaky)
Nature is bigger than science
‘Science’ is shaky stuf • The half life of medical information is 5 years…. • And we don’t know which half will change • That means that a lot of what we ‘know’ is wrong at any particular time.
Universe - T1
Universe - T2
If it gets better, did the treatment work?
If it gets better, did the treatment work? Maybe the treatment actually works: chicken soup‌ Placebo effect: belief and the production of endogenous opiates Regression to the mean – c.f. next slide
If it gets better, did the treatment work? Regression to the mean: Sleep Variability
http://www.babysleepsite.com/tag/baby-development-stages/
If it gets better, did the treatment work? Regression to the mean: Sleep Variability
Galland, B et all 2011 Normal sleep patterns in infants and children: A systematic review of observational studies For groups of kids at any age there is a range of sleep duration that varies about a mean, and
If it gets better, did the treatment work? Regression to the mean: Sleep Variability If the usual pattern is that any particular individual has a range of sleep duration Chances are that a day with shorter sleep will be followed by a day with longer sleep If you respond to a short sleep day by giving a potion, whether it works or not it is likely to be followed by a longer sleep
If it gets better, did the treatment work?
If it gets better, did the treatment work? Regression to the mean: Sleep Variability Hence the development of a belief that it works, whether it does or not. Moreover, if it only works sometimes, it may be even more reinforcing the ‘give the potion’ behavior (intermittent reinforcement). And to add to this, the likelihood of worry is high since over the course of development the duration of sleep gets smaller. How to sort it out? Science! E.g., with random use of the potion with objective measurement of the dependent variable (sleep duration), then look at whether is a correlation between the two.
If it gets better, did the treatment work? Bio - Psycho - Social Biological – Psychological –Social George Engel: cardiac care Carl Whittaker: the buffy coat
If it gets better, did the treatment work? Bio - Psycho - Social Covers a broadening range of possible influences Gets you thinking about all the factors involved Brainstorm with BPS as your guide
Baby’s not sleeping: BPS/SPB Spiritual – Big Bang (OMG!) Galactic: solar, gamma rays (Courchesne) Global: Environmental (Japan radioactivity?, rising tides?), Geopolitical (dad at war) National: Environmental (weather systems, wild fires); Political (ACT Today) Regional: Microenvironmental (dry air; CA autism clusters & SES); State cuts Local: Environmental (artillery exercises, red tide); Political (school district pink slips) Extended Family: far away; ‘Ghosts’; some with genetic (?) issues; babysitter issues? Immediate Family: Dad deployed, mom is ‘down’ and exhausted, worried about SIDS Child: responses to not getting much good mom time (anaclitic?); other (DMIC) General biological: not sleeping well, not eating well, hydration, medications Organ systems: teething, CNS, GI, Immune (OM?), Skin (rash), Hepatic, Renal, injury Cellular: DNA, RNA, mitochondrial function, insulin resistance Biomolecular: receptors and intracellular signaling (histamine, serotonin, etc.) Inorganic molecular/elemental: lead. CO, CO2, post fire particulate matter Atomic and subatomic: gamma rays, 11 dimensional universe 39
1+1 = truth
1+1 = truth Most studies that we see in the press are one-off’s conducted by the people who sell the treatment that show positive results. Bias aside, if there is a ‘significant’ result, this means that there is 19/20 chance that the findings are real (acceptable level of possible error). If you do twenty studies and 19 are negative and one is positive, and if only publish the positive one, then it looks like you have a ‘good’ treatment, when it really doesn’t work.
1+1 = truth If you have 2 positive studies, it’s harder to say that it doesn’t work, as long as ALL TRIALS including negative ones are being reported, and that the studies are conducted well. So it is really important to have not one but at least 2 positive studies, and that some are done by people who are NOT selling the treatment. But it’s hard to get funds to pay for this kind of confirmatory research.
1+1 = truth What’s a Good Study? The null hypothesis: assume ‘this won’t work’ or ‘this doesn’t matter’ and see if you are wrong. For any scientific paper, tease out the null hypothesis underlying the goal, e.g., ‘we looked at the charts of 200 cases to see if DIR helped’ implies the null hypothesis: ‘DIR made no difference for these people’ The gold standard for treatment has been the RCT: Randomized Controlled Trial, where one group of people get a treatment and another group does not and we compare the outcomes.
1+1 = truth Common Nonsense: Typical limitations in design and in thinking • no control group, • not a random sample, • confusing correlation with causation, • confounding variables, • percentages without range of error, • not enough trials to see a difference or a problem, • And there are many other possible issues… (Seethaler – Lies, Damn Lies, and Science).
1+1 = truth Parts to Look at in Research: GOAL: The goal of the experiment METHODS: The methods that were used RESULTS: The results they found ANALYSIS: Strengths, limitations, and problems STUDY RATING: do your own study rating! REFLECTION: How the experiment improved your understanding of stats, research, science, life, the universe, etc.
1+1 = truth
Understanding Statistics
1+1 = truth Random Scatter • Trust the universe to scatter in a predictably random (normal) way; • Predictable scatter gives you predictable math to help you know how confident you are in your data. • Find that scatter and you have the key to reliable statistical analysis. • Always look for the way to set things up so that you find where the data should scatter randomly. • THIS IS REALLY IMPORTANT
1+1 = truth Random Scatter
1+1 = truth Random Scatter
1+1 = truth Norman Curves
1+1 = truth Normal Curves
1+1 = truth
1+1 = truth Measures of central tendency: Mean -arithmetic average: e.g. average birth weight in babies from gestational diabetes. Median- middle value (like the median of a road): e.g. if you have bunch of kids with trisomy 21 (Down’s) average IQ might be 50, ranging about 25-80, but some people are outliers with IQ 100 or so. Mode - most often: e.g. baby names Range - how wide a range of hours of sleep in 6 month old infants? Variability - what’s the amount of scatter in a data set? Why is this important? Tells you how well you can predict the next measurement.
1+1 = truth Comparing Groups
1+1 = truth
1+1 = truth t-tests help you gauge the difference between sample groups t-tests tell you how likely it is that the null hypothesis is wrong. A big t means you may have found something interfering with the universe.
1+1 = truth
1+1 = truth Secondary Analysis A secondary Analysis allows you to find the place in the system where there is something that ought to be randomly scattered if the null hypothesis holds, and therefore something that can be predictably analyzed mathematically to see if it is ‘really’ random.
1+1 = truth: Study Ratings Item
Possible Points
Score
Study Q clearly focused?
1
2 clips scatter randomly?
Study population
2
0 ‘identical clips’, no control
Randomization
2
2 ‘pulled at random
Blinding
2
0 I did the intervention
Interventions
2
0 variability of setup
Outcomes
2
2 appropriate measuring
Statistical Analysis
2
1 limited - drawing
Results
1
1 I showed what’s there
Discussion (incl. limits)
1
1 Did I make my point?
Funding Source
2
2 full disclosure
Total Points
17 possible
11
Score (%)
65% (‘D’ for ‘RCT’)
Evidence Based Practice
Evidence Based Practice How Do We Decide What to Do? (Avoiding Nonsense) From Sackett 1996 to American Academy of Sciences Institute of Medicine 2001 to Buysee 2006 (IMH), and through to today The combination of relevant research with clinical judgment and experience to provide families with the information to make truly informed consent decisions based on their own family culture and values.
Evidence Based Practice Balanced thinking to avoid nonsense: Too much reliance on a research paper might not make sense (teaching to point to colored squares), or might not be appropriate for family (e.g. separation of child from parent) Too much reliance on clinical experience alone might lead to use of ineffective approaches and poor results (e.g. ‘wait and see’ for toddlers at risk for disorders of relating and communicating, overuse of antibiotics for ear infections)
Evidence Based Practice Yet nonsense abides‌ Progress in scientific thinking takes years to take hold Many researchers, advocates, and policy makers have not studied and do not understand this current and modern model It’s our job to teach them.
Evidence Based Practice How Do We Judge A Research Paper? Is it anecdotal information? If so, what is the value to that and what are the limits of that value? Doctors’ joke: 1 case = ‘in my experience’, 2 = ‘in a series…’ Did the researchers go beyond this and collect data? If so how did they make meaning from that data?
Kim, Tearful
Kyle’s…alone How? Why?
I can’t stop spying…
…he just wanders off from circle time…
They tried circles to sit on but Kyle just walks off
When he’s supposed to sit Kyle mostly stares out the window
Or falls asleep at his desk..
..he’s up late at night..
..Kyle’s always wanting carbs…he hardly eats anything else…
And when he does sit with us, we are talking about one thing and he is stuck on his own topic..
His balance isn’t so good either..
Oh! Sorry about that – but look: when there’s a noise he’s a mess
and then I’m a mess…
Kim, let’s take a pause, take a breath, and figure this out
Dr: Anyone else like this in the family? Kim: Uh… yeah.. he’s a lot like Ken’s dad…
I was so happy to have a baby..
Now I’m just so worried and sad…
What do I do?
We’ll figure this out together
An
And I know just the person to help out‌
Beth is great – she gets us playing… it’s like we found each other…
but it was still so maddening..
What is it about the toothbrush??
What is he seeing that’s so cool to him?
I can’t figure it out, so I just go with it. And then he’s so much easier to play with..
But I can’t do this alone. I need to get Ken more involved..
Ken is stressed too. He works so hard. And Kyle doesn’t play sports…
So I pause, I breathe, and figure it out
Now on weekends we are all playing on the living room floor with a gajillion toothbrushes‌.
…there’s still school to figure out..
We make school work about his toothbrush. Kyle draws and writes – I knew he was smart!
 Â
We give toothbrushes to the kids and they are great with him. Now Kyle wants to play chase games with them. The teacher helps and they are all having fun.
And he’s sleeping better too…
Ken got Kyle into gaming with him..
and one day when I spied again..
I cried all afternoon – sad and relieved all at once..
Kyle insisted on being Groot for Halloween – Ken figured it out: Groot looks like a toothbrush!
We still have tough moments. I know it’s going to be a long road.
But we pause, breathe, and we figure it out
Kim, Tearful DIR as an Organizing Philosophy Broad – whole child, supports family Welcoming – all about building love Enriching – closeness can bring progress
Kim, Tearful DIR in a nutshell Developmental levels – from regulation, to warm trust, and then a flow of enriching interactions Individual Differences – sensory, motor, communication, visualspatial, cognitive, etc. Relationship Based – all about connecting, and making time with others for support and help
Kim, Tearful co-regulated, engaged, and in a flow of interaction
Kim, Tearful First goal: regulated calm and attentive
Kim, Tearful Step 2: engaged – connected - in love
Kim, Tearful 3: some meaningful responses, 4: a flow of interactions gets daily things done, 5: able to create stories together
Kim, Tearful 6: solving practical problems together
Kim, Tearful Using our whole selves
Kim, Tearful Goal: fun with people to help us learn and grow
Kim, Tearful And supporting peer interactions
Kim, Tearful Developmental stages: summary • Homeostasis - Regulation • Falling in Love – Engagement (Connected) • Interactions - Circles of interaction • Flow • Symbolic • Logical • Multicausal thinking, nuance, reflective
The best study to date
The best study to date Play Project 4 hours per month Home coaching DIR approach Video review
The best study to date Play Project The PLAY Project completed a three-year multi-site randomized controlled trial which showed improvements in both parent-child interaction and autism symptomatology. This large scale study focused on the impact of our autism intervention: a parent-implemented (a.k.a. parent-mediated) play and relationship focused program
The best study to date Play Project Significant improvements in: • caregiver/parent and child interaction • social interaction of children with autism • social-emotional development of children with autism • autism symptomatology Secondary outcomes: • Improved parent stress and depression; and • PLAY Project consultant fidelity. In other words, they were true to the PLAY Project model and delivered it as trained.
The best study to date: Play Project “While efficacy data for early parent-implemented interventions are promising, there is currently only one parent-implemented ASD intervention with demonstrated effectiveness in community trials (Solomon et al., 2014).� From Aubyn C Stahmer, Lauren Brookman-Frazee, Sarah Rieth, Julia Trigeiro Stoner, Joshua D Feder, Karyn Searcyand Tiffany Wang: Parent perceptions of an adaptedevidence-based practice for toddlers with autism in a community setting Autism April 2016
Summary • Dysregulation is at the heart of many challenges • We don’t know a lot. Stay humble. • If it gets better, it might be due to treatment, but maybe not • 1+1 = truth: Look for two good studies, one by an outsider • Real informed consent demands that people tell you your options and you decide what is best for you and your family • Only one parent-mediated treatment has shown efficacy in a community sample, and outside research on it at UCSD is promising • Want to learn more?
The best study to date: Play Project