Uptake of internet-based therapy for depression: the role of the therapist

UPTAKE OF INTERNET-BASED THERAPY FOR DEPRESSION: THE ROLE OF THE THERAPIST

Mayke Mol

This thesis was prepared at the Department of Research & Innovation of GGZ inGeest, and the Department of Psychiatry, Amsterdam UMC, location VU University Medical Center. The research described in this thesis was financially supported by the MasterMind project (www.mastermindproject.eu) that was partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme (CIP) by the European Commission (Grant Agreement number: 621000).

Financial support for publication and distribution of this thesis was kindly provided by the department of Psychiatry, VU University Medical Center and the VU University Amsterdam, the Netherlands.

ISBN: 978-94-6416-150-2 Printing: Ridderprint, Ridderkerk, The Netherlands Copyright Š 2020 Mayke Mol All rights reserved. No part of this dissertation may be reproduced, or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording or any storage and retrieval without prior permission from the author or publishers of the included papers.

VRIJE UNIVERSITEIT

UPTAKE OF INTERNET-BASED THERAPY FOR DEPRESSION: THE ROLE OF THE THERAPIST

ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad Doctor aan de Vrije Universiteit Amsterdam, op gezag van de rector magnificus prof.dr. V. Subramaniam, in het openbaar te verdedigen ten overstaan van de promotiecommissie van de Faculteit der Geneeskunde op dinsdag 3 november 2020 om 11.45 uur in de aula van de universiteit, De Boelelaan 1105

door Mayke Mol geboren te Naarden

promotoren: copromotoren:

prof.dr. J.H. Smit prof.dr. M.M. Riper dr. D.J.F. van Schaik dr. E. Dozeman

CONTENTS CHAPTER 1

General introduction

CHAPTER 2

Improving Implementation of eMental Health for Mood Disorders in Routine Practice: Systematic Review of Barriers and Facilitating Factors

20

CHAPTER 3

The Therapist’s Role in the Implementation of Internet-based Cognitive Behavioral Therapy for Patients with Depression: Study Protocol

58

CHAPTER 4

Why Uptake of Blended Internet-Based Interventions for Depression Is Challenging: A Qualitative Study on Therapists’ Perspectives

74

CHAPTER 5

Dimensionality of the System Usability Scale among Professionals using Internet-based Interventions for Depression: A Confirmatory Factor Analysis Behind the Scenes of Online Therapeutic Feedback in Blended Therapy for Depression: Mixed-Methods Observational Study

110

General discussion

158

Samenvatting Dankwoord Curriculum vitae Publications and presentations Dissertation series

175 179 181 183 187

CHAPTER 6

CHAPTER 7 A B C D E

8

132

GENERAL INTRODUCTION | 7

CHAPTER 1

1

GENERAL INTRODUCTION

You almost have to be a logistical miracle to manage the administrative part nowadays, and suppose that this online platform is added, then it becomes completely unmanageable Therapist Ella

8 | CHAPTER 1 Although we know from research that internet-based Cognitive Behavioral Therapy (iCBT) can be very promising in the treatment of depression, the uptake in routine care in the Netherlands and globally is low. The gap between promising research findings and the need for upscaling was the reason for the initiation of the European MasterMind project (www.mastermind-project.eu). This project aimed to monitor and upscale the uptake and implementation of iCBT in the treatment of depression in Europe.

Information was gathered on the barriers and facilitators in routine care from the perspectives of different stakeholders: patients, therapists and representatives of mental health care organizations and health care systems from fifteen European regions. Therapists may be essential link in increasing the uptake of iCBT. As being among the most important gatekeepers for indicating the treatment of choice, they can ‘mediate’ the uptake by patients. Hence, the main focus of this thesis is the role of therapists in the provision of CBT in an internet-based format for patients with depression, specifically in routine specialized mental health care.

Depressive disorder According to the DSM-5, the Diagnostic and Statistical Manual of Mental Disorders [1], a person is classified with a major depressive disorder if the following criteria are met: 1) a depressed mood and/or 2) markedly diminished interest or pleasure in all or most activities, for most of the day and nearly every day for at least 2 consecutive weeks. This is accompanied with additional symptoms such as significant weight loss or gain, insomnia or hypersomnia, psychomotor agitation or retardation, fatigue or loss of energy, feelings of worthlessness or excessive or inappropriate guilt, diminished ability to think or concentrate, or indecisiveness, recurrent thoughts of death, recurrent suicidal ideation without a specific plan, or a suicide attempt or a specific plan for committing suicide.

Many people are affected by a depressive disorder. In 2015, global estimations added up to more than 320 million people of all ages that suffered from depression. On a European level, 40 million people were affected [2]. In the Netherlands, it is estimated that almost one out of five adults (18.7%) will have a major depressive disorder during his/her lifetime [3]. A depressive disorder substantially impacts every day functioning along with quality of life for both patients and their relatives. A high economic burden is also a result of depressive disorder [4], mainly caused by health care consumption and reduced work productivity [5].

Current status of depression care In the last decades, many depression intervention-studies have been conducted of which Cognitive Behavioral Therapy (CBT) has been studied most. CBT focuses on challenging and changing unhelpful thoughts, beliefs, attitudes and behaviors. The role of the therapist is to assist a patient in finding and practicing effective strategies to address the identified goals and reduce symptoms [6]. According to national and international guidelines, CBT is one of the treatments of choice next to pharmacotherapy and others forms of psychotherapy (e.g. IPT) [7,8]. CBT has shown to be effective in reducing depressive symptoms compared to control conditions with effect sizes ranging from .51 to .83 [9,10], and has comparable effects with pharmacotherapy [11]. However, most patients with a depressive disorder prefer psychotherapy over pharmacotherapy because antidepressants are often perceived as addictive, while therapy can provide an opportunity for personal exchange and solve the problems underlying the depression [12].

GENERAL INTRODUCTION | 9

Uptake of CBT Patient access to CBT is however limited and not all people with a depressive disorder seek professional care from general practitioner in primary care. One of the main barriers is personal stigma [4]. Many people feel embarrassed about seeking help from professionals and believe that other people would react negatively to them if they sought such help [13]. Once people find their way to specialized mental health care, other barriers arise: costs1, long waiting-lists, and limited available licensed therapists [14,15]. In the Netherlands, only about a third of the people with a depressive disorder receives professional mental health care, and this is estimated to be comparable in other European countries [3].

Another current issue in mental health care is the burden on the therapists; increasing staff shortages result in longer waiting-lists, and negatively impact therapists’ job satisfaction, which causes high rates of absenteeism among the workforce and a large staff turnover [16]. Next to treating patients, a therapists’ workload is affected by different time-consuming tasks such as administrative tasks, frequent cancellations and no-shows by patients who fail to show up without notice. It is not unusual for therapists to spend a third of their working day on administrative tasks [17].

Moreover, the uptake and provision of CBT can be further complicated. When CBT is applied in routine practice, many therapists (un)intentionally modify the CBT protocol and leave out key-elements [18]. When therapists underuse the optimal evidence-based treatment, this is called therapist drift.

Therapist drift is believed to be an important factor in why CBT is commonly less effective than it could be in routine clinical practice [19]. Among others, personal preferences (e.g. believing that there is no harm in selecting only parts of the protocol), attitude (e.g. overestimating the effect of therapeutic alliance), and also the context and working conditions (e.g. training skills, supervision and caseload) are thought to be relevant factors to why many patients do not appropriately receive full CBT.

Digital innovation of CBT Today our world is being digitalized to make our lives easier, faster, smarter, more efficient, cheaper and more durable. Various fields have implemented digital technology: schools (e.g. tablets, digital boards), sports (e.g. video assistant referee), banks (e.g. internet and mobile banking, contactless payment) and transport (e.g. electric cars, busses, bikes, airplanes). Also, we spent a big part of our daily life online; from grocery shopping to social communication.

Within the field of mental health care, various digital innovations slowly find their way to support the therapists. eMental health (eMh) is used as a container concept for the use of technology (and in many cases internet technology) to expand, assist, and enhance human activities, rather than as a substitute for them [20]. Examples in mental health care range from digital patient health records, information and administrative systems that monitor patient behavior with apps or tracking devices.

Treatment can for example be delivered via videoconferencing, mobiles or Virtual Reality, which can be applied throughout all phases of treatment: from prevention, to acute treatment and aftercare [21].

1

In the mental health care system in the Netherlands, patients first have to pay â‚Ź385 for treatments in specialized mental health care (mandatory own risk in 2020). Costs exceeding this amount are paid by health care insurance companies.

10 | CHAPTER 1 One of the innovations is internet-based Cognitive Behavioral Therapy (iCBT). CBT lends itself well to being converted to a digital format, due to its structured approach and psychoeducation component, as well as homework assignments and registration exercises that can be presented via web pages [22].

Although there are different formats created by different providers, the common idea behind iCBT is to provide patients with an online platform allowing them 24/7 access to (depression) modules.

Patients can work through online sessions at their own pace and within the privacy of their own home.

With internet access in 89% of the households in Europe and 98% in the Netherlands [23], the potential reach of iCBT is substantial and can expand patient access to treatment. Moreover, regarding the dissemination of iCBT, the potential of lowering pressure on the mental health care system, by reducing the amount of therapist time per patient, is used as a selling point. In addition, it is assumed that iCBT can be easily prescribed, offered with consistently high-quality and low fixed costs for patients. In general, iCBT can be divided into three formats: unguided, guided and more recently blended formats were also developed.

Unguided iCBT In unguided or self-help iCBT, patients go through online sessions on their own, without the therapeutic support of a professional. These programs can be provided with automated feedback to patients.

MoodGYM is an example of such a program that is freely available and widely-used [24]. Data gathered from research involving almost 4000 patients, has shown that a unguided iCBT format is a viable firststep treatment for depression: compared to control conditions unguided iCBT was significantly more effective (Hedges g = .27) [25]. This can be especially true for patients that would avoid mental health care, for example because of the fear of being stigmatized and a preference for anonymity.

Guided iCBT In guided iCBT, patients receive online therapeutic guidance from a health professional. Depending on the patients and setting, professionals involved can be prevention workers, general practitioners, students, trained and supervised volunteers or CBT therapists. Patients and therapists mostly communicate a-synchronically through secured digital written communication, wherein for example therapists give written feedback on patients’ homework assignments. The communication can also be synchronous, as in case of ‘chat’ support. Furthermore, there is often the possibility for therapists to monitor a patients´ online activity via symptom questionnaires and diary entries. The type of guidance within iCBT varies per setting and severity of the symptoms, and can range from 5-minute weekly telephone support, to extensive two-weekly asynchronous email support [26]. For example in the Netherlands, guided iCBT is offered at Interapy for more than a decade [27]. An impressive amount of research has shown that iCBT in a guided format, is an effective approach to reducing depressive symptoms, also in comparison with face-to-face CBT (Hedges g = .05) [28,29]. A meta-analysis has shown larger effect sizes for guided iCBT (Cohen´s d =.78) than unguided iCBT (Cohen´s d =.36) [30].

Blended CBT In blended CBT (bCBT), online sessions are combined with face-to-face sessions. bCBT can differ in duration, intensity and in ratio of online and face-to-face sessions; can consist of one integrated, standardized CBT treatment protocol [31] or more as an add-on [e.g. 33]. In case of an integrated bCBT

GENERAL INTRODUCTION | 11 protocol, reducing the face-to-face sessions in comparison to traditional CBT, bCBT can potentially still conserve therapist time [33]. This also might positively influence therapists’ workload. Another potential benefit for therapists is the pre-set structure of the blended protocol within an internet-based format. This format covers all core treatment principles and may help to avoid therapist drift in the face-to-face sessions [33,34]. Research has shown that bCBT is a feasible and effective treatment approach for patients with (severe) depression [34,35]. However, bCBT research is in its infancy and has not proven its cost-effectiveness yet [36].

Uptake of iCBT After a decade of iCBT research in 2011, Ruwaard announced: ‘The implementation and dissemination of online treatment has started’ [37]. Research on iCBT has shifted from randomized controlled trials (RCT’s), to investigating the translation from research findings of iCBT in routine practice effectiveness studies [38–40]. These studies have shown that iCBT can also be delivered under routine care conditions to patients with depression, who are characterized as having more complex problems than patient in RCT’s [41].

However, despite promising research results and advantages for the mental health care system, iCBT has not integrated into European routine care as widely as expected [42]. This is largely for reasons unknown. A survey among European mental health stakeholders (e.g. policy makers, care providers, researchers, platforms providers, patient organizations) showed that there is awareness of the potential benefits of iCBT, especially in term of cost-effectiveness. However, in the study of Topooco et al. [42] there were variations between countries and stakeholders in terms of knowledge about iCBT and their view on feasibility of embedding iCBT in routine care. It is possible that in countries where face-to-face treatment is limited or when the distances to mental health care are great, (unguided or guided) iCBT would be an attractive option to implement. When countries however have a good working mental health care system and when the distances are short, then stand-alone online iCBT might be less evident.

The discrepancies between expectations and the readiness of routine care needs to be investigated, in order to gain insight into the barriers and facilitators with implementation [43]. Gaining insight into factors influencing the uptake in routine practice on multiple levels, in complex contexts, can help with a successful implementation of iCBT. In 2016, a scoping review concluded that knowledge about implementation of Internet interventions for depression in regular care is limited and that therapist factors (e.g. qualifications, training and supervision) are underexposed [44]. There are indications that face-to-face therapists in routine care are skeptical about the effectiveness of iCBT; they perceive several limitations to iCBT, such as difficulties dealing with crisis situations, non-eligibility for severe depressive symptoms, and a one-size fits all approach [45–47]. Moreover, studies suggest that therapists prefer iCBT in a blended format [42,48,49]. Therapists believe that online and face-toface modalities could complement each other and address the limitations of stand-alone guided iCBT; patients might then be better prepared for the face-to-face sessions, and therapists could expect patients to be more independent and integrate easier into regular care.

In the Netherlands, the conditions for iCBT uptake can be considered optimal, with financial stimulations from policy and mental health care insurance companies, investments of mental health care organizations and a good technological infrastructure [50,51]. Even despite this, implementation

12 | CHAPTER 1 is still challenging and structure use in the Netherlands is low. A naturalistic study on the implementation of bCBT in a large mental health care organization in the Netherlands showed that only 3.6% of the patients with depression or anxiety were offered bCBT by 18% of all the trained therapists over a time period of 3 years [52]. In the end, suboptimal implementation of iCBT can lead to financial risks for mental health care organizations, and possibly ineffective use of the intervention. Therapists may play a crucial role in this as they might act as ‘mediators’ of attitude formation of their patients [53].

MasterMind This thesis was conducted within the framework of the MasterMind (Management of mental health disorders through advanced technology and services-telehealth for the MIND) project. This European project started in 2014 [54]. Fifteen regions monitored the uptake and implementation of iCBT for depression and targeted to reach 5000 patients. The objective was to understand which factors promote or hinder the uptake of iCBT for depression. The study was conducted throughout Europe to gain a broad perspective. This was done by exploring the role of different stakeholders (e.g. patients, therapists, mental health care organizations, regional mental health care systems) that were involved in the implementation of iCBT for depression.

In MasterMind the ‘Model for the assessment of telemedicine’ (MAST) was used to both guide and evaluate the implementation process [55]. The multi-disciplinary assessment of implementation outcomes included these domains: the health problem and characteristics of the intervention, safety, clinical effectiveness, patient and therapist perspectives, as well as economic, organizational, social, ethical, and legal aspects.

Aim of this thesis Researching the role a therapist plays within iCBT implementation could help close the gap between research and practice. The general aim of this thesis was to gain more insight into the role of the therapist within the implementation and use of blended CBT for depression in routine practice. All of this was done under the umbrella of the MasterMind project. Specifically, we wanted to know what prevents therapists from offering iCBT to their patients, and what factors have facilitated the therapists in the uptake and use of iCBT. Additionally, we aimed to validate a questionnaire measuring usability (the System Usability Scale) in a sample of professionals working with guided and blended iCBT in mental health care settings. Usability is an important factor in the implementation of internet-based interventions. Another goal was to explore the content of online therapist feedback within bCBT, as this is a substantial but unexplored part of internet-based interventions.

Outline of this thesis In line with our objectives, Chapter 2 describes a systematic review of the barriers and facilitators of eMh (e.g. internet-based interventions) in routine practice. This involves looking at patient, staff, organizational, and health care system level to overcome the gap between effectiveness studies and routine care. For this study, PubMed, PsychInfo and Embase were examined using a broad search strategy, with a high sensitivity to four key terms: implementation, mental health care practice, mood disorder, and eMh. A total of 13.159 articles were screened and 48 studies were included. To guide the

GENERAL INTRODUCTION | 13 review and to structure the results, Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework was applied. Chapter 3 presents a study protocol of a therapists‘ role in the implementation of iCBT in mental health care in the Netherlands. This was used to identity factors that promote or hinder therapists in their use of iCBT in depression care. To evaluate the uptake of iCBT via therapists, a mixed method approach was described. This was intended provide an understanding of the implementation factors (quantitative), and to ascertain the therapists’ facilitating and hindering factors in the implementation of iCBT (qualitative). Chapter 4 reports the results of the study protocol of the previous chapter, which focusses on therapists’ perspectives on usability, satisfaction, and the factors that promote or hinder the use of bCBT in routine care. Through focus groups and individual semi-structured interviews, therapists (N=36) from different mental health care organizations, with various background and experiences using bCBT throughout the Netherlands, were asked about their perspectives. Chapter 5 reports the results of a confirmatory factor analysis of the System Usability Scale (SUS), a 10-item questionnaire used to measure usability of iCBT. This was used to investigate whether the SUS is a valid instrument in measuring usability within this context. SUS data was drawn from a sample of professionals across Europe, who are working with guided iCBT for the treatment of depression. Chapter 6 describes the results of a mixed-methods observational study, regarding therapist feedback to patients receiving bCBT. We zoomed in on therapist behavior in written online communication with patients in bCBT for depression, in routine secondary mental health care. This was used to identify the extent to which the therapists adhere to feedback instructions, and to explore whether therapist behavior and adherence to feedback instructions are associated with patient outcome. 219 feedback messages sent from therapists to their patients were analyzed. Finally, a summary of the main findings, a general discussion, and the conclusion of this thesis are presented in Chapter 7.

14 | CHAPTER 1

References [1] [2] [3]

[4]

[5]

[6] [7]

[8] [9] [10]

[11]

[12]

[13]

[14]

[15]

American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Pub. World Health Organization (2017). World Health Organization Report - Depression and Other Common Mental Disorders: Global Health Estimates. World Health Organization; 1–24.

De Graaf, R., Ten Have, M., Van Gool, C., & Van Dorsselaer, S. (2012) Prevalence of mental disorders and trends from 1996 to 2009. Results from the Netherlands Mental Health Survey and Incidence Study-2. Social Psychiatry and Psychiatric Epidemiology; 47 (2), 203–213.

Boerema, A.M., Kleiboer, A., Beekman, A.T.F., Van Zoonen, K., Dijkshoorn, H., & Cuijpers, P. (2016). Determinants of help-seeking behavior in depression: a cross-sectional study. BMC psychiatry; 16(1), 78.

Gustavsson, A., Svensson, M., Jacobi, F., Allgulander, C., Alonso, J., Beghi, E., et al. (2011). Cost of disorders of the brain in Europe 2010. European neuropsychopharmacology; 21(10), 718779.

Beck, A. T, Rush, A. J., Shaw, B. F., & Emery, G. (1979). Cognitive therapy of depression. New York: Guilford Press.

Spijker, J., Bockting, C., & Meeuwissen, J. (2013). Multidisciplinaire Richtlijn Depressie (Derde Revisie): Richtlijn voor de diagnostiek, behandeling en begeleiding van volwassen patiënten met een depressieve stoornis [Multidisciplinary guideline Depression (Third revision): Guideline for the diagnosis, treatment and counseling of adult patients with a depressive disorder].

Utrecht: Trimbos Institute.

NICE (2009). Depression in adults: recognition and management. Clinical Guideline. In London: National Institute for Health and Clinical Excellence.

Butler, A.C., Chapman, J.E., Forman, E.M., & Beck, A.T. (2006). The empirical status of cognitivebehavioral therapy: A review of meta-analyses. Clinical Psychology Review; 26 17–31.

Cuijpers, P., Berking, M., Andersson, G., Quigley, L., Kleiboer, A., & Dobson, K.S. (2013). A metaanalysis of cognitive-behavioural therapy for adult depression, alone and in comparison with other treatments. Canadian Journal of Psychiatry; 58 (7), 376–385.

Cuijpers, P., Sijbrandij, M., Koole, S.L., Andersson, G., Beekman, A.T., & Reynolds, C.F. (2013).

The efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: A meta-analysis of direct comparisons. World Psychiatry; 12 (2), 137–148.

Van Schaik, D.J.F., Klijn, A.F.J., Van Hout, H.P.J., Van Marwijk, H.W.J., Beekman, A.T.F., De Haan, M., et al. (2004). Patients’ preferences in the treatment of depressive disorder in primary care.

General Hospital Psychiatry; 26 (3), 184–189.

Barney, L.J., Griffiths, K.M., Jorm, A.F., & Christensen, H. (2006). Stigma about Depression and its Impact on Help-Seeking Intentions. Australian & New Zealand Journal of Psychiatry; 40 (1), 51–54.

Hutschemaekers, G.J.M., Tiemens, B.G., & De Winter, M. (2007). Effects and side-effects of integrating care: The case of mental health care in the Netherlands. International Journal of Integrated Care; 7 (3),.

Saxena, S., Thornicroft, G., Knapp, M., & Whiteford, H. (2007). Resources for mental health: scarcity, inequity, and inefficiency. Lancet; 370 (9590), 878–889.

GENERAL INTRODUCTION | 15 [16] [17]

[18] [19]

[20] [21] [22] [23] [24] [25]

[26] [27]

[28]

[29]

[30] [31]

[32]

KPMG (2019). Health check. Ontwikkelingen binnen de health check GGZ zorg [Health check/ Developments within the health check mental health care]. Amstelveen: KPMG Nederland.

GGZ Nederland (2017). Het roer moet om: onderzoek naar de administratieve lasten in de geestelijke gezondheidszorg [Things have to change: research into the administrative burden in mental health care]. Amersfoort: GGZ Nederland.

Waller, G. (2009). Evidence-based treatment and therapist drift. Behaviour Research and Therapy; 47 (2), 119–127.

Waller, G. & Turner, H. (2016). Therapist drift redux: Why well-meaning clinicians fail to deliver evidence-based therapy, and how to get back on track. Behaviour Research and Therapy; 77 129–137.

Oh, H., Rizo, C., Enkin, M., & Jadad, A. (2005). What is eHealth (3): A systematic review of published definitions. Journal of Medical Internet Research; 7 (1),.

Cuijpers, P., Kleiboer, A., Karyotaki, E., & Riper, H. (2017). Internet and mobile interventions for depression: Opportunities and challenges. Depression and Anxiety; 34 (7), 596–602.

Przeworski, A. & Newman, M.G. (2006). Efficacy and utility of computer-assisted cognitive behavioural therapy for anxiety disorders. Clinical Psychologist; 10 (2), 43–53.

Eurostat (2019). Level of internet access - households. Data set 2019.

Christensen, H., Griffiths, K.M., & Jorm, A.F. (2004). Delivering interventions for depression by using the internet: Randomised controlled trial. Br Med J; 328 (7434), 265–268.

Karyotaki, E., Riper, H., Twisk, J., Hoogendoorn, A., Kleiboer, A., Mira, A., et al. (2017). Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms a meta-analysis of individual participant data. JAMA Psychiatry; 74 (4), 351–359.

Andersson, G., Wagner, B., & Cuijpers, P. (2016). ICBT for Depression. in: Guid. Internet-Based Treat. Psychiatry, Springer International Publishing, 17–32.

Lange, A., Schoutrop, M., Schrieken, B., & Ven, J.-P. van de (2002). Interapy: A model for therapeutic writing through the Internet. in: S.J. Lepore, J.M. Smyth, S.J. Lepore, J.M. Smyth (Eds.), Writ. Cure How Expressive Writ. Promot. Heal. Emot. Well-Being., American Psychological Association, Washington, DC, US, 215–238.

Carlbring, P., Andersson, G., Cuijpers, P., Riper, H., & Hedman-Lagerlöf, E. (2018). internetbased vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cognitive Behaviour Therapy; 47 (1), 1–18.

Cuijpers, P., Noma, H., Karyotaki, E., Cipriani, A., & Furukawa, T.A. (2019). Effectiveness and Acceptability of Cognitive Behavior Therapy Delivery Formats in Adults with Depression: A Network Meta-analysis. JAMA Psychiatry; 76 (7), 700–707.

Richards, D. & Richardson, T. (2012). Computer-based psychological treatments for depression: A systematic review and meta-analysis. Clinical Psychology Review; 32 (4), 329–342.

Definition blended treatment Heleen Riper. (2017). Triple-E website. URL: https://www.tripleehealth.nl/en/projects-cat/depressie/. Archived at http://www.webcitation.org/6wZFBY9pX on January 18th, 2018.

Hoifodt, R.S., Lillevoll, K.R., Griffiths, K.M., Wilsgaard, T., Eisemann, M., Waterloo, K., et al.

(2013). The clinical effectiveness of web-based cognitive behavioral therapy with face-to-face therapist support for depressed primary care patients: randomized controlled trial. J Med Internet Res; 15 (8), e153.

16 | CHAPTER 1 [33]

[34]

[35]

[36]

[37]

[38]

[39]

[40]

[41]

[42]

[43]

[44]

[45]

[46]

Kooistra, L.C., Wiersma, J.E., Ruwaard, J., van Oppen, P., Smit, F., Lokkerbol, J., et al. (2014).

Blended vs. face-to-face cognitive behavioural treatment for major depression in specialized mental health care: study protocol of a randomized controlled cost-effectiveness trial. BMC Psychiatry; 14 (1), 290.

Mansson, K.N., Skagius Ruiz, E., Gervind, E., Dahlin, M., & Andersson, G. (2013). Development and initial evaluation of an internet-based support system for face-to-face cognitive behavior therapy: a proof of concept study. J Med Internet Res; 15 (12), e280.

Thase, M.E., Wright, J.H., Eells, T.D., Barrett, M.S., Wisniewski, S.R., Balasubramani, G.K., et al.

(2018). Improving the efficiency of psychotherapy for depression: Computer-assisted versus standard CBT. American Journal of Psychiatry; 175 (3), 242–250.

Kooistra, L.C., Wiersma, J.E., Ruwaard, J., Neijenhuijs, K., Lokkerbol, J., van Oppen, P., et al.

(2019). Cost and Effectiveness of Blended Versus Standard Cognitive Behavioral Therapy for Outpatients With Depression in Routine Specialized Mental Health Care: Pilot Randomized Controlled Trial. Journal of Medical Internet Research; 21 (10), e14261.

Ruwaard, J., Lange, A., Schrieken, B., & Emmelkamp, P. (2011) Efficacy and effectiveness of online cognitive behavioral treatment: a decade of interapy research. Stud Health Technol Inform; 167, 9–14.

Hadjistavropoulos, H.D., Pugh, N.E., Nugent, M.M., Hesser, H., Andersson, G., Ivanov, M., et al.

(2014). Therapist-assisted Internet-delivered cognitive behavior therapy for depression and anxiety: translating evidence into clinical practice. J Anxiety Disord; 28 (8), 884–893.

Hedman, E., Ljotsson, B., Kaldo, V., Hesser, H., El Alaoui, S., Kraepelien, M., et al. (2014).

Effectiveness of internet-based cognitive behaviour therapy for depression in routine psychiatric care. J Affect Disord; 155, 49–58.

Mathiasen, K., Riper, H., Andersen, T.E., & Roessler, K.K. (2018). Guided internet-based cognitive behavioral therapy for adult depression and anxiety in routine secondary care: Observational study. Journal of Medical Internet Research; 20 (11), e10927.

Titov, N., Dear, B., Nielssen, O., Staples, L., Hadjistavropoulos, H., Nugent, M., et al. (2018). ICBT in routine care: A descriptive analysis of successful clinics in five countries. Internet Interventions; 13, 108–115.

Topooco, N., Riper, H., Araya, R., Berking, M., Brunn, M., Chevreul, K., et al. (2017). Attitudes towards digital treatment for depression: A European stakeholder survey. Internet Interventions; 8, 1–9.

Wensing, M., Oxman, A., Baker, R., Godycki-Cwirko, M., Flottorp, S., Szecsenyi, J., et al. (2011).

Tailored implementation for chronic diseases (TICD): A project protocol. Implementation Science; 6 (1), 103.

Drozd, F., Vaskinn, L., Bergsund, H.B., Haga, S.M., Slinning, K., & Bjørkli, C.A. (2016). The implementation of internet interventions for depression: A scoping review. Journal of Medical Internet Research; 18 (9).

Erbe, D., Psych, D., Eichert, H.C., Riper, H., & Ebert, D.D. (2017). Blending face-to-face and internet-based interventions for the treatment of mental disorders in adults: Systematic review. Journal of Medical Internet Research; 19 (9), e306.

Hadjistavropoulos, H.D., Nugent, M.M., Dirkse, D., & Pugh, N. (2017). Implementation of

GENERAL INTRODUCTION | 17

[47]

[48]

[49]

[50] [51] [52]

[53] [54]

[55]

internet-delivered cognitive behavior therapy within community mental health clinics: A process evaluation using the consolidated framework for implementation research. BMC Psychiatry; 17 (1).

Folker, A.P., Mathiasen, K., Lauridsen, S.M., Stenderup, E., Dozeman, E., & Folker, M.P. (2018).

Implementing internet-delivered cognitive behavior therapy for common mental health disorders: A comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interventions; 11, 60–70.

Van der Vaart, R., Witting, M., Riper, H., Kooistra, L., Bohlmeijer, E.T., & van Gemert-Pijnen, L.J. (2014). Blending online therapy into regular face-to-face therapy for depression: content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry; 14 (1), 355.

Kivi, M., Eriksson, M.C.M., Hange, D., Petersson, E.-L., Björkelund, C., & Johansson, B. (2015).

Experiences and attitudes of primary care therapists in the implementation and use of internetbased treatment in Swedish primary care settings. Internet Interventions; 2 (3), 248–256.

Riper, H., Ballegooijen, W. van, Kooistra, L., De Wit, J., & Donker, T. (2013). Preventie & eMental-health. Kennissynthese 2013.

Ruwaard, J.J. & Kok, R.N. (2015) Wild West eHealth: Time to Hold our Horses? The European Health Psychologist; 17 (1), 45–49.

Kenter, R.M.F., van de Ven, P.M., Cuijpers, P., Koole, G., Niamat, S., Gerrits, R.S., et al. (2015).

Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: Results from a naturalistic study. Internet Interventions; 2 (1), 77–83.

Gun, S.Y., Titov, N., & Andrews, G. (2011). Acceptability of Internet treatment of anxiety and depression. Australas Psychiatry; 19 (3), 259–264.

Vis, C., Kleiboer, A., Prior, R., Bønes, E., Cavallo, M., Clark, S.A., et al. (2015). Implementing and up-scaling evidence-based eMental health in Europe: The study protocol for the MasterMind project. Internet Interventions; 2 (4).

Kidholm, K., Ekeland, A.G., Jensen, L.K., Rasmussen, J., Pedersen, C.D., Bowes, A., et al. (2012).

A model for assessment of telemedicine applications: mast. International Journal of Technology Assessment in Health Care; 28 (1), 44–51.

18 | CHAPTER 2

CHAPTER 2

The most important thing is that you just do it. In advance you have all kinds of schedules in mind about how complicated it is, but because it is structured in such a way, it is actually very easy Therapist Tom

Systematic Review: Implementation of eMh | 19

Improving Implementation of eMental Health for Mood Disorders in Routine Practice: Systematic Review of Barriers and Facilitating Factors Christiaan Vis, Mayke Mol, Annet Kleiboer, Leah BĂźhrmann, Tracy Finch, Jan Smit & Heleen Riper JMIR Mental Health 2018

2

20 | CHAPTER 2

Abstract

Background: Electronic mental health interventions (eMental health or eMh) can be used to increase accessibility of mental health services for mood disorders, with indications of comparable clinical outcomes as face-to-face psychotherapy. However, the actual use of eMh in routine mental health care lags behind expectations. Identifying the factors that might promote or inhibit implementation of eMh in routine care may help to overcome this gap between effectiveness studies and routine care.

Objective: This paper reports the results of a systematic review of the scientific literature identifying those determinants of practices relevant to implementing eMh for mood disorders in routine practice.

Methods: A broad search strategy was developed with high sensitivity to four key terms: implementation, mental health care practice, mood disorder, and eMh. The reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework was applied to guide the review and structure the results. Thematic analysis was applied to identify the most important determinants that facilitate or hinder implementation of eMh in routine practice.

Results: A total of 13,147 articles were screened, of which 48 studies were included in the review. Most studies addressed aspects of the reach (n=33) of eMh, followed by intervention adoption (n=19), implementation of eMh (n=6), and maintenance (n=4) of eMh in routine care. More than half of the studies investigated the provision of mental health services through videoconferencing technologies (n=26), followed by internet-based interventions (n=20). The majority (n=44) of the studies were of a descriptive nature. Across all RE-AIM domains, we identified 37 determinants clustered in six main themes: acceptance, appropriateness, engagement, resources, work processes, and leadership. The determinants of practices are expressed at different levels, including patients, mental health staff, organizations, and health care system level. Depending on the context, these determinants hinder or facilitate successful implementation of eMh. Conclusions: Of the 37 determinants, three were reported most frequently: 1) the acceptance of eMh concerning expectations and preferences of patients and professionals about receiving and providing eMh in routine care, 2) the appropriateness of eMh in addressing patients’ mental health disorders, and 3) the availability, reliability, and interoperability with other existing technologies such as the electronic health records are important factors for mental health care professionals to remain engaged in providing eMh to their patients in routine care. On the basis of the taxonomy of determinants of practices developed in this review, implementation-enhancing interventions can be designed and applied to achieve better implementation outcomes. Suggestions for future research and implementation practice are provided.

Keywords: eMental health, implementation, routine practice, determinants of practices, RE-AIM, barriers and facilitators, mood disorders, review.

Systematic Review: Implementation of eMh | 21

Introduction Background Electronic mental health interventions (eMental health or eMh) for mood disorders such as depression can increase reach and accessibility of mental health services while maintaining comparable clinical outcomes as face-to-face interventions and superior outcomes compared with waiting lists [1-3]. eMh encompasses the use of digital technologies and new media for the delivery of screening, health promotion, prevention, early intervention, treatment, or relapse prevention, as well as for improvement of health care delivery (e.g. electronic patient files), professional education (e-learning), and Web-based research in the field of mental health [4]. Research on the translation of the results of these studies into routine care is scarce. Translational research can have two dimensions: dissemination and implementation of an innovation in clinical practice. Dissemination concerns the passive and active spread of information about eMh to relevant stakeholders, including consumers, clinical care providers, and decision- and policy makers. Implementation refers to the process of embedding and integrating new practices into actual care settings [5,6]. It seems that eMh interventions are reasonably well disseminated to clinical practice given that a number of preconditions are fulfilled, such as the availability of technical infrastructures and proper reimbursement of these services [7]. Nevertheless, the actual use of eMh in routine mental health care lags behind expectations. It is unclear why implementation of eMh remains difficult.

A logical approach in addressing this implementation challenge is to identify the factors that might promote or inhibit implementation of eMh in routine practice [8]. On the basis of these determinants, implementation-enhancing interventions might be designed and applied with the aim to improve implementation processes and upscaling of eMh care. Many determinants of different care practices have been identified for a variety of clinical interventions. For example, Krause and colleagues [9] identified over 600 context-specific determinants thought to be relevant in implementing evidencebased interventions for patients with chronic health conditions, including depression in the elderly, chronic obstructive pulmonary disease, and obesity. Examples of these determinants are status and quality of evidence and clinical recommendations, characteristics of the innovation, delivery modalities, reimbursement modalities, implementation leadership, and organizational readiness [10-12]. Similarly, examples of implementation barriers for eMh include the perceived importance of computer literacy skills, knowledge and awareness of existing eMh services, as well as credibility of these services [13]. In turn, many of these determinants have been clustered and framed, currently resulting in more than 60 frameworks used to study and understand implementation processes [14,15]. Although such determinants and frameworks are valuable and comprehensive, they lack specificity to any category of intervention and therefore, provide little practical detail to prioritize determinants and guidance for action to improve the implementation of eMh interventions.

The reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework provides a heuristic tool for bridging interventions’ internal validity established in well-controlled conditions and their external validity in real-world conditions [16,17]. It is designed to evaluate the public health impact of health promoting interventions, and it is widely used in implementation research [18]. The framework covers five intervention-related areas of impact: 1) reach as the ability to address those in need of an intervention, 2) effectiveness in terms of the impact of interventions on health outcomes, 3) adoption as a decision to proceed with implementing the clinical intervention, 4)

22 | CHAPTER 2 implementation as the process of embedding and integration of the intervention in routine practice and its consistency of delivery and costs, and 5) maintenance as the institutionalization of the intervention in routine care [16,18-20]. Considering the current evidence-base for eMh and the increasing emphasis on comparative effectiveness research in testing clinical and cost-effectiveness of eMh [21], the RE-AIM framework might be a valuable tool to structure determinants of practices that are specific to eMh. Research Question Given the absence of a comprehensive overview of determinants of practices, we systematically reviewed the literature to develop a taxonomy relevant to the implementation of eMh. Knowledge on these determinants can inform the study of interventions that aim to improve the implementation of eMh in routine practice. The following research question guided the research: “What determinants of practice are identified as relevant to implementing eMh interventions for mood disorders in routine practice?” A broad view on eMh and care practice settings, including clinical and community practices, was adopted to provide a comprehensive taxonomy of determinants of mental health practice relevant to implementing eMh.

Methods Study Design A systematic review of scientific literature was conducted. RE-AIM was used to structure the review.

Various implementation studies in the area of mental health care using RE-AIM substantiate the utility of this framework, including evaluations of the implementation of behavior mental health assessment tools [22]; smoking cessation interventions in people with mental illnesses [23]; mental health, substance abuse, and health behavior interventions into specific primary care behavior health programs [24]; tele-mental health consultation program in pediatric primary care in rural settings [25]; and assessing a therapist’s role in eMh for patients with depressive disorders [26].

Search Strategy Due to the novelty of the topics concerned (ie, eMh and implementation), a broad search strategy was developed with high sensitivity to four key terms (as opposed to a focused strategy with higher specificity [27]): “implementation,” “mental health care practice,” “mood disorder,” and “eMentalhealth.” No time frame was applied. On the basis of literature, benchmark definitions for these concepts were developed, and a total of 408 synonyms were formulated for the search strings. A trained librarian guided the formulation of the search strings. The benchmark definitions and search strings are included in Appendix 1. The search was conducted in July 2015 in the three main bibliographical databases (PubMed, PsycINFO, and EMBASE). All identified papers were examined for eligibility by two researchers (CV and MM) independently. Disagreements were solved by discussion and, where necessary, moderated by a third researcher (AK) to reach consensus.

Systematic Review: Implementation of eMh | 23 Inclusion and Exclusion Criteria The inclusion and exclusion criteria are shown in Boxes 1 and 2.

Box 1. Inclusion criteria.

1)

Reporting of empirical research such as observational studies using ethnographic methods or experimental studies following a pre-post or randomized controlled trial design

2)

The psychotherapeutic intervention under study had an information and communication technology (ICT) component (e.g., using videoconferencing, Web, or mobile technologies to deliver mental health care)

3) 4)

The psychotherapeutic intervention targeted a mood disorder.

5)

The study took place in routine mental health care settings.

The study targeted (a) an adult population, (b) mental health care professionals (HCPs) or, (c) other persons or organizations involved in implementation of eMh.

Box 2. Exclusion criteria.

1) 2) 3)

Studies were reporting clinical effectiveness data only.

The full-text article was not available through Open Access or library loaning services.

The full-text article was not available in the English language.

Data Extraction A systematic qualitative narrative approach was applied for the data extraction, analyses, and synthesis of the results [28-30]. A field guide was developed to extract relevant data from the retained articles.

Items included the study aim, methods, the psychotherapeutic intervention, eMh technology applied, type of mood disorder, implementation intervention (e.g. training of professionals, or a focused marketing campaign to raise awareness of eMh among patients), settings, sample(s), recruitment procedures, results, and findings in terms of determinants of practice. The data were tabulated and categorized in accordance with four of the five RE-AIM dimensions: reach, adoption, implementation, and maintenance. Table 1 presents definitions and adaptations to the RE-AIM dimensions that we applied for the purpose of this study. Effectiveness was not addressed in this review as ample reviews on the clinical effectiveness of eMh for mood disorders are available [1-3]. The implementation dimension was broadened to also include the purposive implementation interventions that might have been employed to achieve better implementation outcomes.

24 | CHAPTER 2 Table 1. Dimensions of reach, effectiveness, adoption, implementation, and maintenance (RE-AIM); their definitions; and its focus.

Dimension

Definitions [16]

Reach

Participation ratio of patients and their characteristics Impact of the (clinical) intervention on patients’ health, quality of life, and economic outcomes Proportion and representativeness of staff and organizations delivering the services (Clinical) interventions’ fidelity and (implementation) costs

Effectiveness Adoption Implementation

Maintenance aeMh:

Comment

Not addressed in this study

Added: deliberate and purposive actions to implement eMha [31]

Extent to which the intervention is and remains to be part of routine care practice

electronic mental health interventions, or eMental health.

Analyses and Synthesis Thematic analysis was applied to identify the recurrent and most important determinants to implementing eMh in routine practice (i.e., themes) arising in the included literature. Thematic analysis is a common method for identifying, grouping, and summarizing findings from included studies in narrative review [29]. The (groups of) determinants were developed inductively (i.e., without a priori defined topics guiding the analysis). We did not apply a threshold for recurrence of certain themes in the data. Data were extracted by three researchers (CV, MM, and LB) independently. Data files were merged and discrepancies solved by discussion to reach consensus. Freely available reference management software (Mendeley, Elsevier), a spreadsheet (Microsoft Excel, Microsoft Corporation), and qualitative analysis software (ATLAS.ti, Scientific Software Development GmbH) were used to organize and conduct the selection, data extraction, and data analysis.

Results Study Selection The searches resulted in 16,718 records. After removing the duplicates, 13,417 unique titles remained and were screened for eligibility against the inclusion and exclusion criteria. In total, 13,159 articles were excluded on the basis of the information in titles and abstracts. A total of 258 articles were retained, and after examining the full-text articles, a total of 48 studies were included in the analysis.

Figure 1 provides an overview of the inclusion and exclusion of studies in the different phases of the systematic review.

Records identified in PubMe, PsycINFO and Embase (n=16.718)

Records after dublicates removed (n=13.417)

Eligibility

Titles and abstracts assessed for eligibility (n=13.417)

Full-text articles assessed for eligiblity (n=258)

Included

Screening

Identification

Systematic Review: Implementation of eMh | 25

Studies included in analysis (N=48)

Articles excluded on basis of inclusion/exclusion criteria (n=13.159)

Full-text articles excluded (n=209), for reasons of: - Non-primary research (protocols, editorials, methodology etc.) (n=59) - Not aimed at eMental health for adult aged population (n=48) - Feasibility studies including piloting and design studies (n=28) - Not adressing mood disorders (n=20) - Clinical efficacy and effectiveness studies (n=17) - Not focussed at implementing a service in routine practice (n=11) - Dissertation, books, and policy reports including guidelines (n=9) - Single retrospective case descriptives (n=11) - Unretrievable and/or available in Engslish (n=6) - Combined due to reporting on same research (n=1)

Figure 1. Information flow through the different phases of the systematic review.

General Study Characteristics Table 2 provides an overview of the main characteristics of the studies, including the RE-AIM dimension(s) addressed, target disorder, therapeutic principles, technology applied, guidance modalities, and study design.

Most studies investigated reach (n=33), followed by adoption (n=19), implementation (n=6), and maintenance (n=4). The specific type of the target disorder was often described in broad terms such as common mental disorders or mood disorders (n=20), or in exemplary disorders such as depression or anxiety (n=17). Most studies (n=39) did not explicitly report the therapeutic principles of the clinical intervention that was implemented. More than half of the studies investigated the provision of mental health services for mood disorders through videoconferencing technologies (n=26), most often by using videoconferencing for support and consultations. The remainder of the studies focused on internet-based interventions (n=20). Three studies looked at purely self-help interventions (through Web and mobile technologies), and 10 studies did report on a specific eMh intervention but did not report the guidance modality. Eighteen studies specified the eMh intervention and described the guidance modality. The majority (n=44) of the studies were of an observational, that is, descriptive nature. Most of these (n=20) applied mixed-methods (e.g., a survey and semi-structured interviews), followed by a large proportion (n=16) of studies that applied qualitative methods such as ethnography

26 | CHAPTER 2 or consensus-seeking methods using focus-group discussions. Five studies were of an experimental design, applying either quantitative or mixed-methods.

More information about the specific studies’ aims, designs, settings, participants, and clinical and implementation-related interventions are reported in Appendix 2, accessible online at https://mental.jmir.org/article/downloadSuppFile/9769/66878.

Table 2. Overview of studies categorized per reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) domain; technology applied; target disorder; therapeutic principles; and study design.

Characteristic Target disorder Depressive disorder Mood disordersc Not specifiedd Therapeutic principlese Cognitive behavior therapy Other(e.g. mindfulness) General psychotherapy Technology applied Internet-based (unguided) Internet-based (guidedf) Internet-based (minimal guidance) Internet-based (therapist guided) Internet-based (blended) Internet-based (not specifiedg) Computer-based Mobile health (unguided) Videoconferencing Study design Experimental—quantitative methods Experimental—mixed-methods Observational—qualitative methods Observational—quantitative methods Observational—mixed-methods aThe

Reach (n=33)

Adoption (n=19)

Implementation (n=6)

Maintenance (n=4)

na

8 16 8

3 9 7

2 4

-b 2 2

10 20 17

5 1 27

3 16

2 4

4

8 1 39

2 3 1 1 1 8 1 1 15

3 1 2 1 12

1 1 4

4

2 5 1 1 1 10 1 1 26

2

-

-

-

2

10

2 9

1 2

1

3 15

6

1

-

1

8

15

7

2

2

20

n in this column are unique references. Some studies were categorized under more than one RE-AIM dimension. bRefers to no studies categorized under that condition. cMood disorders including depressive disorder and/or in combination with other mental health disorders. dRefers to the studies that described the target disorder in exemplary wordings without becoming specific. The generic wordings related to mood disorders. eNot all studies specifically discussed the target disorder or psychotherapeutic principles of the service as studies focused, for example, on perceptions of the delivery method relevant to implementation and not on the specific treatment itself. fSome form of guidance; guidance modality and intensity was not specified. gNot specified if it was a guided intervention or self-help.

Systematic Review: Implementation of eMh | 27 Determinants of Practice In total, 37 specific determinants of practices relevant to implementing eMh in routine care were identified. The 37 determinants were clustered resulting in a taxonomy of six groups: 1) acceptance of eMh by patients and service delivery staff, 2) appropriateness or clinical relevance of eMh, 3) engagement of participants in implementing and delivering eMh, 4) resources for implementing and delivering eMh, 5) work processes in delivering eMh, and 6) leadership in implementing and delivering eMh. Group definitions are provided in Table 3. The spider diagram in Figure 2 shows that the majority of studies reported determinants in the domain reach that were related to acceptance (n=34) and appropriateness (n=23). When categorized under RE-AIM, reach and the domain adoption were studied most often, addressing determinants related to acceptance (n=17), appropriateness (n=11), and engagement (n=10). Least investigated were the domains of implementation and maintenance.

A detailed list of the determinants is included in Table 4, including their definitions, main perspective, RE-AIM dimensions, and references to the source articles. The following subsections detail the determinants for each of the four RE-AIM domains. The perspective from which become apparent are included, differentiating between 1) patients, 2) staff (individuals and groups) involved in delivering mental health services, 3) organizations as the functional and administrative structures aimed to deliver mental health care, and 4) the system perspective as the human and material resources and organizational arrangements on a community level aimed at to preserve, protect, and restore peoples’ health [32].

More detailed information, including the related excerpts of texts retrieved from the articles, are in Appendix 2, accessible online at https://mental.jmir.org/article/downloadSuppFile/9769/66878.

28 | CHAPTER 2 Table 3. Identified groups of determinants of practice and their definitions.

Group

Definition

Determinants

Acceptance

The perception among patients, providers, organizations, and systems that eMha is agreeable, congenial, or satisfactory.

Appropriateness

The perceived fit, relevance, or compatibility of eMh for the patient in addressing his or her mental disorder.

Continuing implementing, delivering, and receiving eMh and remain doing so in the context of concrete treatment plans.

The availability and appropriateness of resources required in implementing and delivering eMh, including human resources, equipment, funding, and other infrastructural aspects.

The course of action (modus of operandi) in service delivery and all other tasks and responsibilities mental health care service organizations have.

Directing and controlling the working processes and organizing activities that enable implementation and delivery of eMh.

Access to treatment; expectations and preferences; observability and experience; evidence base; convenience; technology; awareness; skills and competences; privacy; clinical cultures; education; costs; policy; health care system structures Professional-patient interaction; effectiveness; personal need; flexibility; negative effects; safety; patient characteristics Organizational structures and procedures; leadership; staffing and roles; access and reliability of ICTb; time; collaboration

Engagement

Resources

Work processes

Leadership

aeMh: bICT:

electronic mental health interventions, or eMental health. information and communication technology.

Personnel; funds; infrastructure

Primary process; facilitating processes

Culture; communication; management; strategies and priorities; external relations

Systematic Review: Implementation of eMh | 29

Reach 64

32

16

8

4

2 Maintenance

1

Adoption

Acceptance Appropriateness Engagement Resources Work processes Implementation

Leadership

Figure 2. Spider diagram of the spread of the number of studies (N=48) categorized under the RE-AIM dimensions and the six main groups of determinants we identified in literature: acceptance, appropriateness, engagement, resources, work processes, and leadership. RE-AIM: reach, effectiveness, adoption, implementation, and maintenance

30 | CHAPTER 2 Table 4. Determinants of practice identified in the literature mapped on each reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) dimension, including their proposed definitions, main perspective, and references. Indented are determinants grouped within a group of determinants.

Cluster/Determinant

Perspective

Acceptance: the perception among patients and providers that using

eMhb

RE-AIMa

n

References

is agreeable, congenial, or satisfactory

Access to treatment: the state of accessibility and the act of accessing mental health services.

Patient

R, A

9

[33-41]

Expectations and preferences: individual and collective attitudes, expectations, and preexisting preferences about receiving and providing mental health care in general and eMh specifically.

Patient

R, A, I

1 2

[34,37,41-50]

Expectations and preferences: individual and collective attitudes, expectations, and preexisting preferences about receiving and providing mental health care in general and eMh specifically.

Staff

R, A, I, M

1 3

[43,48,51-61]

Observability and experience: the possibility and actual of observations in use (seeing or hearing about the treatment) and experiences of staff in the process of accepting eMh as a valid treatment option.

Staff

R, A, I

7

[43,5153,59,62,63]

Evidence-base: the scientific evidence of the feasibility and effectiveness of eMh.

Staff

R, A, I

3

[46,52,61]

Convenience: the comfort experienced by patients in accessing and receiving mental health care, including overcoming geographical distances, time constraints, and availability of treatment materials.

Patient

R, A, I, M

1 4

[33,34,39 42,47,59,60,62 ,64-67]

Technology: the technical aspects of eMh, including availability of and familiarity with ICT, complexity, usability, and working procedures.

Patient

R, A, M

1 1

[34,35,37,42,4 8,49,51,54,55, 66,68]

Convenience: the comfort experienced by patients in accessing and receiving mental health care, including overcoming geographical distances, time constraints, and availability of treatment materials.

Staff

R, A, I, M

8

[43,51-57]

Technology: the technical aspects of eMh, including availability of and familiarity with ICT, complexity, usability, and working procedures.

Patient

R, A, M

1 4

Technology: the technical aspects of eMh, including availability of and familiarity with ICT, complexity, usability, and working procedures.

Skills and competences: specific personal capacities and means required for receiving (patients) or providing (staff) eMh.

Staff

R, A, I, M

8

[34,37,4446,4851,59,60,6971] [43,46,51,53,6 2,63,71,72]

Patient

R, A

7

[33,39,48,51,5 4,59,73]

Systematic Review: Implementation of eMh | 31 Skills and competences: specific personal capacities and means required for receiving (patients) or providing (staff) eMh. Privacy: respecting patients’ and providers’ freedom from unauthorized intrusion, including discretion and confidentiality.

Staff

R, A, I, M

5

[48,54,55,61,6 6]

Patient

R, A

4

[35,48,49,73]

Privacy: respecting patients’ and providers’ freedom from unauthorized intrusion, including discretion and confidentiality.

Staff

R, A

1

[48]

Clinical culture: socially defined and agreed “ways of doing,” including norms, habits, and roles.

Staff

R, A, I, M

6

[43,53,60,61,6 5,67]

Education: training of staff in providing eMh in routine care, including technical and therapeutic training, formal education, credentialing, peer-group learning, and supervision.

Costs: the expenditures made to receive or provide eMh.

Staff

R, A, I

1 3

Patient

R, A, M

3

[43,46,5153,58,6163,67,71,72,74 ] [40,66,67]

Appropriateness: the perceived fit, relevance, or compatibility of eMh for the patient in addressing his or her mental disorder Professional-patient relationship: the professional interaction between (mental) health care provider and patient, including the aspects such as trust, comfort, and therapeutic interaction.

Professional-patient relationship: the professional interaction between (mental) health care provider and patient, including the aspects such as trust, comfort, and therapeutic interaction.

Effectiveness: patients’ mental health care needs, including information needs and specific (mental) health conditions.

Personal need: individual mental health care needs, including information needs and specific (mental) health conditions.

Flexibility: the extent to which care providers can alter or adapt the eMh to the (perceived) needs of the patient or care provider.

Patient

R, A, I

1 8

Staff

R, A, I

1 0

[33,35,39,40,4 2,46,48,50,54, 55,59, 6870,73,75-77] [46,52,54,55,5 7-59,61,71,77]

Patient

R

3

[33,35,40]

Patients

R, A, M

8

[33,35,42,58,5 9,65,69,75]

Staff

R, A, I, M

6

[46,58,61,67,6 9,72]

Negative effects: the perceived and actual negative (clinical) outcomes of receiving eMh.

Patient

R, A

3

[33,46,78]

Safety: the physical and mental safety of patients receiving eMh.

Patient

R

3

[35,55,78]

Safety: the physical and mental safety of patients receiving eMh.

Staff

R, A

3

[52,55,59,69]

Patient characteristics: individual patient characteristics, including age, gender, clinical history, social economic status, and clinical symptoms relevant to eMh.

Patient

R, A

7

[37,48,69,70,7 3,78,79]

32 | CHAPTER 2 Patient characteristics: individual patient characteristics, including age, gender, clinical history, social economic status, and clinical symptoms relevant to eMh.

Staff

R, A, I

4

[43,52,59,61]

Engagement: continuing implementing, delivering, and receiving eMh and remain doing so in the context of concrete treatment plans Organizational structures and procedures: the organizing Staff R, A, I 8 [43,48,52,55,5 structures, policies, and procedures for delivery of eMh, 9,61,62,72] including standards and clinical guidelines, administrative support, technical support, and other facilitating services.

Leadership: the managerial capacity and Staff R, A, I 4 [55,58,62,72] operationalization of an organization, including leadership, goal setting, strategies, and supportive measures Staffing and roles: the availability of staff necessary in Staff R, A, I, M 7 [35,48,53,59,6 delivering eMh, including qualifications, roles, and 0,62,72] responsibilities Access and reliability of ICTc: the availability, stability, and reliability of required technology, including interoperability with other existing technology (e.g., electronic patient record).

Time: the time constraints in providing mental health care in general and eMh specifically.

Staff

R, A, I

1 0

[43,48,52,53,5 6,59,62,63,71, 72]

Staff

I

1

[61]

Collaboration: the possibility and actual act of parties involved in delivery of eMh willingly work together, including sharing of information and expertise.

Staff

R, A, I

3

[61,72,77]

Resources: the availability and appropriateness of resources required in implementing and delivering eMh, including human resources, equipment, funding, and other infrastructural aspects Personnel: the availability, capacity, and capabilities of Organization A, I 2 [62,80] persons necessary in the delivering eMh. Funds: the availability and sources of pecuniary resources necessary for delivering eMh and its impact on existing (care) budgets

Organization

A, I, M

3

[66,67,72,80]

Infrastructure: availability, quality, and stability of facilitating structures required for delivering eMh, including offices and equipment.

Organization

R, A, I, M

7

[43,52,53,60,6 2,67,72]

Processes: the course of action (modus of operandi) in service delivery and all other tasks and responsibilities mental health care service organizations have Primary process: a series of actions conducing to the primary objectives of a mental health care organization such as referral processes, establishing diagnosis, and providing treatment.

Facilitating processes: the facilitating activities required for primary processes to deliver mental health care services. Facilitating processes do not directly add value to service delivery but are necessary to provide the services.

Organization

R, A, I, M

7

[43,48,53,60,6 2,67,80]

Organization

R, A, I, M

7

[43,52,60,62,6 7,72,80]

Systematic Review: Implementation of eMh | 33 Leadership: directing and controlling the working processes and organizing activities that enable implementation and delivery of eMh Culture: socially defined and agreed “ways of doing,” Organization R, A, I, M 2 [43,67] including norms, habits, and roles relevant to delivering eMh. Communication: the mechanisms, means, and contents of Organization A, I 1 [62] disseminating information across the mental health care organization.

Management: the managerial capacity and operationalization of an organization delivering eMh, including leadership, goal setting, strategies, and supportive measures.

Strategies and priorities: the operationalization of and operationalized objectives into feasible working plans, including vision, mission, priorities, and work plans.

Organization

A, I, M

3

[60,62,80]

Organization

R, A, I, M

2

[43,67]

External relations: cooperation and collaboration of various external parties involved and/or affected by delivery of eMh, including sharing knowledge.

Organization

A, I, M

3

[65,67]

Health care system: the organization of people, institutions, and resources that deliver mental health care services to meet the health needs of target populations Policy: the plans or courses of actions intended to influence and determine decisions and actions relevant to delivery of eMh.

Setting

R, A, I, M

2

[43,60]

Resources: the availability and appropriateness of resources required in delivering eMh, including HCPsd, ICT and standardization, funding, and other infrastructural aspects.

Setting

R, M

4

[60,65,70,71]

Community acceptance: the shared perception among the community that eMh is agreeable, palatable, or satisfactory.

Setting

M

2

[65,66]

Collaboration: cooperation and collaboration of various parties involved in delivery of eMh, including knowledge sharing.

Setting

R, A, I

1

[43]

Structure: the organizing and organized plan of health services in a given (geographical) context and relevant to the implementation and delivery of eMh.

Setting

M

1

[60]

aRE-AIM:

reach, effectiveness, adoption, implementation, and maintenance. Please refer to Table 1 for the specific definitions of the RE-AIM framework.

The following abbreviations are used in this column: R: reach; A: adoption; I: implementation; and M: maintenance. beMh: electronic mental health interventions. or eMental health. cICT: information and communication technology. dHCPs: health care professionals.

Reach The domain reach includes determinants of practices that are related to patients’ participation in eMh and their characteristics. Of the 33 studies that were categorized under reach, most investigated patients’ and mental HCPs perceptions and attitudes of patients and professionals (n=20), or the actual

34 | CHAPTER 2 use (n=9) of eMh in a routine care setting. Most studies were of an observational nature (n=31). Two studies used an experimental design for testing interventions aimed at increasing access and use of eMh. From the perspective of patients, two main groups of factors appeared to be relevant in implementing eMh in routine care: acceptance and appropriateness. Determinants grouped under acceptance concern the perceived and actual feasibility of interacting with eMh. For example, knowledge about the existence of eMh (awareness, n=13) and technological aspects of the treatment (e.g. usability and stability, n=10) were most often reported in the included literature.

Determinants categorized under appropriateness refers to the patients’ perceived fit, relevance, or compatibility of eMh in addressing his or her mental disorder. Within this group, the professionalpatient relationship was reported most often by both care providers and patients to be an important aspect that requires consideration when implementing eMh. For example, the perceived importance of interaction and verbal communication was highlighted by van der Vaart, et al [58], showing that the lack in nonverbal communication in Web-based treatments can pose limits to discussing more difficult issues with patients.

From the perspective of staff, engagement emerged as a group of factors next to the determinants grouped under acceptance and appropriateness. Engagement relates to the sustained and effective involvement of staff in implementing and delivering eMh for mood disorders in routine care. Most notably, engagement seem to be related to the organizing structures, policies, and procedures within an organization (n=4), as well as the availability and stability of the required information and communication technology (ICT; n=4). For example, in a qualitative study on expectations of both patients and health professionals in commencing in internet-based psychotherapy, Montero-Marin et al [48] noted the importance of standardizing Web-based interventions in an integrated service delivery model.

From the perspective of mental health service providing organizations, resources in terms of available and stability of facilitating infrastructure was mentioned (n=2) as an important determinant.

In addition, the modus operandi in service delivery both in terms of primary care processes (e.g. referral pathways, n=2) as well as facilitating processes (e.g. administrative and ICT support and billing processes, n=1) require consideration when implementing eMh in routine practice. Additionally, leadership in terms of existing cultures, strategies, and priorities emerged from the included articles as a determinant of practice (n=1). Regarding the primary care processes, Buist et al [43] showed that considering eMh as a valid service option can influence actual application. Differences in actual use might be caused by differing levels of interest and experience in the eMh service of the service managers.

At health care system level, there were three aspects reported to be of importance, namely policymaking processes (n=2), the availability of appropriate resources including qualified staff (n=2), and collaboration and cooperation within the system and across disciplines (n=1).

Systematic Review: Implementation of eMh | 35 Adoption Adoption mirrors the decision of staff and organizations involved in delivering the eMh services and the extent to which they actually use and deploy the services to their patients. Of the 19 studies that were characterized under adoption, 16 studies investigated adoption-related perceptions and attitudes toward eMh (n=9), or actual use (n=7) of eMh in routine care settings showing adoption. Three studies investigated and tested an adoption-enhancing intervention aimed at increasing the number of staff involved in the delivery of eMh. Seen from the perspective of staff delivering the services, a frequently mentioned determinant grouped under acceptance was patients’ awareness and knowledge of the existence of eMh (n=5).

Similarly, the awareness of eMh as a viable treatment option among staff was also identified as a relevant determinant in staff adopting eMh (n=6). Adoption can be facilitated by allowing clinicians to gain experience with eMh and the observability of eMh (n=7). In terms of appropriateness of eMh, the studies indicated that patient-professional relationship is an important determinant to consider when designing interventions aimed at improving adoption rates (n=7). To illustrate, May et al [54] reported on the use of videoconferencing technology in delivering psychotherapy, indicating that the therapistpatient relation should include strategies that appropriately addresses the disorder for which verbal interaction might be essential. Furthermore, the availability and stability of the technical aspects, including infrastructure and interoperability of related ICT (n=8), can be an influential factor in facilitating the engagement of professionals in continuing to offer and apply eMh to their patients.

From the organizations’ perspective, the determinants addressing adoption related mostly to the availability of infrastructural resources (n=5) and the primary care process (n=5). Infrastructural resources included the availability, quality, and stability of facilitating structures such as office rooms and ICT equipment. Determinants related to the primary care processes included issues with referral procedures, diagnostic procedures, and therapy guidelines and manuals. For instance, Jameson et al [53] highlighted that clinical policies and procedures for initiating a referral and coordinating between the various partners involved in service delivery are necessary for successful and sustainable use of eMh. One article reported determinants from a health care system perspective. Buist et al [43] reported on the importance of mechanisms that enable collaboration, sharing of information, and policies supporting better use of these mechanisms.

Implementation Determinants categorized under implementation relate to the extent to which eMh is used in realworld settings as intended (i.e. fidelity of use), implementation costs, or deliberate and purposive actions to implement eMh. Of the 6 studies identified under implementation, 2 investigated an implementation-related intervention focusing on training mental health providers to use eMh in daily practice. The other 4 studies performed a process evaluation (n=1) and investigated use and utilization of eMh (n=3).

The most frequently reported determinants from the perspective of staff were related to acceptance. These concerned raising staffs’ awareness about the existence of eMh (n=3) and providing education to staff (n-4) in applying eMh in routine care. Specific determinants included references to technical and therapeutic training, formal education and credentialing, and peer-group learning and supervision. For example, Willhelmsen et al [61] showed the importance of training of general

36 | CHAPTER 2 practitioners (GPs) in increasing patients’ acceptance of eMh, which might strengthen the perceived credibility of eMh. Furthermore, from the perspective of staff, engagement was found to be influenced by the availability of support and facilitating services (n=4). For example, Avey et al [72] reported in a qualitative study on implementation processes that coordination and collaboration between the various persons involved in the service delivery should be facilitated effectively and that a dedicated program coordinator was valued highly among the participating hospitals.

From the viewpoint of an organization, the availability of resources such as staffing (n=2), funding (n=2), and infrastructural facilities (n=2) were reported as relevant determinants. In addition, various factors emerged from the literature related to the primary modes operandi (n=3). For example, Reifels et al [80] discussed that successful implementation might depend on the existence or establishment of effective primary processes in the service delivery structures. Similarly, implementation outcomes can be determined by factors facilitating and supporting the primary processes in delivering mental health care services (n=4). Examples include issues with office space, availability of equipment, and administrative support as Adler et al [62] highlighted. Besides the organizational structures and processes, leadership and management (n=3) need to be considered when implementing eMh. This includes scheduling problems, lack of a clear goals, and managerial support to address issues with existing clinical demands.

From the perspective of health care systems, less rich information was found in the included studies. However, Buist et al [43] did report on determinants of practices relating to the availability of policy measures (n=1) and possibilities to collaborate and share knowledge within and across disciplines and settings (n=1).

Maintenance Under maintenance, determinants were categorized that relate to keeping the eMh as a normal part of routine care practices. All 4 maintenance studies were of a descriptive nature aiming to establish usage and utility figures of videoconferencing- delivered mental health services (n=2), capture end-user perceptions (n=1), or describe potential success factors (n=1) of programs that remained in practice after their implementation phase.

From the patients’ viewpoint, the convenience of eMh was seen as an important determinant in maintaining the service in practice (n=4). In an evaluation of patients’ perceptions of a routine telepsychiatry service in central Alberta, Simpson et al [66] highlighted the importance of reducing waiting times and travel time and that this in the long term might outweigh preferences for face-to-face consultations.

From the perspective of mental health staff, the clinical culture in terms of socially defined and agreed ways of doing (n=2), including norms, habits, and roles, are considered to be important in maintaining the services in routine practice. Hailey et al [65] showed that traditional patterns might keep staff from changing their practice, even if the service is in operation for a considerable time.

At the organizational level, various determinants were reported, including availability of funds (n=2) and infrastructure (n=2), the primary modes of operation (n=2), supporting structures and activities (n=2), and leadership and management (n=3). Regarding the latter, Whitten et al [67] showed in a study

Systematic Review: Implementation of eMh | 37 comparing tele-psychiatry programs that are in routine care for some time that the different business approaches these programs took might have contributed to their success.

From the perspective of the health care system, besides the importance of policy (n=1), community acceptance (n=2), and organizing and organized plans of health services (i.e. structure; n=1), the availability and appropriateness of resources required in maintaining eMh in practice were mentioned (n=2).

Discussion Principal Findings We developed a taxonomy of 37 determinants of mental health care practices known in the literature as relevant to successfully implement eMh for mood disorders. The determinants of practices clustered in six groups are expressed at (a combination of) patient, staff, organization, and setting levels and address one or more RE-AIM dimensions (see Table 3). Three determinants were reported most frequently: 1) acceptance of eMh in terms of the expectations and preferences of patients and professionals; 2) appropriateness of eMh in addressing the mental health disorder, and specifically, the therapeutic interactions mediated by eMh; and 3) the availability, stability, and reliability of required technologies, including successful interoperability with other existing technologies.

Strengths and Limitations The search strategy in this review aimed to capture as much relevant scientific literature as possible.

For this reason, broadly defined search terms were used. By applying a standardized integrative approach (RE-AIM in combination with qualitative thematic analysis), we were able to search for commonalities in the concepts and underlying study characteristics while preserving the heterogeneous nature of the data retrieved from the studies. However, and although we searched three important bibliographic databases, it is likely that important work from social scientist generalist databases was excluded.

The evidence supporting the determinants identified in this study is mostly of a descriptive nature obtained from observational studies. Due to the limited empirical evidence verifying causality of specific determinants of practices and implementation successes, the findings of this work should be interpreted with care. In an attempt to substantiate this, we conducted a quality appraisal analysis. We included a wide variety of studies ranging from observational case studies using qualitative ethnographic methods to randomized controlled trials quantitatively testing specific implementation interventions. However, because of the heterogeneity of these studies and the absence of validated instruments to assess quality, it proved impossible to come to sensible conclusions about the quality of the evidence. An elaborate approach as done by Greenhalgh et al [81,82], meta-narrative approach in developing a model of diffusion of innovations by including the research traditions from which the included studies emerged might be a fruitful approach but was beyond the scope of this review.

Comparison With Other Work Drozd et al [83] conducted a scoping review of 164 publications (including gray literature). The investigators applied the Active Implementation Framework (AIF) to identify implementation-related factors [84]. The AIF describes the components of an implementation practice, including aspects of

38 | CHAPTER 2 staff and patient selection, training, supervision, performance assessment, decision support, administrative support, system intervention, and leadership. Drozd and colleagues found in their review factors similar to those that emerged from our analysis of the literature, including certain competences of patients and professionals and organizational drivers. Regarding the latter, the authors did not find empirical support for determinants such as leadership. The authors conclude that not finding empirical evidence for organizational drivers merely indicates a gap in the implementationrelated research. Despite the low numbers (n=4), our study shows that leadership indeed is found in empirical research to be a relevant determinant in implementing eMh. This difference can perhaps be explained by the methodological choices that were made for reviewing the literature. Where Drozd and colleagues choose to follow a top-down approach (the AIF), our review followed a quantitative inductive process in identifying the topics related to implementing eMh that emerged from the included articles. Furthermore, the search strategy and data sources in light of their quality and comparability most likely influenced the results.

Similarly, Ross et al [85] updated a systematic review (of reviews, n=44) and looked at qualitative accounts of factors that influence implementation of eHealth interventions in a broader context, including somatic care. Factors identified by these researchers are comparable with the ones presented here, including complexity factors and adaptability, adding to the users’ perception of the acceptability of eHealth interventions. However, it should be noted that the concept of eHealth used by the authors included a variety of ICT-mediated health care services in four main categories: management systems, communication systems, clinical decision support systems, and information systems. In this respect, the authors did not address eHealth to contain purposed intrinsic therapeutic content aimed at improving health conditions as we did. This raises the question of whether generic eHealth both in terms of care setting (health care in general vs mental health care for mood disorders) and purpose (information sharing, support systems vs therapeutic interventions focusing on care and cure) give rise to (partial) different taxonomies of determinants of practice.

Recommendations for Implementation Practice Implementation practitioners might benefit in implementing eMh in routine care practices by taking into account the barriers and facilitators that are identified in this systematic review. Specific implementation activities can be designed and applied on the basis of these factors to achieve better implementation outcomes.

One of the most frequently mentioned barriers emerging from the literature concerns the expectations and preferences of patients and professionals about eMh services. Negative individual and collective attitudes, expectations, and existing preferences can prohibit successful implementation of eMh. Ebert et al [45] showed that providing information to patients can enhance their acceptance of eMh. In addressing expectations and preferences of mental health care staff, it is advisable to include service delivery staff in the early stages of decision making and strategy development to increase acceptance and inform concrete implementation activities aimed at the concerns of the end users.

A second important determinant of practice is related to the appropriateness of the eMh intervention in addressing the mental disorder. Within this cluster, the nature and quality of the interactions between the professional and the patient is thought to be highly influential in obtaining favorable clinical outcomes. This includes aspects such as building trust, comfort, and the quality of the

Systematic Review: Implementation of eMh | 39 therapeutic interactions. eMh interventions delivered through ICT are thought to influence these interactions negatively. Hadjistavropoulos et al [74,86] showed that specific training can change knowledge about, attitudes toward and confidence in delivering eMh. Careful development of training programs and (continuous) guidance of HCPs in applying the eMh intervention might lower barriers with perceived patient-professional interaction through eMh. In addition, innovative models of for integrating therapist support in eMh services might address issues with engagement and the patientprofessional relationship [87].

Third, the availability and reliability of required technologies is considered an important determinant for mental HCPs to remain engaged in providing eMh to their patients in routine care. This includes the interoperability with other existing technologies such as electronic health records. It seems important to ensure that the user perspective, including that of the service delivery staff, is taken into account and that the eMh service seamlessly fits within existing technologies and work processes. Here, single-sign on technology and intelligent portal designs might be fruitful avenues to explore.

Future Research To increase impact and added value of future research on implementation of eMh for mood disorders in routine practice, the following two topics should be taken into account: (1) identifying organization and system-level determinants and (2) empirical evidence on the effects of implementation strategies in addressing specific barriers and exploiting facilitating factors.

Until now, most implementation research was focused on practitioner and patient-level determinants. Service delivery takes place in a social context at micro (individuals, teams), mesa (organizations), and macro (systems) level. Knowledge about how these different contexts influence implementation efforts can facilitate further scaling up of eMh. Research on systems level might focus on the possible policy measures that enhance implementation of eMh at service deliverer level. For example, what resources at organization or health care system-level are required to deliver eMh? This can include processes of task shifting, curricula and certification of mental health staff, ICT and standardization, funding, and other infrastructural aspects. Or, what role does community acceptance have in implementing eMh in routine practice, and how can the shared perception of community as a whole be changed? Detailed knowledge of organization and setting level factors might be more likely to come from a combination of clinical psychology, social sciences, organizational psychology, and policy research. Here, the MasterMind project [88] might provide inspiration for further research on determinants of practices of eMh. Furthermore, the field would benefit from well-performed experiments designed to test implementation interventions addressing specific determinants of practices. As shown in this review, there is limited evidence on the causal relationship between determinants and implementation outcomes. Well-designed experiments studying the effects of to the local context–tailored implementation strategies might contribute to the understanding of mechanisms of implementation processes. Do, for example, educational meetings (and in what formats) contribute in raising awareness among GPs about which patient might benefit most from which eMh intervention? Or can championing an internet-based cognitive behavioral therapy service increase the adoption of other therapists in mental health care team while maintaining the flexibility therapists need to adapt parts of the treatments to the patients’ needs? Fusing implementation practices and research into natural implementation laboratories might be a valuable approach to engage in comparative effectiveness

40 | CHAPTER 2 studies of implementation interventions. In these types of studies, experimental implementation interventions can be compared with usual implementation activities for their effects on the degree of normalization of a clinical intervention in real-world service delivery settings. The ImpleMentAll project (project position paper and study protocol forthcoming) might be a good example of this approach.

This type of future research might lead to a shift from practice-based and evidence-informed to evidence-based implementation of clinically effective and relevant eMh interventions.

Conclusions This study systematically reviewed scientific literature and developed an evidence-informed taxonomy of six clusters of 37 determinants of practices we found in literature: 1) acceptance of eMh interventions among patients, providers, organizations, and health care settings; 2) appropriateness of eMh interventions in addressing the disorder; 3) engagement in implementing, delivering, and receiving eMh interventions and remain doing so; 4) the availability and appropriateness of resources for implementing and delivering eMh interventions; 5) processes relating to the modus of operandi in delivering eMh interventions; and 6) leadership directing and controlling processes and organizing activities enabling implementation and delivery of eMh interventions. On the basis of these determinants of practices, implementation-enhancing interventions can be designed, tested, and applied to achieve better implementation outcomes. Suggestions for implementation practice are discussed, such as in-depth training of professionals, careful selection, and continuous development of the eMh technology used. In addition, focal points for future research are provided, including implementation-related factors on organization and system level, as well as (quasi) experimental research to test the effectiveness of specific implementation interventions in attaining better implementation outcomes for eMh service provision.

Funding The MasterMind project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme (CIP) by the European Community (Grant Agreement number: 621000). The ImpleMentAll project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 733025. These funding bodies had no influence in the design, execution, analysis, or interpretation of the results of this study.

Acknowledgments The authors would like to thank Caroline Planting of the VU Medical Centre/GGZ inGeest for providing expert advice and help in operationalizing the search strategy for this systematic review. Furthermore, the authors would like to express gratitude to the MasterMind and the ImpleMentAll project for supporting this work.

Systematic Review: Implementation of eMh | 41

References [1] [2]

[3] [4] [5] [6]

[7]

[8] [9]

[10]

[11]

[12]

[13]

[14]

[15] [16]

Lindefors, N. & Andersson, G. (2016). In: Lindefors N, Andersson G, editors. Guided InternetBased Treatments in Psychiatry. Cham: Springer.

Andersson, G., Cuijpers, P., Carlbring, P., Riper, H., & Hedman, E. (2014). Guided internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry; 13 (3), 288-295.

Richards, D. & Richardson, T. (2012). Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev; 32 (4), 329-342.

Riper, H., Andersson, G., Christensen, H., Cuijpers, P., Lange, A., & Eysenbach, G. (2010). Theme issue on e-mental health: a growing field in internet research. J Med Internet Res; 12 (5), e74.

May, C.R., Johnson, M., & Finch, T. (2016). Implementation, context and complexity.

Implementation Science; 11 (1), 141.

Proctor, E.K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009).

Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health; 36 (1), 24-34.

Topooco, N., Riper, H., Araya, R., Berking, M., Brunn, M., Chevreul, K., et al. (2017). Attitudes towards digital treatment for depression: a European stakeholder survey. Internet Interv; 8, 19.

Wensing, M., Oxman, A., Baker, R., Godycki-Cwirko, M., Flottorp, S., Szecsenyi, J., et al. (2011).

Tailored Implementation For Chronic Diseases (TICD): a project protocol. Implement Sci; 6, 103.

Krause, J., Agarwal, S., Bodicoat, D.H., Ring, A., Shepherdm D., Rogers, S., et al. (2014).

Evaluation of a tailored intervention to improve management of overweight and obesity in primary care: study protocol of a cluster randomised controlled trial. Trials; 15 (1), 82.

Aarons, G.A. & Sommerfeld, D.H. (2012). Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. J Am Acad Child Adolesc Psychiatry; 51 (4), 423-431.

Fleuren, M.A., Paulussen, T.G., Van Dommelen, D.P., & Van Buuren, B.S. (2014). Towards a measurement instrument for determinants of innovations. Int J Qual Health Care; 26 (5), 501510.

Wensing, M., Huntink, E., van Lieshout, L.J., Godycki-Cwirko, M., Kowalczyk, A., Jäger C, et al.

(2014). Tailored implementation of evidence-based practice for patients with chronic diseases.

PLoS One; 9 (7), e101981.

Wozney, L., Newton, A.S., Gehring, N.D., Bennett, K., Huguet, A., Hartling, L., et al. (2017).

Implementation of eMental Health care: viewpoints from key informants from organizations and agencies with eHealth mandates. BMC Med Inform Decis Mak; 17 (1). 1.

Tabak, R.G., Khoong, E.C., Chambers, D.A., & Brownson. R.C. (2012). Bridging research and practice: models for dissemination and implementation research. Am J Prev Med; 43 (3), 337350.

Nilsen, P. (2015). Making sense of implementation theories, models and frameworks.

Implement Sci; 10, 53.

Glasgow, R.E., Vogt, T.M., & Boles, S.M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health; 89 (9), 1322-1327.

42 | CHAPTER 2 [17]

[18] [19] [20] [21]

[22]

[23]

[24]

[25]

[26]

[27] [28]

[29]

[30]

Glasgow, R.E., Klesges, L.M., Dzewaltowski, D.A., Estabrooks, P.A., & Vogt, T.M. (2006).

Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res; 21 (5), 688694.

Gaglio, B., Shoup, J.A., & Glasgow, R.E. (2013). The RE-AIM framework: a systematic review of use over time. Am J Public Health; 103 (6), e38-e46.

Belza, B., Toobert, D.J., & Glasgow, R.E. (2014). RE-AIM for Program Planning: Overview and Applications. In: National Council on Aging. Washington: National Council on Aging; 1-14.

Jilcott, S., Ammerman, A., Sommers, J., & Glasgow, R.E. (2007). Applying the RE-AIM framework to assess the public health impact of policy change. Ann Behav Med; 34 (2), 105-114.

Kleiboer, A., Smit, J., Bosmans, J., Ruwaard, J., Andersson, G., Topooco, N., et al. (2016).

European COMPARative Effectiveness research on blended Depression treatment versus treatment-as-usual (E-COMPARED): study protocol for a randomized controlled, non-inferiority trial in eight European countries. Trials; 17 (1), 387.

Krist, A.H., Phillips, S.M., Sabo, R.T., Balasubramanian, B.A., Heurtin-Roberts, S., Ory, M.G., & MOHR Study Group (2014). Adoption, reach, implementation, and maintenance of a behavioral and mental health assessment in primary care. Ann Fam Med;12 (6), 525-533.

Saw, A., Kim, J., Lim, J., Powell, C., & Tong, E.K. (2013). Smoking cessation counseling for Asian immigrants with serious mental illness: using RE-AIM to understand challenges and lessons learned in primary care-behavioral health integration. Health Promot Pract;14 (5), 70S-79S. Young, J., Gilwee, J., Holman, M., Messier, R., Kelly, M., & Kessler, R. (2012). Mental health, substance abuse, and health behavior intervention as part of the patient-centered medical home: a case study. Behav Med Pract Policy Res;2 (3), 345-354.

Arora, P.G., Connors, E.H., Blizzard, A., Coble, K., Gloff, N., & Pruitt D. (2017). Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example. Eval Program Plann; 60, 56-63.

Mol, M., Dozeman, E., van Schaik, D.J., Vis, C.P., Riper, H., & Smit, J.H. (2016). The therapist's role in the implementation of internet-based cognitive behavioural therapy for patients with depression: study protocol. BMC Psychiatry;16 (1), 338.

Green, S. & Higgins, J. (2008). Preparing a Cochrane review. In: Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons; 11-30.

Arai, L., Britten, N., Popay, J., Roberts, H., Petticrew, M., Rodgers, M., et al. (2007). Testing methodological developments in the conduct of narrative synthesis: a demonstration review of research on the implementation of smoke alarm interventions. Evid Policy J Res Debate Pract;3 (3), 361-383.

Mays, N., Pope, C., & Popay, J. (2005). Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy; 10(1), 6-20.

Popay, J., Roberts, H., Snowden, A., Petticrew, M., Arai, L., Rodgers, M., et al. (2006) Lancaster.ac.uk. Apr. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews: A Product from the ESRC Methods Programme URL: http://www.

Systematic Review: Implementation of eMh | 43

[31]

[32] [33]

[34]

[35]

[36]

[37]

[38]

[39] [40]

[41]

[42]

[43] [44] [45]

lancaster.ac.uk/shm/research/nssr/research/dissemination/publications/NS_Synthesis_Guida nce_v1.pdf [accessed 2018-03-09] [WebCite Cache ID 6xmbUdrz8] Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011).

Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health; 38 (2), 65-76.

Porta, M. (2014). A Dictionary of Epidemiology, Sixth edition. New York: Oxford University Press.

Beattie, A., Shaw, A., Kaur, S., & Kessler, D. (2009). Primary-care patients' expectations and experiences of online cognitive behavioural therapy for depression: a qualitative study. Health Expect;12 (1), 45-59. Included in the review.

De Las Cuevas, C., Artiles, J., De La Fuente, J., & Serrano, P. (2003). Telepsychiatry in the Canary Islands: user acceptance and satisfaction. J Telemed Telecare;9 (4), 221-224. Included in the review.

Gibson, K.L., Coulson, H., Miles, R., Kakekakekung, C., Daniels, E., & O'Donnell, S. (2011).

Conversations on telemental health: listening to remote and rural First Nations communities.

Rural Remote Health; 11 (2), 1656. Included in the review.

Graham, C., Franses, A., Kenwright, M., & Marks, I. (2018). Psychotherapy by computer: a postal survey of responders to a teletext article. Psychiatr bull; 24 (09), 331-332. Included in the review.

Handley, T.E., Kay-Lambkin, F.J., Inder, K.J., Attia, J.R., Lewin, T.J., & Kelly, B.J. (2014). Feasibility of internet-delivered mental health treatments for rural populations. Soc Psychiatry Psychiatr Epidemiol; 49 (2). 275-282. Included in the review.

Hilty, D.M., Nesbitt, T.S., Kuenneth, C.A., Cruz, G.M., & Hales, R.E. (2007). Rural versus suburban primary care needs, utilization, and satisfaction with telepsychiatric consultation. J Rural Health; 23 (2), 163-165. Included in the review.

Mucic, D. (2008). International telepsychiatry: a study of patient acceptability. J Telemed Telecare; 14 (5), 241-243. Included in the review.

Musiat, P., Goldstone, P., & Tarrier, N. (2014). Understanding the acceptability of e-mental health--attitudes and expectations towards computerised self-help treatments for mental health problems. BMC Psychiatry; 14 (1), 109. Included in the review.

Saurman, E., Lyle, D., Perkins, D., & Roberts, R. (2014). Successful provision of emergency mental health care to rural and remote New South Wales: an evaluation of the Mental Health Emergency Care-Rural Access Program. Aust Health Rev; 38 (1), 58-64. Included in the review.

Boggs, J.M., Beck, A., Felder, J.N., Dimidjian, S., Metcalf, C.A., & Segal, Z.V. (2014). Web-based intervention in mindfulness meditation for reducing residual depressive symptoms and relapse prophylaxis: a qualitative study. J Med Internet Res;16 (3), e87. Included in the review.

Buist, A., Coman, G., Silvas, A., & Burrows, G. (2000). An evaluation of the telepsychiatry programme in Victoria, Australia. J Telemed Telecare; 6 (4), 216-221. Included in the review.

Casey, L.M., Joy, A., Clough, B.A. (2013). The impact of information on attitudes toward emental health services. Cyberpsychol Behav Soc Netw;16 (8), 593-598. Included in the review.

Ebert, D.D., Berking, M., Cuijpers, P., Lehr, D., Pรถrtner, M., & Baumeister, H. (2015). Increasing the acceptance of internet-based mental health interventions in primary care patients with depressive symptoms. A randomized controlled trial. J Affect Disord; 176, 9-17. Included in the review.

44 | CHAPTER 2 [46]

[47]

[48]

[49]

[50] [51]

[52]

[53]

[54]

[55]

[56]

[57]

[58]

Jones, R.B. & Ashurst, E.J. (2013). Online anonymous discussion between service users and health professionals to ascertain stakeholder concerns in using e-health services in mental health. Health Informatics J; 19 (4), 281-299. Included in the review.

Mohr, D.C., Siddique, J., Ho, J., Duffecy, J., Jin, L., & Fokuo, J.K. (2010). Interest in behavioral and psychological treatments delivered face-to-face, by telephone, and by internet. Ann Behav Med; 40 (1), 89-98. Included in the review.

Montero-Marín, J., Prado-Abril, J., Botella, C., Mayoral-Cleries, F., Baños, R., Herrera-Mercadal, P., et al. (2015). Expectations among patients and health professionals regarding Web-based interventions for depression in primary care: a qualitative study. J Med Internet Res;17 (3), e67.

Included in the review.

Proudfoot, J., Parker, G., Hadzi, P.D., Manicavasagar, V., Adler, E., & Whitton, A. (2010).

Community attitudes to the appropriation of mobile phones for monitoring and managing depression, anxiety, and stress. J Med Internet Res; 12 (5), e64. Included in the review.

Werner, P. (2004). Willingness to use telemedicine for psychiatric care. Telemed J E Health; 10 (3), 286-293. Included in the review.

Carper, M.M., McHugh, R.K., & Barlow, D.H. (2011). The dissemination of computer-based psychological treatment: a preliminary analysis of patient and clinician perceptions. Adm Policy Ment Health; 40 (2), 87-95. Included in the review.

Gibson, K., O'Donnell, S., Coulson, H., & Kakepetum-Schultz, T. (2011). Mental health professionals' perspectives of telemental health with remote and rural First Nations communities. J Telemed Telecare; 17 (5), 263-267. Included in the review.

Jameson, J.P., Farmer, M.S., Head, K.J., Fortney, J., & Teal, C.R. (2011). VA community mental health service providers' utilization of and attitudes toward telemental health care: the gatekeeper's perspective. J Rural Health; 27 (4), 425-432. Included in the review.

May, C.R., Ellis, N.T., Atkinson, T., Gask, L., Mair, F., & Smith, C. (1999). Psychiatry by videophone: a trial service in north west England. Stud Health Technol Inform; 68, 207-210.

Included in the review.

May, C., Gask, L., Atkinson, T., Ellis, N., Mair, F., Esmail, A. (2001). Resisting and promoting new technologies in clinical practice: the case of telepsychiatry. Soc Sci Med; 52 (12), 1889-1901.

Included in the review.

Monthuy-Blanc, J., Bouchard, S., Maïano, C., & Séguin, M. (2013). Factors influencing mental health providers' intention to use telepsychotherapy in First Nations communities. Transcult Psychiatry; 50 (2), 323-343. Included in the review.

Sinclair, C., Holloway, K., Riley, G., & Auret, K. (2013). Online mental health resources in rural Australia: clinician perceptions of acceptability. J Med Internet Res; 15 (9), e193. Included in the review.

Van der Vaart, R., Witting, M., Riper, H., Kooistra, L., Bohlmeijer, E.T., & van Gemert-Pijnen, L.J. (2014). Blending online therapy into regular face-to-face therapy for depression: content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry; 14 (1), 355. Included in the review.

Systematic Review: Implementation of eMh | 45 [59]

[60]

[61]

[62] [63] [64]

[65]

[66]

[67] [68] [69]

[70]

[71] [72] [73] [74]

Whitten, P. & Kuwahara, E. (2004). A multi-phase telepsychiatry programme in Michigan: organizational factors affecting utilization and user perceptions. J Telemed Telecare; 10 (5), 254-261. Included in the review.

Ohinmaa, A., Roine, R., Hailey, D., Kuusimäki, M.L., & Winblad, I. (2008). The use of videoconferencing for mental health services in Finland. J Telemed Telecare; 14 (5), 266-270.

Included in the review.

Wilhelmsen, M., Høifødt, R.S., Kolstrup, N., Waterloo, K., Eisemann, M., Chenhall, R., et al.

(2014). Norwegian general practitioners' perspectives on implementation of a guided webbased cognitive behavioral therapy for depression: a qualitative study. J Med Internet Res; 16 (9), e208. Included in the review.

Adler, G., Pritchett, L.R., Kauth, M.R., & Nadorff, D. (2014). A pilot project to improve access to telepsychotherapy at rural clinics. Telemed J E Health; 20 (1), 83-85. Included in the review.

Austen, S. & McGrath, M. (2006). Telemental health technology in deaf and general mentalhealth services: access and use. Am Ann Deaf; 151 (3), 311-317. Included in the review.

Casey, L.M., Wright, M.A., & Clough, B.A. (2014). Comparison of perceived barriers and treatment preferences associated with internet-based and face-to-face psychological treatment of depression. Int J Cyber Behav Psychol Learn; 4 (4), 16-22. Included in the review.

Hailey, D., Ohinmaa, A., Roine, R., & Bulger, T. (2016). Uptake of telemental health services in Alberta: a success, but not in all regions. J Telemed Telecare; 13(3), 42-44. Included in the review.

Simpson, J., Doze, S., Urness, D., Hailey, D., & Jacobs, P. (2001). Telepsychiatry as a routine service--the perspective of the patient. J Telemed Telecare; 7 (3), 155-160. Included in the review.

Whitten, P., Do, C., & Kingsley, C. (2000). An analysis of telepsychiatry programs from an organizational perspective. Cyberpsychol Behav; 3 (6), 911-916. Included in the review.

Greenwood, J., Chamberlain, C., & Parker, G. (2004). Evaluation of a rural telepsychiatry service.

Australas Psychiatry; 12 (3), 268-272. Included in the review.

Morrison, C., Walker, G., Ruggeri, K., & Hughes, J.M. (2014). An implementation pilot of the MindBalance web-based intervention for depression in three IAPT services. The Cognitive Behaviour Therapist; 7, e15. Included in the review.

Rowe, N., Gibson, S., Morley, S., & Krupinski, E.A. (2008). Ten-year experience of a private nonprofit telepsychiatry service. Telemed J E Health; 14 (10), 1078-1086. Included in the review.

Simpson, J., Doze, S., Urness, D., Hailey, D., & Jacobs, P. (2001). Evaluation of a routine telepsychiatry service. J Telemed Telecare; 7 (2), 90-98. Included in the review.

Avey, J.P. & Hobbs, R.L. (2013). Dial in: fostering the use of telebehavioral health services in frontier Alaska. Psychol Serv; 10 (3), 289-297. Included in the review.

Rohland, B.M., Saleh, S.S., Rohrer, J.E., & Romitti, P.A. (2000) Acceptability of telepsychiatry to a rural population. Psychiatr Serv; 51 (5), 672-674. Included in the review.

Hadjistavropoulos, H.D., Thompson, M.J., Klein, B., & Austin, D.W. (2012). Dissemination of therapist-assisted internet cognitive behaviour therapy: development and open pilot study of a workshop. Cogn Behav Ther; 41 (3), 230-240. Included in the review.

46 | CHAPTER 2 [75]

[76]

[77] [78]

[79]

[80] [81]

[82]

[83]

[84]

[85] [86]

[87]

[88]

Callahan, E.J., Hilty, D.M., & Nesbitt, T.S. (1998). Patient satisfaction with telemedicine consultation in primary care: comparison of ratings of medical and mental health applications.

Telemed J; 4 (4), 363-369. Included in the review.

Fortney, J.C., Pyne, J.M., Mouden, S.B., Mittal, D., Hudson, T.J., Schroeder, G.W., et al. (2013).

Practice-based versus telemedicine-based collaborative care for depression in rural federally qualified health centers: a pragmatic randomized comparative effectiveness trial. Am J Psychiatry; 170 (4), 414-425. Included in the review Swinton, J.J., Robinson, W.D., & Bischoff, R.J. (2009). Telehealth and rural depression: physician and patient perspectives. Fam Syst Health; 27 (2), 172-182. Included in the review.

DeAndrea, D.C. (2015). Testing the proclaimed affordances of online support groups in a nationally representative sample of adults seeking mental health assistance. J Health Commun; 20 (2), 147-156 Included in the review.

Titov, N., Andrews, G., Kemp, A., & Robinson, E. (2010). Characteristics of adults with anxiety or depression treated at an internet clinic: comparison with a national survey and an outpatient clinic. PLoS One; 5 (5), e10885. Included in the review.

Reifels, L., Bassilios, B., King, K.E., Fletcher, J.R., Blashki, G., & Pirkis, J.E. (2013). Innovations in primary mental healthcare. Aust Health Rev; 37 (3), 312-317. Included in the review.

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q; 82 (4), 581-629.

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., Kyriakidou, O., & Peacock R. (2005).

Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med; 61 (2), 417-430.

Drozd, F., Vaskinn, L., Bergsund, H.B., Haga, S.M., Slinning, K., & Bjørkli, C.A. (2016). The implementation of internet interventions for depression: a scoping review. J Med Internet Res; 18 (9), e236.

Fixsen, D.L., Naoom, S.F., Blase, K.A., Friedman, R.M., & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, Florida, USA: National Implementation Research Network, FMHI Publication; 231, 1-119.

Ross, J,. Stevenson, F., Lau, R., & Murray, E. (2016). Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci; 11 (1), 146.

Hadjistavropoulos, H.D., Pugh, N.E., Nugent, M.M., Hesser, H., Andersson, G., Ivanov, M., et al.

(2014). Therapist-assisted internet-delivered cognitive behavior therapy for depression and anxiety: translating evidence into clinical practice. J Anxiety Disord; 28 (8), 884-893.

Schueller, S.M., Tomasino, K.N., & Mohr, D.C. (2016). Integrating human support into behavioral intervention technologies: the efficiency model of support. Clin Psychol Sci Pract; 24 (1), 27-45.

Vis, C., Kleiboer, A., Prior, R., Bønes, E., Cavallo, M., Clark, S.A., et al. (2015). Implementing and up-scaling evidence-based eMental health in Europe: the study protocol for the MasterMind project. Internet Interv; 2 (4), 399-409.

Systematic Review: Implementation of eMh | 47 Appendix 1: Benchmark definitions, RE-AIM definitions, and search strings Benchmark definitions 

Implementation: "the active and planned efforts to mainstream an innovation within an organization” [1]. In terms of institutionalization or normalization, mainstreaming an innovation can be understood as the set of activities to intentionally embed and integrate practices into their (social) contexts” [2]. Implementation on individual level includes the "decision to make full use of the innovation as the best course of action available” [3], and on organizational level, the means to adapt or assimilate the innovation to the environment in which it is implemented [4]. Assimilation in that sense includes the decision-making and the process that might lead to "the innovation's full acceptance, utilization and institutionalization” [5].



Innovation: "An innovation is an idea, practice or object that is perceived as new by an individual or other unit of adoption.” [3]. In line with Greenhalgh et al.'s deliberation about the concept of innovation in health services [6], the idea, practice or object include behaviors, routines and ways of working, and associated technologies and systems. The new practice should be aimed at improving health outcomes, efficiency, cost-effectiveness, or experience. The object regarded as 'new' is interpreted in relation to previous practices, and should be determined so by a proportion of key stakeholders involved in that practice.



Mental healthcare routine practice: organized healthcare by which persons receive services for mental health problems. These services can differ from screening, diagnose, treatment to relapse prevention. Focus on common mental health problems: depression and anxiety?



eMental health: “E-mental health encompasses the use of digital technologies and new media for the delivery of screening, health promotion, prevention, early intervention, treatment, or relapse prevention as well as for improvement of health care delivery (e.g. electronic patient files), professional education (e-learning), and online research in the field of mental health.” [7].

RE-AIM definitions 

REACH (patient level): Reach refers to the percentage and risk characteristics of persons (clients, patients) who receive or are affected by a policy or program. It includes both a ratio of participation and characteristics of participants to understand "the degree to which a program reaches those in need” [8]. Reach is also operationalized in terms of the program's ability to "attract its intended audience” [9].



EFFECTIVENESS (patient level): Defined as the impact of an intervention on outcomes, including potential negative effects, quality of life, and economic outcomes [10]. This dimension is not addressed in current review



ADOPTION (setting and therapist level): Adoption refers to the proportion and representativeness of settings (e.g. clinics, work sites etc.) or staff (therapists etc.) to adopt a given policy or program [8]. It is defined as "the absolute number, proportion, and representativeness of settings (e.g., health departments) and interventionists (e.g., nurses, educators) who deliver a program” [11].

 

IMPLEMENTATION (setting level): Implementation is also referred to as "intervention fidelity" to the various elements of an intervention’ [9] and is defined in literature as "intervention agents'

48 | CHAPTER 2 fidelity to the intervention's protocol" and includes consistency of delivery and cost of the intervention [10]. 

MAINTENANCE (setting level): Maintenance (at the setting level) is the extent to which a program or policy "becomes institutionalized or part of the routine organizational practices and policies” [10]. In that sense it also is referred to as the sustainability of an intervention in routine practice.

Search strings Pubmed "Health Plan implementation"[Mesh] OR "Diffusion of Innovation"[Mesh] OR "Program Evaluation"[Mesh] OR “Nursing Evaluation Research”[Mesh] OR “Health care reform”[Mesh] OR "Delivery of Health Care"[Mesh] OR "Organizational Innovation"[Mesh] OR "Information Dissemination"[Mesh] OR "Translational Medical Research"[Mesh] OR implement[tiab] OR implementation[tiab] OR implemented[tiab] OR implementing[tiab] OR diffuse[tiab] OR diffused[tiab] OR diffusion[tiab] OR disseminate[tiab] OR disseminated[tiab] OR disseminating[tiab] OR dissemination[tiab] OR upscale[tiab] OR up-scale[tiab] OR uptake[tiab] OR up-take[tiab] OR translation[tiab] OR translational[tiab] OR program evaluation[tiab] OR process evaluation[tiab] OR normalisation[tiab] OR normalise[tiab] OR normalised[tiab] OR normalising[tiab] OR normalization[tiab] OR normalize[tiab] OR normalized[tiab] OR normalizing[tiab] OR mainstream[tiab] OR mainstreamed[tiab] OR mainstreaming[tiab] OR maintainance[tiab] OR maintained[tiab] OR sustain[tiab] OR sustainability[tiab] OR sustainable[tiab] OR sustained[tiab] OR integrate[tiab] OR integrated[tiab] OR integrating[tiab] OR integration[tiab] OR nursing evaluation research [tiab] OR health care reform [tiab] OR delivery of health care [tiab] OR delivery of healthcare [tiab] OR organizational innovation [tiab] OR implement[ot] OR implementation[ot] OR implemented[ot] OR implementing[ot] OR diffuse[ot] OR diffused[ot] OR diffusion[ot] OR disseminate[ot] OR disseminated[ot] OR disseminating[ot] OR dissemination[ot] ORupscale[ot] OR uptake[ot] OR uptake[ot] OR translation[ot] OR translational[ot] OR program evaluation[ot] OR process evaluation[ot] OR normalisation[ot] OR normalised[ot] OR normalization[ot] OR normalize[ot] OR normalized[ot] OR normalizing[ot] OR mainstream[ot] OR mainstreaming[ot] OR maintainance[ot] OR maintained[ot] OR sustain[ot] OR sustainability[ot] OR sustainable[ot] OR sustained[ot] OR integrate[ot] OR integrated[ot] OR integrating[ot] OR integration[ot] OR nursing evaluation research [OT] OR health care reform [OT] OR delivery of health care [OT] OR delivery of healthcare [OT] OR organizational innovation [OT] AND "Mental Health Services"[Mesh] OR "Inpatients"[Mesh] OR "Outpatients"[Mesh] OR "Community Psychiatry"[Mesh] OR "Hospitalization"[Mesh] OR mental health service[tiab] OR mental health services[tiab] OR mental healthcare[tiab] OR mental health care[tiab] OR mental practice[tiab] OR mental practices[tiab] OR mental service delivery[tiab] OR inpatient care[tiab] OR in-patient care[tiab] OR out-patient care[tiab] OR outpatient care[tiab] OR mental health program[tiab] OR mental health programme[tiab] OR mental health programmes[tiab] OR mental health programs[tiab] OR community

Systematic Review: Implementation of eMh | 49 psychiatry[tiab] OR community psychology[tiab] OR community based[tiab] OR mental health service[ot] OR mental health services[ot] OR mental healthcare[ot] OR mental health care[ot] OR mental practice[ot] OR inpatient care[ot] OR out-patient care[ot] OR outpatient care[ot] OR mental health program[ot] OR mental health programme[ot] OR mental health programs[ot] OR community psychiatry[ot] OR community psychology[ot] OR community based[ot] OR "Depression"[Mesh] OR "Depressive Disorder"[Mesh] OR "Mood Disorders"[Mesh] OR "Bipolar Disorder"[Mesh] OR mood disorder[tiab] OR mood disorders[tiab] OR depressed[tiab] OR depressing[tiab] OR depression[tiab] OR depressions[tiab] OR depressive[tiab] OR bipolar[tiab] OR mania[tiab] OR manic[tiab] OR mood disorder[ot] OR mood disorders[ot] OR depressed[ot] OR depressing[ot] OR depression[ot] OR depressions[ot] OR depressive[ot] OR bipolar[ot] OR mania[ot] OR manic[ot] AND "Telecommunications"[Mesh] OR "Computer Systems"[Mesh] OR "Mobile Applications"[Mesh] OR telehealth[tiab] OR tele-health[tiab] OR telemedicine[tiab] OR telemedicine[tiab] OR e-health[tiab] OR ehealth[tiab] OR emental[tiab] OR e-mental[tiab] OR mhealth[tiab] OR m-health[tiab] OR web-based intervention[tiab] OR web-based interventions[tiab] OR web-based treatment[tiab] OR web-based treatments[tiab] OR webbased therapy[tiab] OR web-based therapies[tiab] OR internet intervention[tiab] OR internet interventions[tiab] OR internet treatment[tiab] OR internet treatments[tiab] OR internet therapy[tiab] OR internet therapies[tiab] OR internet-based intervention[tiab] OR internetbased interventions[tiab] OR internet-based treatment[tiab] OR internetbased treatments[tiab] OR internet-based therapy[tiab] OR internet-based therapies[tiab] OR online intervention[tiab] OR online interventions[tiab] OR online treatment[tiab] OR online treatments[tiab] OR online therapy[tiab] OR online therapies[tiab] OR computer-based intervention[tiab] OR computerbased interventions[tiab] OR computer-based treatment[tiab] OR computer-based treatments[tiab] OR computer-based therapy[tiab] OR phone-based intervention[tiab] OR phone-based interventions[tiab] OR phone-based treatment[tiab] OR telephone-based intervention[tiab] OR telephone-based interventions[tiab] OR telephone-based treatment[tiab] OR smartphone-based intervention[tiab] OR smartphone-based interventions[tiab] OR guided[tiab] OR unguided[tiab] OR blended[tiab] OR computer mediated[tiab] OR computer assisted[tiab] OR computer augmented[tiab] OR econsult[tiab] OR econsultation[tiab] OR econsultations[tiab] OR e-consult[tiab] OR e-consultation[tiab] OR econsultations[tiab] OR remote consultation[tiab] OR remote consultations[tiab] OR teleconsult[tiab] ORteleconsultation[tiab] OR teleconsultations[tiab] OR tele-consult[tiab] OR teleconsultation[tiab] OR tele-consultations[tiab] OR telemonitoring[tiab] OR telemonitoring[tiab] OR electronic communication[tiab] OR electronic communications[tiab] OR electronic mail[tiab] OR email[tiab] OR email[tiab] OR emailing[tiab] OR e-mailing[tiab] OR telecommunication[tiab] OR telecommunications[tiab] OR tele-communication[tiab] OR telecommunications[tiab] OR teleconference[tiab] OR teleconferences[tiab] OR teleconferencing[tiab] OR tele-conference[tiab] OR tele-conferences[tiab] OR teleconferencing[tiab] OR videoconference[tiab] OR videoconferences[tiab] OR videoconferencing[tiab] OR video-conference[tiab] OR video-conferences[tiab] OR telehealth[ot] OR tele-health[ot] OR telemedicine[ot] OR tele-medicine[ot] OR e-health[ot] OR ehealth[ot] OR emental[ot] OR e-mental[ot] OR mhealth[ot] OR m-health[ot] OR web-based intervention[ot] OR webbased interventions[ot] OR web-based treatment[ot] OR web-based treatments[ot] OR web-based

50 | CHAPTER 2 therapy[ot] OR web-based therapies[ot] OR internet intervention[ot] OR internet interventions[ot] OR internet treatment[ot] OR internet therapy[ot] OR internet therapies[ot] OR internet-based intervention[ot] OR internet-based interventions[ot] OR internet-based treatment[ot] OR internetbased treatments[ot] OR internet-based therapy[ot] OR online intervention[ot] OR online interventions[ot] OR online treatment[ot] OR online therapy[ot] OR computer-based intervention[ot] OR computer-based interventions[ot] OR computer-based therapy[ot] OR telephone-based intervention[ot] OR guided[ot] OR unguided[ot] OR blended[ot] OR computer mediated[ot] OR computer assisted[ot] OR computer augmented[ot] OR econsultation[ot] OR e-consultation[ot] OR remote consultation[ot] OR teleconsultation[ot] OR tele-consultation[ot] OR telemonitoring[ot] OR tele-monitoring[ot] OR electronic communication[ot] OR electronic mail[ot] OR email[ot] OR e-mail[ot] OR telecommunication[ot] OR telecommunications[ot] OR teleconference[ot] OR teleconferences[ot] OR teleconferencing[ot] OR videoconference[ot] OR videoconferencing[ot] OR video-conference[ot] Psychinfo DE "Mental Health Program Evaluation" OR DE "Program Development" OR DE "Program Evaluation" OR "Innovation" OR DE "Health Maintenance Organization" OR DE "Utilization Reviews" OR DE "Health Care Delivery" OR DE "Health Care Utilization" OR DE "Information Dissemination" OR DE "Mainstreaming" OR DE “Health Care Reform” OR TI implement OR TI implementation OR TI implemented OR TI implementing OR TI diffuse OR TI diffusion OR TI diffused OR TI disseminate OR TI disseminated OR TI disseminating OR TI dissemination OR TI upscale OR TI up-scale OR TI uptake OR TI up-take OR TI translational OR TI translation OR TI program evaluation OR TI process evaluation OR TI program development OR TI normalization OR TI normalisation OR TI normalized OR TI normalised OR TI normalise OR TI normalize OR TI normalising OR TI normalizing OR TI mainstream OR TI mainstreamed OR TI mainstreaming OR TI maintained OR TI maintenance OR TI sustain OR TI sustainable OR TI sustained OR TI sustainability OR TI integration OR TI integrated OR TI integrate OR TI integrating OR TI ‘organizational innovation’ OR TI ‘utilization reviews’ OR TI delivery of healthcare OR TI delivery of health care OR AB implement OR AB implementation OR AB implemented OR AB implementing OR AB diffuse OR AB diffusion OR AB diffused OR AB disseminate OR AB disseminated OR AB disseminating OR AB dissemination OR AB upscale OR AB up-scale OR AB uptake OR AB up-take OR AB translational OR AB translation OR AB program evaluation OR AB process evaluation OR AB program development OR AB normalization OR AB normalisation OR AB normalized OR AB normalised OR AB normalise OR AB normalize OR AB normalising OR AB normalizing OR AB mainstream OR AB mainstreamed OR AB mainstreaming OR AB maintained OR AB maintenance OR AB sustain OR AB sustainable OR AB sustained OR AB sustainability OR AB integration OR AB integratedOR AB integrate OR AB integrating OR AB ‘organizational innovation’ OR AB ‘utilization reviews’ OR AB ‘delivery of healthcare’ OR AB ‘delivery of health care’ AND DE "Community Mental Health Services" OR DE "Mental Health Services" OR DE "Community Mental Health" OR DE "Community Mental Health Centers" OR DE "Community Psychiatry" OR DE "Psychiatric

Systematic Review: Implementation of eMh | 51 Units" OR DE "Partial Hospitalization" OR DE "Hospitalization" OR DE "Psychiatric Hospitalization" OR TI mental health service OR TI mental health services OR TI mental healthcare OR TI mental health care OR TI mental practice OR TI mental service delivery OR TI mental health center OR TI mental health centers OR TI inpatient care OR TI in-patient care OR TI out-patient care OR TI outpatient care OR TI mental health program OR TI mental health programs OR TI community psychiatry OR TI community psychology OR TI community based OR AB mental health service OR AB mental health services OR AB mental healthcare OR AB mental health care OR AB mental practice OR AB mental service delivery OR AB mental health center OR AB mental health centers OR AB inpatient care OR AB in-patient care OR AB out-patient care OR AB outpatient care OR AB mental health program OR AB mental health programs OR AB community psychiatry OR AB community psychology OR AB community based OR DE "Anaclitic Depression" OR DE "Dysthymic Disorder" OR DE "Endogenous Depression" OR DE "Postpartum Depression" OR DE "Reactive Depression" OR DE "Recurrent Depression" OR DE "Treatment Resistant Depression" OR DE "Depression (Emotion)" OR DE "Bipolar Disorder" OR DE "Major Depression" OR DE "Mania" OR DE “Affective disorders� OR TI mood disorder OR TI mood disorders OR TI affective disorder OR TI affective disorders OR TI depression OR TI depressed OR TI depressive OR TI bipolar OR TI mania OR TI manic OR AB mood disorder OR AB mood disorders OR AB affective disorder OR AB affective disorders OR AB depression OR AB depressed OR AB depressive OR AB bipolar OR AB mania OR AB manic AND DE "Telemedicine" OR DE "Internet" OR DE "Electronic Communication" OR DE "Teleconferencing" OR DE "Mobile Devices" OR DE "Cellular Phones" OR TI telehealth OR TI tele-health OR TI telemedicine OR TI tele-medicine OR TI e-health OR TI ehealth OR TI emental OR TI e-mental OR TI mhealth OR TI mhealth OR TI web-based intervention OR TI web-based interventions OR TI web-based treatment OR TI web-based treatments OR TI webbased therapy OR TI web-based therapies OR TI internet intervention OR TI internet interventions OR TI internet treatment OR TI internet treatments OR TI internet therapy OR TI internet therapies OR TI internet-based intervention OR TI internet-based interventions OR TI internet-based treatment OR TI internet-based treatments OR TI internet-based therapy OR TI internetbased therapies OR TI online intervention OR TI online interventions OR TI online treatment OR TI online treatments OR TI online therapy OR TI online therapies OR TI computer-based intervention OR TI computer-based interventions OR TI computerbased treatment OR TI computer-based treatments OR TI computer-based therapy OR TI phone-based intervention OR TI phone-based interventions OR TI phone-based treatment OR TI telephone-based intervention OR TI telephone-based interventions OR TI telephone-based treatment OR TI smartphone-based intervention OR TI smartphone-based interventions OR TI guided OR TI unguided OR TI blended OR TI computer mediated OR TI computer assisted OR TI computer augmented OR TI econsult OR TI econsultation OR TI econsultations OR TI econsult OR TI e-consultation OR TI e-consultations OR TI remote consultation OR TI remote consultations OR TI teleconsult OR TI teleconsultation OR TI teleconsultations OR TI teleconsult OR TI tele-consultation OR TI tele-consultations OR TI telemonitoring OR TI telemonitoring OR TI electronic communication OR TI electronic communications OR TI electronic mail OR TI email OR TI e-mail OR TI emailing OR TI e-mailing OR TI telecommunication OR TI telecommunications OR TI tele-communication OR TI tele-communications OR TI teleconference OR TI teleconferences OR TI teleconferencing OR TI

52 | CHAPTER 2 tele-conference OR TI tele-conferences OR TI tele-conferencing OR TI videoconference OR TI videoconferences OR TI videoconferencing OR TI video-conference OR TI video-conferences OR AB telehealth OR AB tele-health OR AB telemedicine OR AB tele-medicine OR AB e-health OR AB ehealth OR AB emental OR AB e-mental OR AB mhealth OR AB m-health OR AB webbased intervention OR AB web-based interventions OR AB web-based treatment OR AB webbased treatments OR AB web-based therapy OR AB web-based therapies OR AB internet intervention OR AB internet interventions OR AB internet treatment OR AB internet treatments OR AB internet therapy OR AB internet therapies OR AB internet-based intervention OR AB internet-based interventions OR AB internet-based treatment OR AB internet-based treatments OR AB internet-based therapy OR AB internet-based therapies OR AB online intervention OR AB online interventions OR AB online treatment OR AB online treatments OR AB online therapy OR AB online therapies OR AB computer-based intervention OR AB computer-based interventions OR AB computer-based treatment OR AB computerbased treatments OR AB computerbased therapy OR AB phone-based intervention OR AB phone-based interventions OR AB phone-based treatment OR AB telephone-based intervention OR AB telephone-based interventions OR AB telephone-based treatment OR AB smartphone-based intervention OR AB smartphone-based interventions OR AB guided OR AB unguided OR AB blended OR AB computer mediated OR AB computer assisted OR AB computer augmented OR AB econsult OR AB econsultation OR AB econsultations OR AB econsult OR AB e-consultation OR AB e-consultations OR AB remote consultation OR AB remote consultations OR AB teleconsult OR AB teleconsultation OR AB teleconsultations OR AB tele-consult OR AB tele-consultation OR AB tele-consultations OR AB telemonitoring OR AB telemonitoring OR AB electronic communication OR AB electronic communications OR AB electronic mail OR AB email OR AB e-mail OR AB emailing OR AB e-mailing OR AB telecommunication OR AB telecommunications OR AB tele-communication OR AB telecommunications OR AB teleconference OR AB teleconferences OR AB teleconferencing OR AB tele-conference OR AB tele-conferences OR AB teleconferencing OR AB videoconference OR AB videoconferences OR AB videoconferencing OR AB videoconference OR AB videoconferences Embase 'health care planning'/exp OR 'program evaluation'/exp OR 'nursing evaluation research'/exp OR 'health care policy'/exp OR 'health care delivery'/exp OR 'information dissemination'/exp OR 'translational research'/exp OR implement:ab,ti OR implementation:ab,ti OR implemented:ab,ti OR implementing:ab,ti OR diffuse:ab,ti OR diffused:ab,ti OR diffusion:ab,ti OR disseminate:ab,ti OR disseminated:ab,ti OR disseminating:ab,ti OR dissemination:ab,ti OR upscale:ab,ti OR up-scale:ab,ti OR uptake:ab,ti OR up-take:ab,ti OR translation:ab,ti OR translational:ab,ti OR (program NEAR/2 evaluation):ab,ti OR (process NEAR/2 evaluation):ab,ti OR normalisation:ab,ti OR normalise:ab,ti OR normalised:ab,ti OR normalising:ab,ti OR normalization:ab,ti OR normalize:ab,ti OR normalized:ab,ti OR normalizing:ab,ti OR mainstream:ab,ti OR mainstreamed:ab,ti OR mainstreaming:ab,ti OR maintainance:ab,ti OR maintained:ab,ti OR sustain:ab,ti OR sustainability:ab,ti OR sustainable:ab,ti OR sustained:ab,ti OR integrate:ab,ti OR integrated:ab,ti OR integrating:ab,ti OR integration:ab,ti OR (nursing NEAR/2 evaluation NEAR/2 research):ab,ti OR (health NEAR/1 care NEAR/2 reform):ab,ti OR (delivery NEAR/2 health NEAR/3 care):ab,ti OR (delivery NEAR/2 healthcare):ab,ti OR (organizational

Systematic Review: Implementation of eMh | 53 NEAR/2 innovation):ab,ti OR (health NEAR/1 care NEAR/2 planning):ab,ti OR (diffusion NEAR/2 innovation):ab,ti OR (health NEAR/1 care NEAR/2 policy):ab,ti AND 'mental health service'/exp OR 'hospital patient'/exp OR 'outpatient'/exp OR 'social psychiatry'/exp OR 'hospitalization'/exp OR (mental NEAR/1 health NEAR/2 service):ab,ti OR (mental NEAR/1 health NEAR/2 services):ab,ti OR (mental NEAR/1 healthcare):ab,ti OR (mental NEAR/1 health NEAR/1 care):ab,ti OR (mental NEAR/2 practice):ab,ti OR (mental NEAR/2 practices):ab,ti OR (mental NEAR/2 service NEAR/2 delivery):ab,ti OR (inpatient NEAR/1 care):ab,ti OR (in-patient NEAR/1 care):ab,ti OR (out-patient NEAR/1 care):ab,ti OR (outpatient NEAR/1 care):ab,ti OR (mental NEAR/1 health NEAR/2 program):ab,ti OR (mental NEAR/1 health NEAR/2 programme):ab,ti OR (mental NEAR/1 health NEAR/2 programmes):ab,ti OR (mental NEAR/1 health NEAR/2 programs):ab,ti OR (community NEAR/2 psychiatry):ab,ti OR (community NEAR/2 psychology):ab,ti OR (community NEAR/1 based):ab,ti OR (hospital NEAR/1 patient):ab,ti OR (social NEAR/2 psychiatry):ab,ti OR 'mood disorder'/exp OR (mood NEAR/2 disorder):ab,ti OR (mood NEAR/2 disorders):ab,ti OR depressed:ab,ti OR depressing:ab,ti OR depression:ab,ti OR depressions:ab,ti OR depressive:ab,ti OR bipolar:ab,ti OR mania:ab,ti OR manic:ab,ti AND 'telecommunication'/exp OR 'computer system'/exp OR 'mobile application'/exp OR telehealth:ab,ti OR tele-health:ab,ti OR telemedicine:ab,ti OR tele-medicine:ab,ti OR ehealth:ab,ti OR ehealth:ab,ti OR emental:ab,ti OR e-mental:ab,ti OR mhealth:ab,ti OR mhealth:ab,ti OR (web-based NEAR/2 intervention):ab,ti OR (web-based NEAR/2 interventions):ab,ti OR (web-based NEAR/2 treatment):ab,ti OR (web-based NEAR/2 treatments):ab,ti OR (web-based NEAR/2 therapy):ab,ti OR (web-based NEAR/2 therapies):ab,ti OR (internet NEAR/2 intervention):ab,ti OR (internet NEAR/2 interventions):ab,ti OR (internet NEAR/2 treatment):ab,ti OR (internet NEAR/2 treatments):ab,ti OR (internet NEAR/2 therapy):ab,ti OR (internet NEAR/2 therapies):ab,ti OR (internet-based NEAR/2 intervention):ab,ti OR (internet-based NEAR/2 interventions):ab,ti OR (internet-based NEAR/2 treatment):ab,ti OR (internet-based NEAR/2 treatments):ab,ti OR (internet-based NEAR/2 therapy):ab,ti OR (internet-based NEAR/2 therapies):ab,ti OR (online NEAR/2 intervention):ab,ti OR (online NEAR/2 interventions):ab,ti OR (online NEAR/2 treatment):ab,ti OR (online NEAR/2 treatments):ab,ti OR (online NEAR/2 therapy):ab,ti OR (online NEAR/2 therapies):ab,ti OR (computerbased NEAR/2 intervention):ab,ti OR (computer-based NEAR/2 interventions):ab,ti OR (computerbased NEAR/2 treatment):ab,ti OR (computer-based NEAR/2 treatments):ab,ti OR (computerbased NEAR/2 therapy):ab,ti OR (phone-based NEAR/2 intervention):ab,ti OR (phone-based NEAR/2 interventions):ab,ti OR (phone-based NEAR/2 treatment):ab,ti OR (telephone-based NEAR/2 intervention):ab,ti OR (telephone-based NEAR/2 interventions):ab,ti OR (telephonebased NEAR/2 treatment):ab,ti OR (smartphone-based NEAR/2 intervention):ab,ti OR (smartphone-based NEAR/2 interventions):ab,ti OR guided:ab,ti OR unguided:ab,ti OR blended:ab,ti OR (computer NEAR/2 mediated):ab,ti OR (computer NEAR/2 assisted):ab,ti OR (computer NEAR/2 augmented):ab,ti OR econsult:ab,ti OR econsultation:ab,ti OR econsultations:ab,ti OR e-consult:ab,ti OR e-consultation:ab,ti

54 | CHAPTER 2 OR e-consultations:ab,ti OR (remote NEAR/2 consultation):ab,ti OR (remote NEAR/2 consultations):ab,ti OR teleconsult:ab,ti OR teleconsultation:ab,ti OR teleconsultations:ab,ti OR teleconsult:ab,ti OR tele-consultation:ab,ti OR tele-consultations:ab,ti OR telemonitoring:ab,ti OR telemonitoring:ab,ti OR (electronic NEAR/2 communication):ab,ti OR (electronic NEAR/2 communications):ab,ti OR (electronic NEAR/2 mail):ab,ti OR email:ab,ti OR e-mail:ab,ti OR emailing:ab,ti OR e-mailing:ab,ti OR telecommunication:ab,ti OR telecommunications:ab,ti OR telecommunication:ab,ti OR tele-communications:ab,ti OR teleconference:ab,ti OR teleconferences:ab,ti OR teleconferencing:ab,ti OR tele-conference:ab,ti OR teleconferences:ab,ti OR tele-conferencing:ab,ti OR videoconference:ab,ti OR videoconferences:ab,ti OR videoconferencing:ab,ti OR videoconference:ab,ti OR videoconferences:ab,ti

Systematic Review: Implementation of eMh | 55

References [1]

[2] [3] [4]

[5] [6] [7]

[8]

[9] [10] [11]

Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. The Milbank Quarterly. 82(4), 581-629.

May, C., & Finch, T. (2009). Implementing, Embedding, and Integrating Practices: An Outline of Normalization Process Theory. Sociology. 43(3), 535-554.

Rogers, E. M. (2003). Diffusion of Innovations, 5th Edition. New York: Simon & Schuster.

Damanpour, F., & Gopalakrishnan, S. (1998). Theories of organizational structure and innovation adoption: the role of environmental change. Journal of Engineering and Technology Management. 15(1), 1-24.

Meyer, A. D., & Goes, J. B. (1988). Organizational Assimilation of Innovations: A Multilevel Contextual Analysis. The Academy of Management Journal. 31(4), 897-923.

Greenhalgh, T., Robert, G., Bate, P., Macfarlane, F., & Kyriakidou, O. (2005). Diffusion of Innovations in Health Service Organisations. Malden, MA: Blackwell Publishing.

Riper, H., Christensen, H., Cuijpers, P., Lange, A., & Eysenbach, G. (2010). Theme Issue on EMental Health: A Growing Field in Internet Research. Journal of Medical Internet Research.

12(5), e74.

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health. 89(9), 1322-1327.

Belza, B., Toobert, D., & Glasgow, R. E. (2014). RE-AIM for Program Planning: Overview and Applications. Washington, DC: National Council on Aging, 165.

Gaglio, B., Shoup, J. A., & Glasgow, R. E. (2013). The RE-AIM Framework: A Systematic Review of Use Over Time. American Journal of Public Health. 103(6), e38-e46.

Jilcott, S., Ammerman, A., Sommers, J., & Glasgow, R. E. (2007). Applying the REAIM framework to assess the public health impact of policy change. Annals of Behavioral Medicine: A Publication of the Society of Behavioral Medicine. 34(2), 105-114.

56 | CHAPTER 3

CHAPTER 3

I think that iCBT demands organizational investment to set it up properly, a secure environment, it must fit into existing processes, therapists must be motivated and comfortable to the new method.

And we need good guidelines for when to do it and when to use it Therapist Ella

Study Protocol: Therapists’s Role| 57

The Therapist’s Role in the Implementation of Internet-based Cognitive Behavioral Therapy for Patients with Depression: Study Protocol Mayke Mol, Els Dozeman, Anneke van Schaik, Christiaan Vis, Heleen Riper & Jan Smit BMC Psychiatry 2016

3

58 | CHAPTER 3

Abstract

Background: internet-based Cognitive Behavioral Therapy (iCBT) for the treatment of depressive disorders is innovative and promising. Various studies have demonstrated large effect sizes up to 2.27, but implementation in routine practice lags behind. Mental health therapists play a significant role in the uptake of internet-based interventions. Therefore, it is interesting to study factors that influence the therapists in whether they apply internet-based therapy or not.

Objective: This study, as part of the European implementation project MasterMind, aims to identify the factors that promote or hinder therapists in the use of iCBT in depression care.

Methods: The uptake of iCBT by therapists in routine mental health care practice for the treatment of depression will be evaluated by a mixed method approach, to provide an understanding of the implementation factors (quantitative), and to ascertain the facilitating and hindering factors in the involvement of therapists in the implementation of iCBT (qualitative). The involvement of therapists in the implementation of iCBT is analyzed following the RE-AIM framework on the five dimensions Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance. This enables us to evaluate the reach of therapists, the impact of iCBT on depression care, the extent to which therapists adopt iCBT, the extent to which iCBT is delivered as intended, and how iCBT can be maintained over time.

Conclusions: The results will provide valuable insight into the role of therapists in the implementation of iCBT for depression in secondary mental health care settings. They will result in concrete recommendations for how therapists can be facilitated in implementing and up-scaling iCBT for depression.

Keywords: Implementation, Therapist’s role, E-mental health, Routine practice, Depression, internetbased cognitive behavioral therapy, Online treatment, Blended treatment

Study Protocol: Therapists’s Role| 59

Introduction Background Despite its potential advantages and the evidence for its effectiveness, the implementation of internetbased Cognitive Behavioral Therapy (iCBT) for depression [1] in routine mental health care is lower than expected. Since 2000 more than 100 trials on iCBT have been conducted for various clinical disorders, such as depression, anxiety and chronic pain [2]. Most of the included studies focused on clinical efficacy, and a small minority on cost effectiveness. The results from these various studies are promising, with large within-group effect sizes in the treatment of depression ranging from 0.38 to 2.27, with a mean of 0.94. Treatment with iCBT showed equivalent effects compared to conventional CBT. However, the use of iCBT is not scaled up to the extent that it actually helps to reduce the disease burden of depression. In the yearly national report on eHealth in the Netherlands only 6% of a representative sample of mental health patients used online treatment in combination with usual care in 2015 [3]. The low uptake of iCBT is recognized by multiple health care organizations and policy makers across different countries but is not yet well understood.

The MasterMind project (Management of mental health disorders through advanced technology and services-telehealth for the MIND) [4] aims to provide understanding of the factors that promote or hinder the uptake of iCBT for depression throughout Europe from a broad perspective. MasterMind explores the role of different stakeholders that are involved in the successful implementation of the intervention (e.g. patients, therapists, mental health care organizations, regional mental health care systems) [5]. In Mastermind videoconferencing services will also be implemented in routine mental health care. In the current study protocol we focus specifically on the role of therapists in implementing iCBT in routine practice in the Netherlands.

To evaluate the role of the therapists in interaction with other barriers and facilitators at multiple levels of mental health care, the use of a framework and mixed method approach is needed [6, 7]. In this study the RE-AIM framework [8, 9] is used to structure the evaluation of the different implementation factors [10]. RE-AIM is widely used in implementation research, especially in e-mental health research. RE-AIM stands for: Reach, Efficacy or Effectiveness, Adoption, Implementation, and Maintenance. Together these dimensions help shed light on the barriers and facilitators and can ease the transition from research to practice. The RE-AIM framework is applied for evaluations at an individual level and an organizational level, as each level can provide valuable independent information on intervention impact. Reach and Effectiveness have individual levels of impact, whereas Adoption and Implementation have organizational levels of impact. Maintenance can have both an individual and an organizational level of impact [11].

The current situation in the Netherlands is that most mental health care organizations have adopted iCBT; there are many different providers for online platforms to choose from and almost every mental health care organization already has on online platform in use for the internet-based treatment of mental disorders. However, mental health care organizations have different developmental levels regarding the implementation of iCBT in their care system. Some organizations have worked with online interventions for several years now, while others have just started implementing iCBT. Although many therapists in the mental health care organizations have been trained in using an online platform, a recent naturalistic study on the implementation of iCBT in a mental health service organization in the Netherlands showed that only 3.6% of the patients with depression or anxiety were offered iCBT (in a

60 | CHAPTER 3 blended format: face-to-face and online sessions) by 18% of the trained therapists over a time period of 3 years [12]. Thus, the reach of the intervention is still limited. A similar result was found by Carper and others [13] who showed that in the US there is a lack of awareness of iCBT in depression care among patients and clinicians. The findings concluded that a more widespread dissemination of iCBT interventions is needed.

In the literature some attention has already been paid to therapist related factors that play a role in the implementation of iCBT. Donovan et al. [14] found that knowledge of iCBT among mental health care professionals can play a role in the uptake of iCBT: greater knowledge of the intervention’s effectiveness was associated with fewer perceived disadvantages of iCBT in general, and more circumstances under which iCBT was perceived to be useful. Wilhemsen and others [15] indicated that lack of time, varied practice, inadequate knowledge of the program and changing professional’s habits are hindering factors in the implementation of iCBT. A qualitative study conducted by Kivi et al. [16] with four therapists identified similar barriers to implementing iCBT in mental health care. Therapists perceived a lack of knowledge, skills, resources and time and they had the feeling that they never fully mastered the protocol. Another explorative qualitative study with 11 therapists showed that several therapists see “iCBT as focused, structured, evidence-based, clear and safe” [17]. Although they feel that iCBT in general can provide the therapist more work-time control because of the structured nature, several therapists view iCBT as inferior to face-to-face CBT and think that a working alliance in the latter is achieved faster and more easily.

It is important to take clinical implications and therapists’ concerns into account when implementing iCBT in routine practice. These considerations may provide more insight in how to optimize the uptake by therapists. In this study we will further explore factors regarding the therapists’ role in implementing iCBT for depression in adults in secondary mental health care.

Study objectives There is a need for more information about the barriers therapists experience in adopting and implementing iCBT and how to overcome these barriers. To this end we focus on the needs of the therapist and (patient and organization-related) facilitating and impeding factors. The MasterMind study provides an excellent opportunity to study these therapist factors. The objectives of this study are: 1) To gain insight into the extent to which therapists integrate iCBT in their routine practice.

2) To increase insight into the factors that promote or hinder the use of iCBT by therapists in depression care.

3) To provide recommendations on how to overcome barriers for therapists in the implementation of iCBT in routine depression care.

In order to obtain insight into strategies for optimizing therapist uptake of iCBT we use the RE-AIM framework to answer the following questions: 

Reach: What is the proportion and representativeness of therapists in the participating mental health organizations that offer iCBT for depression during the study?

Study Protocol: Therapists’s Role| 61 

Effectiveness: What is the (positive and negative) impact of iCBT on the therapists regarding perceived effectiveness, satisfaction and usability? How is this related to patient outcomes regarding symptom reduction, satisfaction and usability?



Adoption: To what extent do mental health care organizations adopt iCBT; how is the therapist facilitated in this and which efforts are made to this end in the participating mental health organizations?



Implementation: To what extent is iCBT implemented as intended in routine practice? What are implementation barriers and facilitators from therapists’ perspective?



Maintenance: To what extent does iCBT become a sustained part of routine practice, and what are facilitating and hindering factors in maintenance?

Methods Study design To reach the study objectives, the uptake of iCBT for depression in several mental health care practices in the Netherlands will be monitored. This multicenter evaluation will assess the involvement of therapists in the implementation of iCBT in interaction with organizational and patient factors. The study will investigate how implementation is reflected in the extent to which organizations adopt the intervention, and how the therapist’s perspective is affected by patient related factors such as changes in symptoms, satisfaction and perceived usability of the iCBT. A mixed-method approach will be used in order to provide an understanding of the implementation factors (quantitative) and to evaluate the facilitating and hindering factors in the involvement of therapists in the implementation of iCBT (qualitative) [8, 18, 19].

Sample For the MasterMind study on an European level, the target number of 5230 participants is based on purposive quota sampling meaning the number of participants needed to be enrolled to regard the interventions as being implemented in routine practice after the study is finished, for the study in the Netherlands the target number is 300 participants and their therapists [5]. With estimating the sample size, feasibility and service delivery characteristics of iCBT in routine care in the Netherlands are taken into account. Ten mental health care organizations spread throughout the Netherlands will be approached. The selection of organizations is based on purposive sampling, meaning the organizations have different stages of iCBT implementation, as measured by the number of trained therapists and patients treated with iCBT; the organizations are located in different parts of the Netherlands and may use different iCBT treatment platforms for depression in secondary mental health care. The diversity of the participating organizations is necessary to enrich the (qualitative) data needed for the evaluation of the therapists’ involvement in the Netherlands. In the participating organizations we will approach key stakeholders at different levels such as eHealth project managers and directors.

Therapists The therapists who are trained in iCBT will be invited to participate in the study. The therapists can be psychologists, mental health nurses or psychiatrists. Some of them are already experienced in delivering iCBT for depression; others are trained but not yet experienced.

62 | CHAPTER 3 Patients Inclusion criteria: Patients from the participating mental health care organizations will be invited for participation if they are aged 18 years or older, have a mild, moderate or severe depression as a primary diagnosis according to the clinician, and are indicated for cognitive behavioral treatment for depression following routine care procedures. All patients need to explicitly consent to taking part in the study.

Patients may provide electronic consent to use their data for the evaluation study. Exclusion criteria: Patients will be excluded from the study if they a) do not have a valid email address and do not have a computer with internet access b) do not have adequate Dutch language skills; verbal and written.

Representatives of the mental health care organizations From each mental health care organization representatives will be invited to take part in the study. The representatives are persons who have a substantive decision-making position in the implementation of iCBT in the organization’s mental health care practice.

Intervention Cognitive behavioral therapy relies on cognitive input and behavioral change and is well structured.

This makes it suitable for an innovative information-technology based application [20]. In such an application an online treatment platform provides patient and therapist with an overview of the treatment sessions, information and exercises. Through the platform the patient can access psychoeducation, treatment information and exercises at any time within the home environment.

Furthermore, the therapist can monitor the progress of the patient online. With the integration of technology, iCBT can have several promising advantages; iCBT may improve access to treatment, because it can lower the barriers to seeking mental health care; iCBT can promote self-monitoring and self-management of the patient; iCBT might reduce treatment costs because of a reduction in face-toface time [21]; the online platform is available for the patient 24 h, 7 days a week; and the online sessions can be repeated [22]. Furthermore, iCBT may improve the quality of delivery of the treatment, because of the standardized nature that gives a clear overview of the treatment, for both therapist and patient.

Most mental health care organizations in the Netherlands have contracts with providers of webbased treatments for common mental disorders and are therefore able to offer iCBT for depression.

The organizations use various secure web-based online treatment platforms. In routine secondary mental health care, iCBT is mostly delivered in a blended format, consisting of the combination of online treatment modules and face-to-face sessions. The iCBT treatment modules are based on evidence based treatment protocols for face-to-face cognitive behavioral therapy and are in agreement with the Dutch multidisciplinary guidelines for depression [23]. The core components of the treatment are: 1) psycho-education, 2) cognitive restructuring, 3) behavioral activation, and 4) relapse prevention. As there is a distinction in secondary health care in the Netherlands between basic mental health care (mild to moderate depression), and specialized mental health care (more complex moderate to severe depression) the number of iCBT sessions differs between these health care levels: up to a maximum of 10–15 sessions in basic mental health care and usually 15–20 sessions in specialized mental health care.

At both health care levels, in blended treatment, the number of face-to-face sessions is reduced because half or more of the face-to-face sessions are replaced by online treatment sessions [24, 25].

Study Protocol: Therapists’s Role| 63 Outcome measures The outcomes will be structured according to the RE-AIM framework. The concepts of the multidisciplinary assessment outcomes will be used to obtain a better understanding of the change in mental health care delivery and to help make decisions about the implementation more explicit and transparent. In this study the RE-AIM framework was somewhat adapted to fit to our focus on the therapists’ role, as was done earlier by Boersma et al. [26, 27]. Reach is the proportion and representativeness of therapists in the participating mental health organizations that offer iCBT for depression. Effectiveness is the impact of iCBT on therapists regarding perceived effectiveness, satisfaction and usability of iCBT and the impact of iCBT on patients regarding symptom reduction, satisfaction and usability. Adoption is the extent to which mental health care organizations adopt iCBT and how the therapist is facilitated in this. Implementation is the extent to which iCBT is implemented as intended in routine practice, including implementation barriers and facilitators from the therapist’s perspective. Maintenance is the extent to which iCBT becomes part of the normal routine of therapists.

In other words, to have an impact on depression care, iCBT must be adopted by the mental health care organizations and therapists, reach the therapists who deliver the intervention, be implemented as intended, effectively improve outcomes and be maintained over time. The definitions of the RE-AIM frame work are presented in Table 1. In Table 2 an overview is given of the RE-AIM dimensions, the related content, factors and measures.

64 | CHAPTER 3 Table 1. Definitions of the five RE-AIM dimensions and the definitions in this study Dimension REAIM (level)

Definition RE-AIM (Glasgow et al. 1999)

Definition RE-AIM in this study

Reach (therapist)

The absolute number, proportion, and representativeness of individuals who are willing to participate in a given initiative, intervention, or program.

The number, proportion and representativeness of therapists in the participating mental health organizations that offered iCBT for depression during the study.

Effectiveness (patient and therapist)

The impact of an intervention on important outcomes, including potential negative effects, quality of life, and economic outcomes.

The (positive and negative) impact of iCBT on the therapists and patients regarding perceived effectiveness, satisfaction and usability

Adoption (organization)

The absolute number, proportion, and representativeness of settings and intervention agents (people who deliver the program) who are willing to initiate a program.

The extent to which mental health care organizations adopt iCBT and how the therapist is facilitated in this.

Implementation (therapist and organization)

At the setting level, implementation refers to the intervention agents’ fidelity to the various elements of an intervention’s protocol, including consistency of delivery as intended and the time and cost of the intervention.

The extent to which iCBT is implemented as intended in routine practice, including implementation barriers and facilitators from the therapist’s perspective.

Maintenance (therapist and organization)

The extent to which a program becomes institutionalized or part of standard organizational practices and policies.

The extent to which iCBT becomes a sustained part of standard practice and facilitating and hindering factors from therapists’ and organizational perspective.

Study Protocol: Therapists’s Role| 65 Table 2. Overview of the RE-AIM dimensions, the related content and measures RE-AIM dimension

Content

Measures/factors

Adoption

Participating Organizations: n and %

Characteristics of adopters’ Influencing factors: region, size, capacity, type, previous experience with other platforms

Reach

Total potential therapists, n ↓ Therapists eligible n and %↓

Therapists excluded n, % and reasons

Therapists enrolled n and %

Therapists who decline n, % and reasons

Therapists not contacted/other n and %

Characteristics of enrollers vs decliners. Influencing factors: mandatory vs voluntary, available resources.

Implementation

Extent iCBT is delivered by therapists (as in protocol)

Extent iCBT is delivered as intended. Influencing factors: complexity of the intervention, costs in time and money, training, implementation activities, adaptations

Effectiveness

Impact of ICBT on therapists regarding perceived effectiveness, satisfaction and usability

The positive and negative impact of ICBT on therapists regarding perceived effectiveness, satisfaction and usability and related patient outcomes.

Influencing factors: evidence for effectiveness of iCBT, impact across subgroups

Maintenance

Extent organizations maintain and/or modify iCBT

The extent iCBT becomes a sustained part of routine practice. Influencing factors: benefits vs costs. Amount of training, technical assistance, funding

Measurements Qualitative and quantitative measures will be used to help understand the factors related to the implementation of iCBT according to the RE-AIM framework. Focus-group interviews and semistructured interviews with the therapists will be conducted face-to-face at the end of the study (after 24 months of data collection). The interviews with therapists will focus on the experiences and

66 | CHAPTER 3 perceptions of the trained therapists who participated in delivering iCBT in their daily work. In addition, interviews will be held with trained therapists who do not deliver the treatment, in order to explore their reasons and their perceptions of iCBT. In the focus groups with the therapists discussion on the following themes will be held: the patient (needs, profile, safety), the therapist (needs and profile), implementation barriers and facilitators and usability and satisfaction with iCBT. Prior to the focusgroup interviews and semi-structured interviews, participants will be asked to fill out a short questionnaire to obtain general information about the interviewees and to prepare them for the interviews. If therapists stop working with iCBT (due to other jobs, positions), they will be asked for their input at an earlier time point.

The semi-structured interviews with key figures of the participating mental health care organizations will be conducted face-to-face and they will also take place at the end of study. In the interviews the focus will be on implementation barriers and facilitators, usability, satisfaction and maintenance, from an organizational perspective. Interviewers and moderators for the focus groups will be provided with training and comprehensive interview guides including tips for interview techniques as well as suitable questions. In addition, relevant documents provided by the mental health care organizations will be reviewed (e.g. implementation and quality improvement plans, treatment protocols).

Quantitative data will be collected at baseline and at the end of treatment from patients, and from therapists at the end of the study. Information on demographic characteristics of the therapists that will be collected comprises age, gender, education, profession, years of professional experience, number of hours working and experience with iCBT. From patients demographic characteristics will be collected about age, gender, education, employment, marital status and use of antidepressant medication. In addition, to measure symptoms of depression, we will adhere to routine practice and this will be registered in terms of several symptom questionnaires, depending on the questionnaires used at the participating organization. Health care organizations in the Netherlands use Routine Outcome Measurements (ROM). ROM consists of a broad selection of validated instruments to monitor treatment effects. The use of ROM is a mandatory component of treatment as required by health insurance companies. The most frequently used questionnaires are the Patient Health Questionnaire (PHQ-9) [28], the Inventory of Depressive Symptoms (IDS-SR) [29, 30] and the Quick Inventory of Depressive Symptomatology (QIDS) [31].

Perceived usability and satisfaction with iCBT will be measured from both therapists’ and patients’ perspective. Patient treatment satisfaction will be measured with the 8 item Client Satisfaction Questionnaire (CSQ-8) [32]. System usability will be measured with the System Usability Scale (SUS) [33]. The perceived satisfaction and usability of iCBT by patients will be measured at the end of treatment. In addition, therapist satisfaction and usability of iCBT will be measured at the end of the study; satisfaction will be measured with the 3 items CSQ and usability will be measured with the SUS. To assess the extent to which the treatment is delivered by protocol, the usage data from the online platforms will be gathered (e.g. number of online sessions and usage of other platform functionalities).

Statistical analyses Analysis will be of a summative nature combining quantitative data with qualitative data [34, 35] to draw a broad picture of implementation and to describe the role of the therapist in the uptake of iCBT in connection with the patient and the organization. On a patient level, data of all participants will be

Study Protocol: Therapists’s Role| 67 included in the statistical analysis when they are eligible and agree to receiving treatment, regardless whether the participants complete their treatment or not.

The evaluation of the implementation will be performed by using the outcomes as described above following the RE-AIM dimensions. Descriptive analyses (frequencies, means and percentages) of the quantitative data (demographics of patients and therapists, depressive symptoms, perceived satisfaction and perceived usability) will be performed with the Statistical Package for the Social Sciences (SPSS), version 22.0. The qualitative data collected during the focus groups (including field notes) and the semi-structured interviews will be audiotaped and transcribed verbatim. We will use standard thematic content analysis techniques by the following steps: 1) familiarizing and summarizing the data 2) identifying codes and themes 3) coding the data and 4) organizing the codes and themes [36]. In order to ensure reliability, agreement on codes and concepts between the members of the research team will be sought. Qualitative data analyses will be performed by using the software Atlasti.7.

Discussion The primary aim of the study is to gain insight into the extent to which therapists integrate internetbased Cognitive Behavioral Therapy (iCBT) in routine depression care in the Netherlands. The second aim is to gather insight into the factors that promote or hinder the implementation of iCBT. More specifically, we want to gain knowledge on the role of the therapist in the implementation of iCBT, in relation to patient and organizational factors; how do therapists evaluate iCBT, what resources are available and are needed, and what decision-making processes are important? The third aim is to provide recommendations for implementing iCBT in different contexts and organizations, in order to make high quality treatment more widely available for adults with depression.

The strength of this protocol is its multifaceted evaluation with a focus on the role of the therapist in relation to the organization and the patients with depression. A second key strength is the mixed method approach to incorporate quantitative and qualitative aspects of the implementation outcomes.

Glasgow [8] argues that qualitative data can help in interpreting quantitative results, “such as why potential users decline participation or why they do not remain engaged over time” and that “qualitative measures can be very helpful in understanding contextual issues”. The focus groups and interviews with therapists will give us more insight into how to implement iCBT in routine practice.

Furthermore, the semi-structured interviews with representatives of the mental health care organizations will contribute to a better understanding of the implementation on an organizational level. The use of multiple data sources will provide a detailed understanding of how iCBT can be implemented. The RE-AIM framework helps to guide data collection and analysis for the evaluation. To our knowledge, this is one of the first studies to examine the implementation of iCBT for depression in routine mental health care systematically.

In sum, the findings of this study will provide a number of implications for future practice. When we know more about the barriers and facilitators for up-scaling and maintaining interventions and have more knowledge about the effective strategies that will help identify and characterize these, it will help us make mental health care for depression more widely available and effective in future practice.

68 | CHAPTER 3 Abbreviations CSQ: Client satisfaction questionnaire; EPD: Electronic patient dossier; iCBT: internet-based cognitive behavioral therapy; IDS-SR: Inventory of depressive symptoms; PHQ: Patient health questionnaire; REAIM: Reach, effectiveness, adoption, implementation, maintenance; ROM: Routine outcome measurement; SUS: System usability scale.

Funding The MasterMind project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme (CIP) by the European Commission (Grant Agreement number: 621000).

Study Protocol: Therapists’s Role| 69

References [1]

[2]

[3]

[4] [5]

[6] [7]

[8] [9] [10] [11] [12]

[13]

[14]

[15]

[16]

Andersson, G., Cuijpers, P., Carlbring, P., Riper, H., & Hedman, E. (2014). Guided internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry; 13, 288–95.

Hedman, E., Ljótsson, B., & Lindefors, N. (2012). Cognitive behavior therapy via the Internet: a systematic review of applications, clinical efficacy and cost-effectiveness. Expert Rev Pharmacoecon Outcomes Res; 12, 745–64.

Krijgsman, J., Peeters, J., Burghouts, A., Brabers, A., de Jong, J., Moll, T., et al. (2015). Tussen vonk en vlam: eHealth-monitor 2015. [Between experiment and investment: eHealth monitor 2015].

Management of mental health disorders through advanced technology and services-telehealth for the MIND. (2014). http://mastermind-project.eu/. Accessed 22 Aug 2016.

Vis, C., Kleiboer, A., Prior, R., Bønes, E., Cavallo, M., Clark, S.A., et al. (2015). Implementing and up-scaling evidence-based eMental health in Europe: the study protocol for the MasterMind project. Internet Interv; 2, 399–409.

Gaglio, B., Shoup, J.A., & Glasgow, R.E. (2013). The RE-AIM framework: a systematic review of use over time. Am J Public Health; 103, 38–46.

Cochrane, L.J., Olson, C.A., Murray, S., Dupuis, M., Tooman, T., & Hayes, S. (2007). Gaps between knowing and doing: understanding and assessing the barriers to optimal health care.

J Contin Educ Health Prof; 27, 94–102.

Glasgow, R.E. (2007). eHealth evaluation and dissemination research. Am J Prev Med ;32, 119– 26.

Glasgow, R.E., Vogt, T.M., & Boles, S.M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health; 89, 1322–7.

Nilsen, P. (2015). Making sense of implementation theories, models and frameworks.

Implement Sci;10, 53.

RE-AIM framework (2016). http://www.re-aim.org/. Accessed 22 Aug 2016.

Kenter, R.M.F, van de Ven, P.M., Cuijpers, P., Koole, G., Niamat, S., & Gerrits, R.S., et al. (2015).

Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: results from a naturalistic study. Internet Interv;2, 77–83.

Carper, M.M., McHugh, R.K., & Barlow, D.H. (2013). The dissemination of computer-based psychological treatment: a preliminary analysis of patient and clinician perceptions. Adm Policy Ment Heal; 40, 87–95.

Donovan, C.L., Poole, C., Boyes, N., Redgate, J., & March, S. (2015). Australian mental health worker attitudes towards cCBT: What is the role of knowledge? Are there differences? Can we change them? Internet Interv; 2, 372–81.

Wilhelmsen, M., Hoifodt, R.S., Kolstrup, N., Waterloo, K., Eisemann, M., Chenhall, R., et al.

(2014). Norwegian general practitioners’ perspectives on implementation of a guided webbased cognitive behavioral therapy for depression: a qualitative study. J Med Internet Res; 16:9.

Kivi, M., Eriksson, M.C.M., Hange, D., Petersson, E.L., Björkelund, C., & Johansson, B. (2015).

Experiences and attitudes of primary care therapists in the implementation and use of internetbased treatment in Swedish primary care settings. Internet Interv; 2, 248–56.

70 | CHAPTER 3 [17] [18]

[19]

[20] [21]

[22] [23]

[24]

[25]

[26]

[27]

[28] [29]

[30] [31]

Bengtsson, J., Nordin, S., & Carlbring, P. (2015). Therapists’ experiences of conducting cognitive behavioural therapy online vis-à-vis face-to-face. Cogn Behav Ther; 6073, 1–10.

Landsverk, J., Brown, H., Chamberlain, P., Palinkas, L.A., & Horwitz, S.M. (2012). Design and analysis in dissemination and implementation research. In: Brownson, R., Colditz, G.A., Proctor, E.K. editors. Dissemination and implementation research in health: translating science to practice. Oxford: Oxford Univ Press; 225–60.

Palinkas, L.A., Aarons, G.A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2010).

Mixed method designs in implementation research. Adm Policy Ment Heal Ment Heal Serv Res; 38, 44–53.

Proudfoot, J.G. (2004). Computer-based treatment for anxiety and depression: is it feasible? Is it effective? Neurosci Biobehav Rev; 28, 353–63.

Kaltenthaler, E., Brazier, J., De Nigris, E., Tumur, I., Ferriter, M., Beverley, C., et al. (2006).

Computerised cognitive behaviour therapy for depression and anxiety update: a systematic review and economic evaluation. Health Technol Assess; 10, 1–70.

National Institute for Clinical Excellence. (2013). Computerised cognitive behaviour therapy for depression and anxiety. London: NICE. Technology Appraisal Guidance [TA97].

Spijker, J., Bockting, C., Meeuwissen, J., van Vliet, I., Emmelkamp, P., Hermens, M., et al. (2013).

Multidisciplinaire richtlijn Depressie (Derde revisie). Richtlijn voor de diagnostiek, behandeling en begeleiding van volwassen patiënten met een depressieve stoornis [Multidisciplinary Guideline for Depression (Third Revision)] Utrecht: Guideline for Diagnostics.

Kooistra, L.C., Wiersma, J.E., Ruwaard, J., van Oppen, P., Smit, F., Lokkerbol, J., et al. (2014).

Blended vs. face-to-face cognitive behavioural treatment for major depression in specialized mental health care: study protocol of a randomized controlled cost-effectiveness trial. BMC Psychiatry; 14, 290.

Van der Vaart, R., Witting, M., Riper, H., Kooistra, L., Bohlmeijer, E.T., & van Gemert-Pijnen, L.J. (2014). Blending online therapy into regular face-to-face therapy for depression: content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry; 14, 355.

Boersma, P., van Weert, J.C.M., Lakerveld, J., & Dröes, R.M. (2015). The art of successful implementation of psychosocial interventions in residential dementia care: a systematic review of the literature based on the RE-AIM framework. Int Psychogeriatr; 27, 19–35.

Boersma, P., van Weert, J.C.M., van Meijel, B., & Dröes, R.M. (2016). Implementation of the Veder contact method in daily nursing home care for people with dementia: a process analysis according to the re-aim framework. J Clin Nurs; 26 (3-4), 436-455.

Kroenke, K, Spitzer, R.L., & Williams, J.B.W. (2001). The PHQ-9. J Gen Intern Med; 16, 606–13.

Rush, A., Giles, D.E., Schlesser, M.A., Fulton, C.L., Weissenburger, J., & Burns, C. (1986). The inventory for depressive symptomatology (IDS): preliminary findings. Psychiatry Res; 18, 65– 87.

Rush, A., Gullion, C., Basco, M., Jarrett, R., & Trivedi, M. (1996). The inventory of depressive symptomatology (IDS): psychometric properties. Psychol Med; 26, 477–86.

Rush, A.J., Trivedi, M.H., Ibrahim, H.M., Carmody, T.J., Arnow, B., Klein, D.N., et al. (2003). The 16-Item quick inventory of depressive symptomatology (QIDS), clinician rating (QIDS-C), and

Study Protocol: Therapists’s Role| 71

[32] [33] [34] [35]

[36]

self-report (QIDS-SR): a psychometric evaluation in patients with chronic major depression. Biol Psychiatry; 54, 573–83.

Larsen, D.L., Attkisson, C.C., Hargreaves, W.A., & Nguyen, T.D. (1979). Assessment of client/patient satisfaction: development of a general scale. Eval Program Plann; 2, 197–207.

Brooke J. (1996). SUS-A quick and dirty usability scale. Usabil Eval Ind; 189, 4–7.

Caracelli, V. & Greene, J. (1993). Data analysis strategies for mixed-method evaluation designs.

Educ Eval Policy Anal; 15, 195–207.

Gaglio, B. & Glasgow, R. (2012). Evaluation approaches for dissemination and implementation research. In: Brownson R, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice. Oxford: Oxford Univ Press; 327–58.

Green, J. & Thorogood, N. (2013). Qualitative methods for health research. 3. London: Sage.

72 | CHAPTER 4

CHAPTER 4

Well, what just happened is what I need: that you get inspiration from colleagues who work with it, because then you think ‘oh yes, you can that as well ‘ Therapist Tom

Qualitative Study: Therapists’ Perspectives| 73

Why Uptake of Blended Internet-Based Interventions for Depression is Challenging: A Qualitative Study on Therapists’ Perspectives Mayke Mol, Claire van Genugten, Els Dozeman, Anneke van Schaik, Stasja Draisma, Heleen Riper & Jan Smit Journal of Clinical Medicine 2020

4

74 | CHAPTER 4

Abstract

Background: Blended cognitive behavioral therapy (bCBT; online and face-to-face sessions) seems a promising alternative alongside regular face-to-face CBT depression treatment in specialized mental health care organizations. Therapists are key in the uptake of bCBT. Objective: This study focuses on therapists’ perspectives on usability, satisfaction, and factors that promote or hinder the use of bCBT in routine practice.

Methods: Three focus groups (n=8, n=7, n=6) and semi-structured in-depth interviews (n=15) were held throughout the Netherlands. Beforehand, the participating therapists (N=36) completed online questionnaires on usability and satisfaction. Interviews were analyzed by thematic analysis.

Results: Therapists found the usability sufficient and were generally satisfied with providing bCBT. The thematic analysis showed three main themes on promoting and hindering factors: 1) therapists’ needs regarding bCBT uptake, 2) therapists’ role in motivating patients for bCBT, and 3) therapists’ experiences with bCBT. Conclusions: Overall, therapists were positive; bCBT can be offered by all CBT-trained therapists and future higher uptake is expected. Especially the pre-set structure of bCBT was found beneficial for both therapists and patients. Nevertheless, therapists did not experience promised time-savings—rather, the opposite. Besides, there are still teething problems and therapeutic shortcomings that need improvement in order to motivate therapists to use bCBT. Keywords: cognitive behavioral therapy; blended treatment; depressive disorder; implementation; therapists’ perspective; routine care

Qualitative Study: Therapists’ Perspectives| 75

Introduction Background Face-to-face (FtF) Cognitive Behavioral Therapy (CBT) is an evidence-based treatment for depression [1]. According to international guidelines, CBT is a standard option for treating patients with depression in specialized mental health care [2]. Unfortunately, patient access to CBT is hindered by several barriers such as perceived personal stigma, costs, long waiting lists, and limited availability of licensed therapists [3,4]. It seems therefore promising to offer CBT with the support of technology and without the FtF sessions, thereby potentially reducing therapists’ time and costs due to increased involvement of patients themselves. In internet-based cognitive behavioral therapy (iCBT), treatment access can be improved to offer CBT on an online platform. This would help reduce stigma, waiting lists, and costs, since patients can work through sessions at their own pace, in their own time, and therapist time is lessened. It has been shown that iCBT, especially with online therapist guidance, has good clinical effects for patients with depression [5,6].

In geographically dispersed countries like Canada and Australia, online clinics have successfully reached many patients through iCBT (e.g., Mindspot Clinic, Online Therapy Unit) [7]. In the Netherlands, iCBT, with guidance at a distance, has been offered by Interapy for more than two decades (www.interapy.nl). This mental health care organization (MHO) was founded as an online therapy center and provides specialized online evidenced-based care for among other thing depression, trauma and panic disorder. Other MHO’s, where face-to-face treatments are the standard, also are implementing iCBT to improve patient access and make treatments more cost-efficient.

However, iCBT uptake in MHOs is limited. This might be partly because therapists perceive iCBT to be limited in crisis situations, ineligible for severe depressive symptoms, and having a one-size fits all approach [8,9]. In addition, given the relatively high density of mental health services, the need for iCBT in the Netherlands may differ from elsewhere.

Blended CBT, a new format of iCBT, which has been recently adopted by MHOs in routine care systems in the Netherlands, could address the limitations since online sessions are combined with FtF sessions in a single, integrated, standardized CBT treatment protocol [10]. bCBT aims to improve both quality of care and cost-efficacy by replacing a portion of the FtF sessions with online sessions [11,12].

Through reducing the number of FtF sessions, therapists’ time is saved, they can potentially treat more patients, and waiting lists can be shortened. In addition, within the online platform, therapist contact is extended beyond the FtF sessions by online communication. A number of studies have shown that bCBT can be effective in treating patients with depression [13–15]. However, similarly to iCBT in MHOs, the uptake of bCBT by therapists and patients is limited as it has not yet been implemented across the board in MHOs in the Netherlands and elsewhere [16].

The gap between promising research findings and the limited iCBT and bCBT uptake in routine practice was one of the reasons behind the European MasterMind project, designed to increase and evaluate uptake and implementation of internet-based interventions in ten European countries (www.mastermind-project.eu) [17]. In MasterMind there was a specific focus on therapists’ perspectives, as they play a substantial role in the uptake of iCBT and bCBT. And besides, studies concerning therapists’ perspectives in routine practice are scarce [18].

In a qualitative study in Germany, several barriers and facilitators were identified based on the perspectives of therapists (N=5) providing bCBT alongside a randomized controlled study [19]. Key

76 | CHAPTER 4 issues varied from problematic platform technology to an unclear concept of embedding bCBT in the mental health care system. Despite the barriers, therapists viewed bCBT as an adequate treatment option for patients with depression. To build on this, more in-depth knowledge on therapists’ perspectives in specialized routine care could contribute to optimizing uptake of bCBT, as the therapists’ workflow and patient-related factors are more complex in daily practice than in research settings.

The objective of this qualitative study was to investigate therapists’ perspectives in routine care when transitioning to using bCBT as an alternative alongside CBT for patients with depression within outpatient specialized mental health care settings in the Netherlands. To increase insight into the factors that influence uptake of bCBT in routine care, the following questions were answered using data triangulation [20] in the context of questionnaires, focus groups, and semi-structured interviews with therapists: 

What are the therapist perspectives on bCBT usability (e.g., perceived ease of use of the platform)?



How satisfied are therapists with offering bCBT (e.g., overall satisfaction with bCBT)?



What are the therapist perspectives on factors that promote or hinder the use of bCBT in daily practice (e.g., training, patient eligibility, safety)?

Methods Therapists Therapists were (CBT trained) psychologists, mental health nurses, or psychiatrists working at seven MHOs where bCBT had been implemented. To be included in the study, therapists had to be trained in the use of the online platform prior to or during the study period.

The therapists were employed at small (200 registrations for depression per year) to large privatenon-profit MHOs (5000 depression registrations per year) spread across the Netherlands. All organizations had bCBT available on a platform from the same large commercial provider (Minddistrict;www.minddistrict.com)—most since 2014. MHOs provided bCBT training to (all or a part of) their therapists either ‘in-house’ or externally by the provider, or via online training. The estimated duration of the training varied from four hours to two days. MHO’s stimulated therapists in different ways for bCBT (e.g., by setting bCBT targets or by special blended consultation hours), however therapists could make their own decision about the treatment of choice.

Recruitment Through other users of the Minddistrict platform and personal contacts of the authors, 10 MHOs were invited to be part of the study because they implemented an online platform into routine care. One MHO withdrew because it ceased to exist, another MHO was too busy with ongoing other projects and trials, and one MHO switched to a different platform provider and needed time to explore other providers. In total, seven MHOs participated.

Therapists were identified through team managers, and other specially members of different teams, appointed by the manager of the MHOs. Therapists for focus groups or in-depth interviews were purposively sampled to cover diversity of use and experience with bCBT, age, professional background, and geographic region. First, therapists from two MHOs were invited to participate in a focus group.

The focus groups were held within the context of the European implementation project MasterMind.

Qualitative Study: Therapists’ Perspectives| 77

March 2014

Patient data collection, monitoring patients receiving treatment

February 2017

Start MasterMind project

April 2016

The MasterMind project ran from March 2014 to February 2017. For a more detailed description, see the study protocol of the MasterMind study in the Netherlands [18]. In total, 3 focus groups were held with 21 participants.

Second, to add more depth to the topics discussed in the focus groups, complementary semi-structured individual interviews were held with therapists (who had not participated in the focus groups) from seven MHOs. After 15 interviews, no new information or themes were brought up and data saturation was reached. Two weeks prior to the focus group and interviews, all participating therapists received an informed consent form and questionnaires were used to gather information on therapist demographics, satisfaction, and usability of bCBT via a secure online survey tool (Survalyzer; www.survalyzer.com). Figure 1 provides an outline of the study.

Focus groups + questionnaires Therapist data collection

Semi-structured interviews + questionnaires

Invited = 78

MHO 1+2

Consented = 38

Withdrew n= 17 conflicting schedule

Completed = 21 Invited = 24

MHO 1-7

Consented = 16

Withdrew n= 17 conflicting schedule

Completed = 15

Figure 1. Study outline.

Intervention Therapists worked with an integrated bCBT protocol in which FtF and online sessions were alternated.

The blended protocol for depression mostly consisted of 10 online sessions. These were supported by FtF sessions (for mild or moderate depression 5 FtF sessions every two weeks; and for (very) severe depression 10 to 11 weekly FtF-sessions). bCBT started and ended with a FtF session. Therapists could individualize the protocol to the patient’s needs by repeating and/or skipping online sessions. More detailed information about the protocol can be found elsewhere [21].

Patients could be offered bCBT if 1) they were aged 18 years or older; 2) had a mild, moderate, or severe depression as a primary diagnosis according to the therapist; and 3) were indicated for cognitive behavioral treatment for depression following routine secondary mental health care procedures.

Exclusion criteria were 1) not having a valid email address and a computer with Internet access and 2) not having adequate Dutch language skills (both verbal and written).

78 | CHAPTER 4 Therapists had access to the (Minddistrict) platform through a dashboard, which showed treatment sessions, including homework assignments, and had optional functions for a patient diary and depressive symptom questionnaires. The role of the therapists was to monitor the patients’ online progress and to provide patients with personalized written feedback to homework assignments after each completed online session. Therapists could additionally communicate through a message function on the platform about practical issues (e.g., upcoming appointments, reminders, or questions about assignments). The safety of the platform was guaranteed by meeting the European requirements for certified data security [22].

Data collection Questionnaires The 10 items of the System Usability Scale measured perceived usability (SUS) [23]. The SUS is a validated questionnaire for evaluating usability of internet-based interventions as perceived by therapists and showed a good reliability (omega coefficient = 0.91) [24]. It is unofficially translated in Dutch [25]. The score for the total score range from 0 to 100. A score above the cut-off of 68 is considered to be above average [26]. The Client Satisfaction Questionnaire-3 measured satisfaction with bCBT (CSQ-3), is officially translated to Dutch [27] and was adapted to therapists (e.g., CSQ1 = To what extent has the bCBT intervention met your needs in treating depressed patients?). The CSQ showed good reliability (McDonald omega = 0.95) and validity in a sample of internet-based depression intervention users [28].

Focus group interviews To identify relevant therapist factors in bCBT uptake, dimensions of the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, Maintenance) [29], the MAST framework (Model for assessment of telemedicine) [30], and the normalization process theory [31] were used to build and structure the focus group topic list (see Appendix 1). The main predetermined themes were: patient and therapist factors, barriers and facilitators, satisfaction and usability. The focus group was pilot tested with a moderator and three participants who acted as therapists to refine the topic list and time management. Key questions were presented on a PowerPoint presentation during the therapist focus groups. Via a mobile voting system (Sendsteps;www.sendsteps.com), participants could give their opinion on certain questions and statements in order to stimulate discussion. Discussions were moderated by senior researcher and co-author S.D. with experience in conducting focus groups. They were assisted by researcher and first-author M.M. The focus groups were audio-recorded. The duration of the focus groups was 100–115 min and they took place at the participating MHOs. To increase validity, a transcript of each focus group was sent to each participant for feedback.

Semi-structured in-depth interviews In semi-structured interviews, therapists were asked individually to reflect on the same topics as in the focus groups to further explore personal experiences. In addition, questions were added based on interview guides from previous research [19,32,33] as well as outcomes of the focus groups; there were slightly different questions for therapists with and without experience in providing bCBT for depression (see Appendix 2 and 3 for the topic lists). After 10 interviews, 3 additional topics were added since these seemed relevant to therapists (e.g., experience with writing online feedback, preferred ratio of

Qualitative Study: Therapists’ Perspectives| 79 online versus FtF sessions and preferred starting point for introduction of online platform). Interviews were conducted by two researchers (M.M. and C.v.G.). The interviews lasted 35–90 min. Most interviews were held at the MHOs (n=10); five were conducted by telephone for practical reasons. The interviews were audio-recorded. An interview transcript was sent to each participant for respondent validation.

Data analyses Descriptive analyses (frequencies, means, and percentages) of the quantitative data (demographics of therapists, perceived satisfaction, and usability) were performed with SPSS, version 22.0 (IBM Corp.

Armonk, NY, USA). Atlas-ti.7 software was used for qualitative analysis (version 7 and 8, Scientific Software Development GmbH. Berlin, Germany).

Focus groups and semi-structured in-depth interview transcripts were concurrently analyzed with thematic analysis techniques guided by the following steps [34]: 1) familiarization with the data; (re)reading of the transcripts and field notes; 2) generating initial codes; developing a codebook; 3) searching for (sub)themes; identifying broad topics; refining codebook; 4) reviewing the (sub)themes; 5) defining and naming (sub)themes; and 6) reporting. Anonymity was assured by using code numbers instead of names. Each sentence in the transcripts could contain multiple codes, and concurrent sentences of the same thematic code could be conjoined into one unit.

In the results section, therapist’ quotes were selected to illustrate the essence of a (sub)theme. The statements reflected perspectives from both focus groups and interviews. We chose to pool these sources as similar topics were discussed and no overlap existed between the therapists who participated in the focus groups and interviews. Both contributed to a more comprehensive understanding of therapists’ perspectives. When a perspective concerned a minority of therapists, patients, or other therapists within the participating organizations, this is explicitly mentioned.

To ensure reliability, agreement on codes and concepts between members of the research team was sought. Researchers M.M., C.v.G., and S.D. coded 4 interviews together until acceptable agreement (Fleiss kappa >0.50) was reached [35]. The remaining interviews were divided equally between researchers M.M. and C.v.G. Data collection process, data analysis, and any interim findings, as well as quality and methodological aspects, were discussed in the research group continuously.

Ethical issues The study was approved by the Medical Ethics Committee of the VU medical center, Amsterdam. They confirmed that the Medical Research Involving Human Subjects Act does not apply (registration number 2014.580) because therapists in this study were not required to follow certain procedures on behalf of the research (no randomization), and routine practice was followed. An internal scientific research committee approved the research proposal.

Results Table 1 presents therapist demographics and experience. With respect to distribution of experience with bCBT for depression, 42% of the therapists had considerable bCBT experience (providing bCBT 5 to 20 times or more), 42% had little experience (providing bCBT 1 to 4 times), and 17% had no experience.

80 | CHAPTER 4 Table 1. Therapist demographics.

Therapist demographics

N=36

Gender, female, n (%) Age in years, mean (SD; range) Professional background, n (%) Licensed psychologists Psychologists in training under supervision for health care psychologists Mental health nurses Other (e.g. psychiatrists, prevention workers) Professional experience, n (%) 0-2 years 3-4 years 5-9 years 10 years or more bCBT depression treatment experience, n (%) 0 times 1-4 times 5-9 times 10-19 times 20 or more times

29 (81) 38 (9.5; 24-60) 20 (56) 5 (14) 5 (14) 6 (16) 3 (8) 8 (22) 10 (27) 15 (42) 6 (17) 15 (42) 8 (22) 2 (6) 5 (14)

bCBT usability The mean SUS total score was 69.5 (SD 11.1; range 42.5–90.0). This score indicates sufficient usability of the bCBT platform. In the focus groups and interviews, therapists indicated that the usability had evolved over time. However, the current technical status still needs further improvement. For some, platform usability was a barrier to providing bCBT because many extra actions were required (e.g., logging in, finding the module, setting up the sessions). In addition, there was no integration with other (administrative) computer systems that therapists frequently use (e.g., electronic medical records).

Moreover, therapists felt hampered by the lack of knowhow and of a clear overview of all the functions and were unsure how to adapt the protocol. For example, it was unclear to them how to add sessions focused on panic attacks. Some therapists felt the platform usability was complicated and not intuitive.

Unexpected updates by the provider did not improve this. For therapists with little bCBT experience, the concern that they would be unable to assist patients with technical problems was also a major barrier. In addition, therapists had negative experiences with contacting the helpdesk (e.g., lack of expertise): “I trained myself to not contact the helpdesk anymore” [T23, experienced]. bCBT satisfaction On the CSQ-3, 77% of therapists stated that bCBT met all or almost all their needs; 94% were overall very or mostly satisfied with bCBT; and 97% would recommend bCBT in the future to their patients.

Interestingly, in the focus groups and interviews, the opinions of other therapists about satisfaction were divided: some thought that most therapists were generally satisfied, others said: “I also think that therapists are rather dissatisfied. bCBT isn’t used on a large scale. This means they must be dissatisfied, otherwise they would use it” [T28, no/little experience].

Qualitative Study: Therapists’ Perspectives| 81 Most of the therapists indicated that they would like to use bCBT in the future, but this depended on several preconditions that needed to be addressed: platform usability, their current work routine, and more guidelines on how to use bCBT. Factors that promote or hinder uptake of bCBT Regardless of the amount of bCBT experience, therapists were in favor of bCBT. Their views on the factors that promoted or hindered the use of bCBT in daily practice were structured along main themes identified in thematic analysis: 1) therapists’ needs regarding bCBT uptake, 2) therapists’ role in motivating patients for bCBT, and 3) therapists’ experiences with bCBT (See Table 2 for main and subthemes).

Table 2. Therapists’ bCBT uptake in routine practice, main themes and subthemes.

1) Therapists’ needs regarding bCBT uptake

2) Therapists’ role in motivating patients for bCBT

3) Therapists’ experiences with bCBT

Therapist training Therapist motivation Therapist readiness for uptake

Informing patients Patient eligibility Patient resistance

Effectiveness for depression Positive effects Negative effects Treatment protocol Therapeutic relationship Online feedback skills Drop-out & safety

Theme 1) Therapists’ needs regarding bCBT uptake This theme addresses factors related to therapists’ needs in providing bCBT to patients with depression.

The identified subthemes were: therapist training, therapist motivation, and therapist readiness for uptake (Table 3).

82 | CHAPTER 4 Table 3. Theme 1) Therapists’ needs regarding bCBT uptake, subthemes, influencing factors and illustrative quotes.

Subtheme

Influencing factors

Illustrative quotes

Therapist training

(+) sufficient to learn the technical aspects

“It is nice to have learned beforehand where all the buttons are and what the possibilities are.” [T11, no/little experience] “I miss therapeutic content in the training and the qualifications for the therapists to work with it.” [T25, experience] “There is one training and then you’re released with the notion: ‘now you can manage everything’.” [T7, experience] “There is a lack of time to delve into it, it should be facilitated better and not end with a single training.”

[T14, no/little experience]

(-) lack of therapeutic content and guidelines in the training (-) training alone is not enough for uptake in daily practice (-) lack of ongoing support

Therapist motivation

(-) demotivated therapists

(+) motivated therapists (-/+) undecided therapists

Therapist readiness uptake

for

“What bothers me most when it comes to blended is that it has been made mandatory, mainly because we’ve to meet a certain percentage of blended treatments.” [T34, experience] “If you are a psychologist, and you are interested in your profession, you should just use it.” [T9, experience] “You have to get out of your comfort zone to discover what the added value of blended is.” [T31, no/little experience]

(-) small select group uses bCBT on a regular basis

“When I look at my team, it seems only one person is trained, I don't really hear others talk about it.” [T36, experience]

(+) expected to grow in future

“I’d like to use blended in the future, but first I’d like more ideas on how to use it.” [T6, no/little experience]

(-/+) making bCBT mandatory

“I once heard that you need a certain percentage of online contacts - that would work for me.” [T2, experience]

Note. (-) factor that influenced the bCBT uptake negatively, (+) factor that influenced the bCBT uptake positively, (-/+) factor that influenced the bCBT uptake positively and negatively.

Therapist training Most therapists found the training formats sufficient for providing bCBT. The focus in the training formats was primarily targeted at technical aspects and less at therapeutic content. The therapists reported that they needed more guidelines and tools for providing online feedback, to keep patients engaged and to prevent them from deviating from the protocol. The training formats in their current form were probably sufficient for the ‘early adopters’ but less suited for therapists with limited computer skills, the older generation, or new graduates.

Moreover, undergoing training alone was not enough. Therapists found it difficult to familiarize themselves with the platform, and integrate bCBT into existing systems and into their daily workflow.

Time constraints were the most frequently cited barrier for uptake of bCBT. This was mainly due to a

Qualitative Study: Therapists’ Perspectives| 83 high caseload. Plus, therapists felt that the implementation ceased after the training was provided, and that the training was fragmented and was introduced too suddenly. Consequently, some organizations offered therapists a special consultation hour or extra training day. Yet, only a select group of motivated therapists turned up at these events. In order to integrate bCBT into their work routine, therapists needed more ongoing technical and content support in the form of supervision or peer review.

Above all, every qualified therapist was considered able to provide blended treatment, mostly because bCBT was perceived as being very similar to regular CBT. However, they also believed that therapist qualifications and professional background have so far been overlooked. They felt sufficient knowledge and professional experience of CBT should be a criterion for being trained in bCBT. Therapist motivation Therapists in this study were motivated for bCBT. Their colleagues could roughly be divided into three groups based on their motivation: demotivated therapists, motivated therapists, and an inexperienced group that needed more motivation.

Therapists estimated that about half of their therapist colleagues were not motivated for bCBT. They thought that perceived pressure from the MHOs and health care insurance companies worked counterproductively. Other reasons were varied. Some thought that a number of colleagues tended to resist every innovation implemented by the organization. For others, the reasons were fear-based: 1) fear that technology will eventually take over, 2) fear of losing contact when a patient is inactive on the platform, 3) fear of not offering patients the amount of help they need, and/or 4) fear that they would do something wrong or would come across as unprofessional towards patients. A specific group of therapists, who were enthusiastic at the start but due to negative experiences with (other/outdated/unguided) internet-based interventions, lost interest or became skeptical about the clinical effects of bCBT. This was mainly due to technical issues and higher patient drop-out rates.

Motivated therapists were described as being younger, having affinity with technology, and being interested in innovations in their field. Not every therapist was open to the idea of bCBT in the beginning, but some adjusted their view based on (positive) experiences.

Non-experienced therapists were mainly profiled as lacking computer skills, having technical skills insecurities, and belonging to an older generation with a traditional view on therapy. Interestingly, it was stated multiple times that many therapists were open to bCBT and also were capable of delivering it, but had reservations that first needed to be addressed. On top of a strong preference for FtF treatment over bCBT, there are thought to be many therapists to whom the possibilities of bCBT were unknown or remained unclear. They were often unaware of its added value or of research findings. For them, the need for blended did not come from within. However, even though there might be many reasons not to provide bCBT, therapists believed that only a small proportion are truly resistant to it.

Therapist readiness for uptake Though most therapists received training on the back of an organization-wide roll-out, a number observed that only a select group used bCBT on a daily base. To some, this was related to readiness for uptake: the implementation of bCBT in routine care is considered to be at an early, transitional stage.

Although it might be too innovative for therapists and patients, they believed that a change in perception toward bCBT will take place, and that readiness would increase as more patients start asking for bCBT.

84 | CHAPTER 4 There was disagreement with respect to making bCBT obligatory or not. For some, it would feed resistance toward bCBT and fuel the suspicion that bCBT was solely implemented for financial reasons.

Others believed that every therapist should ‘just use it’ or at least try it. It was also felt that treatment should be ‘blended, unless a therapist has limited computer skills, or a different therapeutic background or interest’. Most therapists stressed that for them, the precondition that blended stays blended is of great importance, because of the fear that MHOs in the future might gradually transition to online CBT. Theme 2) Therapists’ role in motivating patients for bCBT The second theme concerns the role of therapists in motivating patients with depression to use bCBT. The identified subthemes were: informing patients, patient eligibility, and patient resistance (Table 4).

Table 4. Theme 2) Therapists’ role in motivating patients for bCBT, subthemes, influencing factors and illustrative quotes.

Subtheme

Influencing factors

Illustrative quotes

Informing patients

(-) difficult to motivate some patients

“Sometimes there is so much going on, which leads me to inform patients in an insecure or doubtful way, that makes them say ‘no, I don’t want that’.” [T35, no/little experience]

(-) room for improvement

“I think that therapists should ‘sell’ or promote bCBT more with patients.” [T15, experience]

(-) eligible patient unknown

“It is difficult to assess for whom it is eligible. Sometimes someone who grew up with computers says, ‘oh no, not for me’, and then a 65-year old says ‘nice, let’s do it’.”

[T10, experience] “I imagine that blended is less suitable for more complex cases. But if I consider what I have in my caseload now, I think it should be doable .”[T5, no/little experience] “In principle everyone is eligible for bCBT. I rarely think that it’s not an option.” [T7, experience]

Patient eligibility

(-/+) discussion on comorbidity/complexity/depression severity (+) eligibility bCBT= eligibility CBT Patient resistance

(-) unclear image, too demanding, not enough, negative past experiences (+) patient demand expected to increase in the future

“Patients don’t have a good image of bCBT. In their view it’s more like an online questionnaire than a complete module, 'I’ve already answered those questions…'.” [T9, experience] “I think it will make an essential difference if patients ask for it. That’s just a matter of time.” [T1, no/little experience]

Note. (-) factor that influenced the bCBT uptake negatively, (+) factor that influenced the bCBT uptake positively, (-/+) factor that influenced the bCBT uptake positively and negatively.

Informing patients Many therapists had difficulties informing patients about bCBT for various reasons. Reasons given included the following: they were convinced that patients preferred FtF contact; blended was considered too demanding for patients; or they thought that patients would not need bCBT. Therapists

Qualitative Study: Therapists’ Perspectives| 85 thought that more effort could be put into convincing patients since the manner in which bCBT is offered could influence patients’ reactions.

Therapists found it a disadvantage that patients did not ask for bCBT or lacked a clear idea of what blended entails. Therefore, therapists believed it could be facilitating to offer it with enthusiasm, to explain the added value: ‘patients can do more in their own time, there is more information to their disposal, and they have more opportunities to practice, all in a secure environment’. It was indicated that there is a lot of room for improvement when it comes to informing patients. It helped, for example, if patients knew that it is their own therapist who communicates with them on the platform. In addition, to increase uptake, therapists should not offer it as a treatment option, but as the standard approach to treating depression. Some therapists offered blended as something extra, whereas others showed patients the platform during intake to facilitate the switch to the platform at a later point during treatment.

Patient eligibility Experienced therapists mentioned that the criteria for bCBT were very similar to the criteria for CBT. No specific patient bCBT profile or type existed; eligibility was found to be person specific, not predictable and sometimes even very surprising. Therapists with little or no bCBT experience gave many different criteria for patient eligibility, which were predominantly based on perceptions.

A contributing factor to difficulties with informing patients was the unknown typology of the eligible bCBT patient. Stated criteria for perceived non-eligibility were extensive: lack of computer skills, preference for FtF treatment, fear of failure, lack of illness insight, low cognitive capacities, limited language skills, insecure home environment, psychotic symptoms, suicidal ideation, personality disorder, trauma, and complex or severe depressive symptoms. Perceived eligible patient criteria were less elaborate: young age, having children, being employed, and mild to moderate depressive symptoms.

Two interesting discussions took place about patient eligibility. Therapists disagreed about patients who, besides depression, had autism, avoidance issues, limited social contacts, poor concentration skills, or had undergone CBT for depression in the past. Some therapists observed that these factors prevented patients from engaging with bCBT, whereas others experienced that these factors made bCBT explicitly eligible for these patients as well. Another point of discussion was the severity of the depressive symptoms. Some therapists thought that bCBT was suitable for patients with complex, severe problems, while others did not: it would be harder to activate them than patients with less complex and more moderate symptoms. They would need more tools and explanations than bCBT can possibly offer them.

Patient resistance Several reasons for patient resistance toward bCBT were observed: unclear bCBT image, misfit with the traditional image of therapy among patients who had CBT in the past, fear of not receiving the right amount of help, and thinking that the online component was too demanding. Therapists’ found that some patients were more difficult to motivate because they were inclined to generalize negative experiences with previous internet-based treatments (often unguided treatments in primary care) to all other internet-based interventions. However, most therapists, especially the experienced ones,

86 | CHAPTER 4 thought the vast majority of patients were positive about bCBT when offered. Most therapists agreed that patient demand will increase in the future.

Theme 3) Therapists’ experiences with bCBT Within this theme, experiences with providing bCBT in routine care are discussed. The identified subthemes were: effectiveness for depression, positive effects, negative effects, treatment format, therapeutic relationship, online feedback skills, drop-out, and safety (see Table 5).

Effectiveness for depression Therapists agreed that bCBT offered a good foundation for treating patients with depression.

Nevertheless, they also found that the therapeutic content of the online sessions needed improvement, especially for patients with severe depressive symptoms who were characterized by more inactivity, complex problems, and longer treatment history. Whether ‘the blended’ treatment format exists, remained a question. Not based on research findings, but based on their experience, therapists uniformly thought that bCBT and FtF CBT could be equally effective. bCBT mainly helped to structure treatment and both patient and therapist to remain focused.

Positive effects The most cited positive effect was containing therapist drift; platform functionalities facilitated therapist (and patient) focus and protocol adherence more than in regular CBT because of the pre-set structure of bCBT. Therapists found it easier to control homework assignments and patients knew what was scheduled next. It was believed that all this positively impacted the quality of the treatment. An additional effect was that other diagnoses besides depression (e.g., autism) or certain problems (e.g., suicidal ideation, non-adherence, low cognitive capacities) were made more visible to the therapists at an earlier treatment stage. This was due to a discrepancy between online symptom monitoring, patients’ online expressions, and what was said and shown in FtF sessions. Plus, for therapists, it was more noticeable from the patients’ writing skills what the cognitive abilities were.

In the therapists’ experience, patients found it easier to remember what was discussed in FtF sessions with bCBT. The information on the platform supported therapists—they had more information available, not only about depression but on other treatment options as well (e.g., sleep, self-image). In addition, because patients with depression often have concentration problems, the unlimited and easy access to treatment information and psychoeducation was very useful.

Reducing travel-time was mentioned as being very convenient for patients, not only for practical reasons (e.g., having children, employment), but also for therapists. bCBT provided an extra tool to stay connected with patients, especially in cases of no-show. Therapists discovered that patients shared the platform content with their immediate circle more easily (e.g., partner, family, or friends). Plus, the effect of writing on the online platform was a big advantage to patients. Patients wrote down their problems, came to new insights, and shared shameful problems more easily because of the distance effect and having time for reflection. bCBT offered patients greater responsibility. As for self-efficacy, some therapists thought that patients tended to attribute the treatment success to themselves more than in FtF treatment.

Qualitative Study: Therapists’ Perspectives| 87 Table 5. Theme 3) Therapists’ experiences with bCBT, subthemes, influencing factors and illustrative quotes.

Subtheme

Influencing factors

Illustrative quotes

Effectiveness for depression

(+) effectiveness bCBT= effectiveness of CBT

“It is CBT in another format, it’s not suddenly a different treatment.”

[T29, experience]

Positive effects

(+) containing therapist drift (+) making other diagnosis or problems more easily visible (+) helps to remember information more easily (+) reducing travel time

“I don’t skip things. With bCBT, I think that I am much more thorough than without.” [T29, experience] “You notice much sooner if a patient has difficulties adhering to therapy, whether or not they are doing the work. This becomes clearer faster than in a FtF session.” [T1, no/little experience]

(+) more contact with patient (+) sharing content with system (+) effect of writing (+) monitoring homework (+) facilitating patient selfefficacy

“In a FtF session you say a lot, but afterwards patients also forget quickly.

Now they can review what they’ve learned as many times as they want.”

[T10, experience] “In other countries, factors such as distances and climate play a role, but of course in the middle of the country we have traffic jams. bCBT is very nice for this.” [T9,experience] “Throughout the treatment you have more frequent short contact moments, which are actually very good.” [T10, experience] “For the patient you remove the treatment from therapist’s office to their own home.” [T12, no/little experience].

“Patients have to write, this is for many a very good instrument to order things and gain new insights. I find the writing a very positive addition to the verbal component.” [T15, experience] “I find it very attractive that there is more control over the homework that they have or haven’t done.” [T2, experience] “In the online part a degree of independence makes patients themselves engage with the treatment. I think that they attribute treatment success to themselves, more than with FtF treatment.” [T22, experience]

Negative effects

(-) no gain in time, costs more (-) unsuitable content

“I find it takes more time than I expected. Especially if you’re still unfamiliar with the module .” [T7, no/little experience] “I had a traumatized patient, because she couldn’t have children. The first example on the platform was ‘I have children and I want to be a good mother ...'.” [T27, no/little experience]

Treatment format

(-/+) discussion on structure protocol (-/+) discussion on freedom in protocol to

“I find it difficult to be flexible in the protocol and not to become a sort of CBT machine.” [T21, no/little experience] “Usually, when I start without the platform, I think ‘never mind’. But also, when the patient isn’t ready for it initially, you can say ‘Later on, we will start with the online platform’.” [T31, no/little experience]

88 | CHAPTER 4 explore depression (-/+) discussion on ‘what is blended’ (-/+) discussion on introduction platform

“I don't think it's 50/50. The main focus is more on the FtF sessions. I notice that the conversations and the modules do not always run parallel to each other.” [T14, no/little experience] “Whether you start treatment with an online module, right at the outset or only when a basis has been formed, is dependent on the therapist. I think am very comfortable with starting right away.” [T1, no/little experience]

Therapeutic relationship

(-/+) discussion on quality therapeutic relationship

“It’s possible to build a therapeutic relationship, but sometimes it isn’t.

Yet I think that this isn’t dependent on bCBT. Also, with FtF therapy it’s possible to succeed in building a therapeutic relationship and sometimes not.” [T15, experience]

Online feedback skills

(-) time providing feedback (-) permanent character of online feedback

“I reflect more on what I want to write, that is perhaps the reason why it takes more time.” [T7, no/little experience] “It’s difficult to assess whether someone has understood something in the way you intended. If this happens in FtF contact you can talk around it, look friendly and explain. In black and white this is not possible.” [T5, no/little experience] “I notice that patients like to come back to a point during the conversations, to go through the exercises together.” [T12, no/little experience]

(+) connection online feedback and FtF conversations Dropout & safety

(-/+) discussion on safety risks (+) bCBT has no extra risks with regards to suicidal ideation (+) drop-out bCBT= drop-out CBT (+) sufficient patient data safety

“In the meantime, I experienced that it’s okay if the patient also comes to the FtF sessions. You’ve to make this clear at the start. The risk you lose contact is lower than I expected.” [T9, experience] “I thought that if you see someone less that it’d be more difficult to assess the suicide risk. But it can just as well lower the threshold to say something online more easily than to discuss FtF.” [T31, no/little experience] “I don’t have one type of patient that drops out of blended. It’s very unpredictable who completes it.” [T30, experience] “I understood that the online platform is very safe, safer than email.”

[T33, experience]

Note. (-) factor that influenced the bCBT uptake negatively, (+) factor that influenced the bCBT uptake positively, (-/+) factor that influenced the bCBT uptake positively and negatively.

Negative effects Most therapists questioned whether bCBT could shorten treatments and solve waiting list problems.

First, therapists spent a lot of time getting to know the platform and secondly, providing therapy via the online environment was time-consuming in contrast to expectations. Perhaps it might be timesaving in the future. However, the common thought was that bCBT is ‘just another way of doing the same thing’. Some felt that there is a long way to go before blended interventions are considered equally effective as regular CBT by therapists in general.

Qualitative Study: Therapists’ Perspectives| 89 Some judged the content on the platform to be too textual, too complex, or too superficial, thus requiring repeat sessions and extra clarification. Moreover, sometimes patients did not recognize themselves in the text and video examples; some found the content pejorative, overly directive, or too intense, resulting in drop-out or even worsening of their mood. Therefore, some therapists felt a strong need for personalization of the content to adapt to the patients’ preferences and background.

Treatment format While experienced therapists unanimously stated that the pre-set structure of the blended protocol was very beneficial, for therapists with less experience, this was not the case. For them, the protocol was too structured and therefore inflexible. Experienced therapists, who had a better overview of all the functionalities, mentioned that bCBT was very similar to FtF CBT, in which you also adapt to the protocol when needed.

Therapists had different views on the ratio of FtF versus online sessions within bCBT. Some regarded and used bCBT as an addition to the regular number of FtF sessions. Moreover, for some the question ‘what is blended?’ remained. Therapists had different views on degrees of ‘blendedness’.

Repeatedly, it was seen as an interchange of FtF sessions with the functionalities of the online platform.

For some patients 50%–50% was considered beneficial, for others 90% FtF and 10% online was more appropriate.

Another issue was the time point at which the online platform was introduced. Some therapists preferred to start with weekly FtF sessions and introduced the platform at a later date. For them, it felt unnatural and too distracting for patients to meet every two weeks instead of weekly in the first phase of treatment. They wanted more time to establish a foundation, especially for patients who lacked insight into their depression. Other therapists, who thought it would be better to start with the online platform at once, reasoned that the threshold (for patients and therapists) may be higher when the platform is introduced at a later stage. These therapists also imagined that it motivated patients to be focused from the beginning with the support of the platform.

Therapeutic relationship Regarding the quality of a therapeutic relationship in bCBT, therapists’ opinions were divided. In the beginning, some doubted whether it was possible to develop a therapeutic relationship because they would see patients less FtF, but changed their opinion later on. In some therapists’ experience, it was challenging to build a therapeutic relationship compared to regular CBT; when a patient experienced difficulties with the online platform and asked more attention from the therapist, this frustrated the contact because it was at the expense of the time they had FtF. Nevertheless, others experienced that the relationship in bCBT was similar or even better compared to FtF treatment. bCBT reinforced the relationship, since there was more frequent contact throughout. In addition, the treatment course was more visible to therapists because of online monitoring and this also positively influenced the relationship. Importantly, as in every treatment, the relationship was mainly perceived as more dependent on patient motivation and activation than other factors such as online contact.

Online feedback skills Therapists agreed that every therapist is able to provide online feedback. However, learning how to provide feedback took time and for many, felt like ‘a job in its own right’. They were needed for tips,

90 | CHAPTER 4 tools, and examples; how to address something negative, how to handle or prevent miscommunication, how to write in a concise manner, how to make feedback personalized and adapt to different patient types. In contrast to FtF sessions, where therapists are able to correct themselves, everything that is written online, has a permanent character: for some, online feedback thus felt uncomfortable.

Therapists stressed the importance of connecting to the language of the patient. They found they had to be careful not to use difficult terminology or be too distant in their language. Most of all, they had to plan sufficient time to write feedback. Therapists noticed that writing online feedback became faster with more experience. Connecting online feedback and topics in FtF sessions and vice versa, was experienced as helpful and was done quite frequently.

Drop-out and safety Two safety risks cited predominantly by inexperienced therapists were the suggestion of 24/7 accessibility to patients and not being able to see how patients reacted to therapist feedback or other online information. Experienced therapists did not mention this or had learned that this was not an issue.

Regarding a much-discussed risk of suicidal ideation, therapists experienced that when there was a clear agreement with patients on not expressing crisis situations on the platform, bCBT had no extra risks compared to FtF treatment. Importantly, it even gave therapists more rapid awareness of whether patients were losing contact. In addition, bCBT provided therapists with an extra preventive communication tool.

Furthermore, experienced therapists reported that there was no specific type of patient that dropped out. Sometimes therapists saw patients who started enthusiastically, but dropped out easily.

Importantly, they thought this was no different than with regular CBT: patients dropped out because of certain patient characteristics or other problems than for bCBT-related issues.

Overall, therapists expressed that safety of patient data on the online platform is guaranteed sufficiently. Sharing patient data with, for example, a helpdesk, is still something that therapists were reluctant to do and prevented them from contacting the helpdesk when technical issues occurred. They often then tried to solve their issues with colleagues, but often also broke off the online sessions. Some patients seemed too lax when it came to their data safety, and some were distrustful. Therapists emphasized that it is their responsibility to explain how safe the online platform is, but that this was not clear to all therapists.

Discussion In this study, therapists expressed overall satisfaction with providing bCBT to patients with depression.

The perceived usability of the online platform was sufficient. This was also found by other studies exploring patients’ perspectives on usability and satisfaction [21,36]. This study showed that therapists see room for improvement with regard to platform usability (e.g., lack of integration with other administrative systems), therapists’ work routines, and guideline use on bCBT. From interviews in this study, it became clear that there are specific barriers that prevented therapists from providing bCBT to patients with depression, such as a lack of ongoing support for technical and clinical issues with bCBT. In addition, therapists reported several practical and therapeutic challenges in routine care such as not experiencing timesaving. Nevertheless, there have been factors that might have influenced the uptake positively as well.

Qualitative Study: Therapists’ Perspectives| 91 Barriers to bCBT uptake Less experienced bCBT therapists felt several barriers prevented them from providing bCBT. First, they found that the training was not sufficient to enable them to adopt bCBT into their work routine. This was partly because the training mainly covered technical aspects and did not provide sufficient guidance on how to work in a blended way with the protocol and how to communicate online. In addition, after undergoing training, ongoing support and attention for bCBT was limited or absent. This was also a reported therapist barrier by Folker et al. [37] who identified implementation challenges perceived by therapists and managers of iCBT in routine care settings in five European countries.

Moreover, inexperienced therapists were unsure about the indication for bCBT and the eligibility of the patients. A third issue was that some therapists experienced low patient demand for bCBT. A fourth hindering factor was that for a number of the therapists, the added value of bCBT in terms of clinical effectiveness remained unclear; for others, fear (e.g., of doing something wrong) or distrust (e.g., regarding the intentions of health insurance companies) stood in the way of considering bCBT. Challenges with bCBT Once therapists started working with bCBT, they experienced several challenges. They had difficulty integrating bCBT into their daily workflow. Not only because of technical disconnections with existing IT-systems for patients’ administration, but also because of uncertainties regarding the bCBT protocol and logistic integration into their daily therapeutic and administrative schedules. Besides, therapists lacked the conviction that bCBT saved time; for some, it even generated a higher workload, especially in the beginning, and providing online feedback is considered time-consuming as well. It can be estimated, that on average per feedback message, 30 min of therapist time is needed [13]. Internal helpdesks were unable to support therapists sufficiently which even discouraged some from further providing bCBT. Moreover, therapists displayed different interpretations of the protocol which partly resulted for some in using the online platform on top of the FtF treatment instead of replacing a portion of the FtF sessions. This was previously reported in a naturalistic study on bCBT uptake in routine care as well [38], making the implementation more costly than intended. Kenter et al. [38] argued that their therapists may have needed more extensive training to master specific bCBT skills (e.g., providing online feedback) and that clear guidelines on how to use bCBT in routine practice by the MHO would have helped the therapist to provide bCBT more efficiently. Plus, using the online platform on top of the FtF treatment may result in patient dissatisfaction. A study on patients’ experiences with bCBT showed that a diminished interplay between the online and FtF sessions was unsatisfactory for patients, as therapists were less aware of patient activity on the online platform and limited time was made available to discuss the online activity in the FtF sessions [39].

Advantages of bCBT One of the main reported advantages of bCBT was the focus and pre-set structure that made therapists and their patients more adherent to the protocol and contained therapist drift. This was also experienced by several therapists who worked with bCBT [14,19] or blended group therapy [40] for depression. Although content improvement (e.g., online patient examples) for severely depressed patients was considered necessary, therapists agreed that bCBT is a good treatment format for the (complex) patient group therapists see in daily practice. They found that bCBT was in many ways similar

92 | CHAPTER 4 to FtF CBT when it comes to patient eligibility, effectiveness, therapeutic relationship, and drop-out: ‘it is CBT in another format’. Many believed that the implementation phase is still in its infancy; it might be a long way to go before all therapists accept blended interventions as being equally effective to FtF therapy. This idea is also partly justified since the first studies on clinical effects on blended interventions show that it is at least as effective [12,13]. Nonetheless, it is thought that bCBT will eventually be included in care pathways in the future because of the perceived benefits for patients and therapists (e.g., pre-set structure, number of flexible contact moments on demand, information access); an important precondition is to continue to be combined with FtF contact, as providing standalone guided iCBT might not yet be realistic for most therapists in routine practice, where FtF treatment is the norm Level of experience In our study, therapists with bCBT experience reported different views on safety, flexibility and personalization of the protocol, patient eligibility, and therapeutic relationship than therapists who lacked experience. For example, non-experienced therapists assumed that there were many reasons why patients would be ineligible for bCBT, such as severe depressive symptoms, or found this difficult to assess. By contrast, more experienced therapists stated that it was impossible to predict eligibility; in principle, any patient could be offered bCBT. This may indicate that with experience, perceptions on these important clinical factors of bCBT will change and that when promoting uptake, therapists’ needs can shift. This is consistent with Feijt et al. [41] who showed that there are different barriers and facilitators depending on the level of therapist experience with online services, and that potentially, the experience level has unique requirements to be addressed when it comes to the uptake of internetbased interventions.

Limitations and strengths Although some therapists in this study were skeptical about the effectiveness of bCBT or had negative experiences, all were quite open-minded toward bCBT. Selection bias in our sample is possible, as untrained therapists or those not interested in participating in the study were not reached and therapists were identified through their team managers, and other specially appointed members of different teams. This could have diminished the positive view on bCBT. Another point is the limited transferability of our findings to other national contexts. In the Netherlands, the digital infrastructure is quite good and MHO’s and therapists have the experience that every patient has proper equipment. Moreover, bCBT is covered by health care insurance companies in the same way as face-to-face CBT. Compared to other countries, the Netherlands and others, including Sweden and the United Kingdom, can be considered a frontrunner in implementing bCBT within routine practice [16]. On the other hand, our findings can be taken into account by the ‘followers’ in the earlier stages of developing, testing, and implementing bCBT. A strength of this study is the usage of data triangulation to record the perspective of a diverse group of therapists, with different professional backgrounds, differences in experiences with bCBT for depression, and exposure to various implementation strategies from a number of mental health care organizations.

Qualitative Study: Therapists’ Perspectives| 93 Future uptake of bCBT To speed up bCBT uptake, it is important to identify how barriers can be overcome, challenges solved, and how successes in routine care can be strengthened. From the field of implementation research, it is known that implementation strategies for innovations in mental health care need ‘to address multiple levels and barriers to change, (and) are interwoven and packaged as a protocolized or branded implementation intervention’ [42]. Based on this and on the findings of this study, the following strategies can be considered in practice and in research to move forward in the implementation of bCBT: 1) Dissemination strategies: to change the focus of cost-effectiveness of bCBT towards personalization, adherence, and treatment quality improvement in disseminating information to therapists and patients. Within organizations, the uptake might gain from informing and motivating therapists and subsequently patients as well. There is a lot of room for improvement when it comes to sufficient knowledge about bCBT. Outside the organizations, patient demand can be stimulated by general practitioners and health insurance companies. Further, although evidence-based knowledge on bCBT is slowly becoming more available, this should be more widespread among patients, therapists, MHOs, and health insurance companies.

2) Top down strategies: bCBT must be embedded top-down from start to finish overcoming its mere project status in many of the MHOs. This could be achieved by intensifying and extending training beyond technical aspects, and by facilitating the integration of ongoing support into existing consultation structures, (instead of creating special consultation options), in order to exchange knowledge and experience with bCBT. It would also help to remind therapists to offer bCBT to patients and to solve technical disconnections with administration- and other IT-systems within a well-equipped help desk. Whether imposing minimum usage norms (e.g., that 25% of the treatments must be blended in the Netherlands) could have a promoting effect is unclear, as this causes resistance for some, but motivated others.

3) Bottom-up strategies: Fragmented use and individual uptake of the blended protocol can create a risk of it being costlier and of leading to a protocolled manner of working. Therapists’ need to personalize modules might be a result of uncertainty and their limited knowledge of all functionalities within the module and platform. Thus, besides a top-down approach, therapists might be more motivated and skilled if uptake is stimulated on a bottom-up basis, through discussing it on the work floor: For instance, it could help to start implementation with a small rollout with a motivated team of therapists, focusing on content development, enhancing online feedback skills and protocol use; to let the therapists be (co-) creators of modules; to construct a place and time within MHOs where therapists can think of ways to further develop bCBT; to experiment; to keep lines with the technical innovations on the platform within reach; including recurring evaluation in order to adapt strategies on the level of therapist experience and updates of the platform.

Future research Testing strategies for uptake would be a fruitful area for further research. The European project, ImpleMentAll (www.implementall.eu), will hopefully provide routine practice with more knowledge as this is uncharted territory for Internet interventions in mental health care [43]. In ImpleMentAll,

94 | CHAPTER 4 personalized implementation strategies will be tested to facilitate the use of internet-based and blended interventions in routine care in several different countries.

In addition, it would support uptake of bCBT to further investigate the ratio and integration of online and FtF sessions. So far, research has focused on an equal distribution. However, it is reasonable to think that depending on the severity of the symptoms, the proportion of blended could be adapted.

Moreover, it would be interesting for future studies to also look at the patient perspectives on blended care, preferably done using dyad interviews with their therapists to generate a rich, deeper understanding of the relevant factors in bCBT uptake.

Conclusions Overall, the therapists in this study were satisfied with providing bCBT to patients with depression. That said, a large group of therapists is still wondering how much FtF contact should be included in blended.

Plus, important preconditions for implementation were unmet and the technical infrastructure is not free from teething problems. It cannot be expected from therapists that uptake happens spontaneously. It can be considered positive that having bCBT experience can be a possible key to change a therapist’s view on important factors such as eligibility and drop-out. Therapists found that there is a good base that can be further developed in terms of ongoing technical support, therapeutic support from their peers and supervisors, while at the same time strategically implementing bCBT on an organizational level to facilitate therapists in the transition of integrating bCBT into their daily practice.

Funding This research was partially funded under the Information and Communications Technologies (ICT) Policy Support Programme as part of the Competitiveness and Innovation Framework Programme (CIP) by the European Community, grant number 621000.

Acknowledgments The authors would like to thank several people for their contributions: Thuur Smet (data extraction), Nanda Mooij, Bep Verkerk (data management), Martine de Meijer (transcribing interviews). We also would like to thank the therapists and the participating mental health care organizations for their time and effort.

Qualitative Study: Therapists’ Perspectives| 95

References [1]

[2] [3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

Cuijpers, P., Noma, H., Karyotaki, E., Cipriani, A., & Furukawa, T.A. (2019). Effectiveness and Acceptability of Cognitive Behavior Therapy Delivery Formats in Adults with Depression: A Network Meta-analysis. JAMA Psychiatry; 76, 700–707.

National Institute for Clinical Excellence (2009). Depression: The treatment and management of depression in adults. In Clinical Guideline 90, NICE: London, UK. World Health Organization (2017). Depression and Other Common Mental Disorders: Global Health Estimates. In World Health Organization Report, World Health Organization: Geneva, Switzerland; 1–24.

Boerema, A.M., Kleiboer, A., Beekman, A.T.F., van Zoonen, K., Dijkshoorn, H., & Cuijpers, P. (2016). Determinants of help-seeking behavior in depression: A cross-sectional study. BMC Psychiatry; 16.

Andrews, G., Basu, A., Cuijpers, P., Craske, M.G., McEvoy, P., English, C.L., & Newby, J.M. (2018).

Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: An updated meta-analysis. J. Anxiety Disord; 55, 70–78.

Karyotaki, E., Riper, H., Twisk, J., Hoogendoorn, A., Kleiboer, A., Mira, A., MacKinnon, A., Meyer, B., Botella, C., Littlewood, E., et al. (2017). Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms a meta-analysis of individual participant data. JAMA Psychiatry; 74, 351–359.

Titov, N., Dear, B., Nielssen, O., Staples, L., Hadjistavropoulos, H., Nugent, M., Adlam, K., Nordgreen, T., Bruvik, K.H., Hovland, A., et al. (2018). ICBT in routine care: A descriptive analysis of successful clinics in five countries. Internet Interv; 13, 108–115.

Erbe, D., Psych, D., Eichert, H.C., Riper, H., & Ebert, D.D. (2017). Blending face-to-face and internet-based interventions for the treatment of mental disorders in adults: Systematic review. J. Med. Internet Res; 19.

Hadjistavropoulos, H.D., Nugent, M.M., Dirkse, D., & Pugh, N. (2017). Implementation of internet-delivered cognitive behavior therapy within community mental health clinics: A process evaluation using the consolidated framework for implementation research. BMC Psychiatry; 17.

Definition Blended Treatment Heleen Riper. Triple-E Website. Available online:https: //www.triple-ehealth. nl/en/projects-cat/depressie/ and http: //www.webcitation.org/6wZFBY9pX(accessed on 18 January 2018).

Kleiboer, A., Smit, J., Bosmans, J., Ruwaard, J., Andersson, G., Topooco, N., Berger, T., Krieger, T., Botella, C., Baños, R., et al. (2016). European COMPARative Effectiveness research on blended Depression treatment versus treatment-as-usual (E-COMPARED): Study protocol for a randomized controlled, non-inferiority trial in eight European countries. Trials; 17.

Romijn, G., Batelaan, N., Koning, J., van Balkom, A., de Leeuw, A., Benning, F., Hakkaart-van Roijen, L., & Riper, H. (2019). Effectiveness and cost-effectiveness of blended cognitivebehavioural therapy for anxiety disorders in specialised mental health care: A 15-week randomised controlled trial with 1-year follow-up. Submitted for publication.

Kooistra, L.C., Wiersma, J.E., Ruwaard, J.J., Neijenhuijs, K., Lokkerbol, J., Smit, F., van Oppen, P., & Riper, H. (2019). Costs and effectiveness of blended versus standard cognitive behavioral

96 | CHAPTER 4

[14]

[15]

[16]

[17]

[18]

[19]

[20] [21]

[22]

[23] [24]

[25] [26] [27]

therapy for depressed outpatients in routine specialized mental healthcare: A pilot randomized controlled trial. J. Med. Internet Res; 21.

Mansson, K.N., Skagius Ruiz, E., Gervind, E., Dahlin, M., & Andersson, G. (2013). Development and initial evaluation of an internet-based support system for face-to-face cognitive behavior therapy: A proof of concept study. J. Med. Internet Res; 15, e280.

Thase, M.E., Wright, J.H., Eells, T.D., Barrett, M.S., Wisniewski, S.R., Balasubramani, G.K., McCrone, P., & Brown, G.K. (2017). Improving the Efficiency of Psychotherapy for Depression: Computer-Assisted Versus Standard CBT. Am. J. Psychiatry; 175.

Topooco, N., Riper, H., Araya, R., Berking, M., Brunn, M., Chevreul, K., Cieslak, R., Ebert, D.D., Etchmendy, E., Herrero, R., et al. (2017). Attitudes towards digital treatment for depression: A European stakeholder survey. Internet Interv; 8, 1–9.

Vis, C., Kleiboer, A., Prior, R., Bønes, E., Cavallo, M., Clark, S.A., Dozeman, E., Ebert, D., Etzelmueller, A., Favaretto, G., et al. (2015). Implementing and up-scaling evidence-based eMental health in Europe: The study protocol for the MasterMind project. Internet Interv; 2.

Mol, M., Dozeman, E., van Schaik, D.J.F., Vis, C.P.C.D., Riper, H., & Smit, J.H. (2016). The therapist’s role in the implementation of internet-based cognitive behavioural therapy for patients with depression: Study protocol. BMC Psychiatry; 16, 338.

Titzler, I., Saruhanjan, K., Berking, M., Riper, H., & Ebert, D.D. (2018). Barriers and facilitators for the implementation of blended psychotherapy for depression: A qualitative pilot study of therapists’ perspective. Internet Interv; 12, 150–164.

Denzin, N.K. (2012). Triangulation 2.0 *. J. Mix. Methods Res; 6, 80–88.

Kooistra, L.C., Ruwaard, J., Wiersma, J.E., van Oppen, P., van der Vaart, R., van Gemert-Pijnen, J.E.W.C., & Riper, H. (2016). Development and initial evaluation of blended cognitive behavioural treatment for major depression in routine specialized mental health care. Internet Interv; 4, 61–71.

Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation). Official Journal of the European Union L 119/1. Available online:http: //data.europa.eu/eli/reg/2016/679/oj (accessed on 10 December 2019).

Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability Eval. Ind; 189, 4–7.

Mol, M., van Schaik, A., Dozeman, E., Ruwaard, J., Vis, C., Ebert, D.D., Etzelmueller, A., Mathiasen, K., Moles, B., Mora, T., et al. (2019). Dimensionality of System Usability Scale among professionals using internet-based interventions for depression: A confirmatory factor analysis.

Submitted for publication.

Sauro, J. (2011). A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices, Measuring Usability LLC: Denver, CO, USA. Sauro, J. & Lewis, J.R. (2016). Quantifying user research. In Quantifying the User Experience, Elsevier: Amsterdam, The Netherlands; 9–18.

Attkisson, C.C. & Greenfield, T.K. (1995). The Client Satisfaction Questionnaire (CSQ) Scales.

Outcome Assessment in Clinical Practice, Williams & Wilkins: Baltimore, MD, USA.

Qualitative Study: Therapists’ Perspectives| 97 [28]

[29] [30]

[31]

[32]

[33]

[34] [35] [36]

[37]

[38]

[39]

[40]

[41]

Boß, L., Lehr, D., Reis, D., Vis, C., Riper, H., Berking, M., & Ebert, D.D. (2016). Reliability and validity of assessing user satisfaction with web-based health interventions. J. Med. Internet Res; 18, e234.

Glasgow, R.E., Vogt, T.M., & Boles, S.M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. Am. J. Public Health; 89, 1322–1327.

Kidholm, K., Ekeland, A.G., Jensen, L.K., Rasmussen, J., Pedersen, C.D., Bowes, A., Flottorp, S.A., & Bech, M. (2012). A model for assessment of telemedicine applications: Mast. Int. J. Technol.

Assess. Health Care; 28, 44–51.

May, C.R., Mair, F., Finch, T., MacFarlane, A., Dowrick, C., Treweek, S., Rapley, T., Ballini, L., Ong, B.N., Rogers, A., et al. (2009). Development of a theory of implementation and integration: Normalization Process Theory. Implement. Sci; 4.

Kivi, M., Eriksson, M.C.M., Hange, D., Petersson, E.L., Björkelund, C., & Johansson, B. (2015).

Experiences and attitudes of primary care therapists in the implementation and use of internetbased treatment in Swedish primary care settings. Internet Interv; 2, 248–256.

Wilhelmsen, M., Hoifodt, R.S., Kolstrup, N., Waterloo, K., Eisemann, M., Chenhall, R., & Risor, M.B. (2014). Norwegian general practitioners’ perspectives on implementation of a guided web-based cognitive behavioral therapy for depression: A qualitative study. J. Med. Internet Res; 16, e208.

Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology. Qual. Res. Psychol; 3, 77– 101.

Fleiss, L., Levin, B., & Paik, M.C. (2003). The measurement of interrater agreement. In Statistical Methods for Rates and Proportions, 3rd ed., John Wiley & Sons, Inc.: Hoboken, NJ, USA. Etzelmueller, A., Radkovsky, A., Hannig, W., Berking, M., & Ebert, D.D. (2018). Patient’s experience with blended video- and internet based cognitive behavioural therapy service in routine care. Internet Interv; 12, 165–175.

Folker, A.P., Mathiasen, K., Lauridsen, S.M., Stenderup, E., Dozeman, E., & Folker, M.P. (2018).

Implementing internet-delivered cognitive behavior therapy for common mental health disorders: A comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interv; 11, 60–70.

Kenter, R.M.F., van de Ven, P.M., Cuijpers, P., Koole, G., Niamat, S., Gerrits, R.S., Willems, M., & van Straten, A. (2015). Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: Results from a naturalistic study. Internet Interv; 2, 77–83.

Urech, A., Krieger, T., Möseneder, L., Biaggi, A., Vincent, A., Poppe, C., Meyer, B., Riper, H., & Berger, T. (2018). A patient post hoc perspective on advantages and disadvantages of blended cognitive behaviour therapy for depression: A qualitative content analysis. Psychother. Res; 29, 986–998.

Schuster, R., Sigl, S., Berger, T., & Laireiter, A.R. (2018). Patients’ Experiences of Web- and Mobile-Assisted Group Therapy for Depression and Implications of the Group Setting: Qualitative Follow-Up Study. JMIR Ment. Health; 5, e49.

Feijt, M.A., De Kort, Y.A.W., Bongers, I.M.B., & IJsselsteijn, W.A. (2018). Perceived drivers and barriers to the adoption of eMental health by psychologists: The construction of the levels of adoption of eMental health model. J. Med. Internet Res; 20.

98 | CHAPTER 4 [42]

[43]

Powell, B.J., McMillen, J.C., Proctor, E.K., Carpenter, C.R., Griffey, R.T., Bunger, A.C., Glass, J.E., & York, J.L. (2012). A compilation of strategies for implementing clinical innovations in health and mental health. Med. Care Res. Rev; 69, 123–157.

BĂźhrmann, L., Schuurmans, J., Ruwaard, J. et al. Tailored Implementation of internet-based Cognitive Behavioural Therapy in the Multinational Context of the ImpleMentAll Project: A Study Protocol for a Stepped Wedge Cluster Randomized Trial. Submitted for publication.

Qualitative Study: Therapists’ Perspectives| 99

Appendix 1: Topic list focus groups 1. Introduction This focus group is being held in the context of the MasterMind research. MasterMind stands for: MAnagement of mental health diSorders Through advancEd technology and seRvices-telehealth for the MIND. MasterMind is a large European study. It is an implementation study in which cohorts of patients are followed in 11 countries who receive internet treatment for depression. Expectations are high and the results are promising, but internet treatments are still used relatively little in practice. We therefore want to investigate how this treatment can best be applied, which factors are facilitating and which ones actually get in the way of implementation.

In the Netherlands, we are investigating how blended CBT treatment is applied in basic and specialist mental healthcare for patients with depression older than 18 years. Data is collected from 300 patients, practitioners and the representatives of the mental healthcare organizations, including your organization.

The aim of the focus group is to provide insight into the experiences and ideas of you as a blended therapist.

1.1. One word 

[Question mobile voting system] What do you think is, in one word, the most important thing when implementing eHealth in mental health care (open answer)?

1.2. Patient factors 

[Statement mobile voting system]: Patients need the blended treatment for depression (yes/no).

The patient's need: 

Which needs of the patient does the blended treatment meet?

- What is the most important aspect of this?



What makes the blended treatment attractive to patients?



Which needs of the patients does blended treatment not meet?



[Question mobile voting system]: How suitable is the blended treatment for patients with depression? (number 1-10)

Patient profile: 

Which patients start this treatment?



Which patients can benefit from the blended treatment?



Which patients drop out with the blended treatment?



What are the most common reasons for patients not to complete the blended treatment?



[Statement mobile voting system]: There are many safety risks for the patient associated with the blended treatment (yes/no).

Safety 

What are the risks associated with blended treatment?

100 | CHAPTER 4 

Is the safety of the patient sufficiently guaranteed?

1.3. Therapist factors 

[Statement mobile voting system]: Therapists need the blended treatment for depression (yes/no).

The therapist’s need: 

Which needs of the therapist does the blended treatment meet (possible gain of time, structure of treatment, influence of therapeutic relationship) - What is the most important aspect of this?



What makes the blended treatment attractive to therapists?



Which needs of the therapist does the blended treatment not meet?



What do you need to make blended treatment part of your treatment (more time/technical support/training)?



Does the blended treatment make you perform your work as a therapist differently than before?



[Statement mobile voting system]: All therapists must provide the blended treatment (yes/no).

Profile of the therapist: 

What type of therapist do you have to be to be able or willing to treat blended?



What type of therapist should you not be to be able or willing to treat blended?



[Statement mobile voting system]: If you may decide, would you continue to use the blended treatment (yes/no)?



What could possibly be done differently?

1.4. Facilitators and barriers [Question mobile voting system]: Which implementation barrier is your number 1 (open answer)? 

What are the most important impeding factors for you as a therapist in implementing blended treatment in your daily work?

- And what are the most important solutions for this?



[Question mobile voting system]: Which implementation facilitator is your number 1 (open answer)?



What are the most important facilitating factors for you as a therapist when implementing blended treatment in your daily work?

1.5. Satisfaction and usability 

[Question mobile voting system]: Patients are about the blended treatment ...(very dissatisfied/quite dissatisfied/quite satisfied/very satisfied).



[Question mobile voting system]: Therapists are about the blended treatment ...(very dissatisfied/quite dissatisfied/quite satisfied/very satisfied).



[Question mobile voting system]: Patients find the platform user-friendly (disagree entirely/disagree/neutral/I agree/strongly agree).



[Question mobile voting system]: Therapists find the platform user-friendly (disagree entirely/disagree/neutral/I agree/strongly agree).

Qualitative Study: Therapists’ Perspectives| 101

[Presentation by moderator: introduction of the results of the CSQ and SUS of both patients and practitioners. This concerns the meaning and value of satisfaction (CSQ) and user-friendliness/usability (SUS)] 

What do you think of the results regarding patient satisfaction?



What do you think of the results regarding the satisfaction of the therapists?



What do you think of the results regarding user-friendliness for patients?



What do you think of the results regarding user-friendliness for the therapists?

102 | CHAPTER 4

Appendix 2: Topic list semi-structured interviews experienced therapists 1. Introduction This focus group is being held in the context of the MasterMind research. MasterMind stands for: MAnagement of mental health diSorders Through advancEd technology and seRvices-telehealth for the MIND. MasterMind is a large European study. It is an implementation study in which cohorts of patients are followed in 11 countries who receive internet treatment for depression. Expectations are high and the results are promising, but internet treatments are still used relatively little in practice. We therefore want to investigate how this treatment can best be applied, which factors are facilitating and which ones actually get in the way of implementation.

In the Netherlands, we are investigating how blended CBT treatment is applied in basic and specialist mental healthcare for patients with depression older than 18 years. Data is collected from 300 patients, practitioners and the representatives of the mental healthcare organizations, including your organization.

With this interview we want to know wat your perspective, as a therapist, is on the implementation of internet-based interventions in the treatment of depression.

1.1. Grand tour question: What is your opinion about internet-based interventions in the treatment of depression?

1.2. Barriers Now, I would like to ask you about the barriers for using bCBT: 

What are the most important barriers for providing blended CBT for depression in routine practice?



Why do you think that bCBT for depression isn’t used often in routine practice?

- Are there more aspects that prevent you (or other) from using bCBT? - Do you have negative experiences in using bCBT for depression?

1.3. Facilitators Now, I would like to ask you about the facilitators for using bCBT: 

What are the most important facilitators providing blended CBT for depression in routine practice?



What would help you (or others) as a therapist to treat more patients with a depression with bCBT? - Are there more aspects that motivate you (or others) to use bCBT? - Do you have positive experiences in using bCBT for depression?

1.4. Barriers and facilitators: therapist factors Now, I would like to elaborate on the barriers and facilitators for using bCBT for depression in routine practice from the perspective of you as a therapist: 

How well do you know the different treatment elements of the online platform (e.g. platform, mobile application, treatment protocol)?



What do you know about the effectiveness of bCBT?



What kind of training is necessary for therapists to provide bCBT?



What skills does a therapist need to treat patients with a depression with bCBT?

Qualitative Study: Therapists’ Perspectives| 103 -

What do you think of your own skills?



To what extent do you consider the use and integration of online elements in the treatment of depression as a part of your role as a therapist?



Considering your experience, what would you add or remove from the bCBT?



What is your opinion about the advantages and possibilities of bCBT? - What are the advantages/possibilities of bCBT vs regular CBT for you as a therapist?

- What are the advantages/possibilities of bCBT vs regular CBT for your patients?



What is your opinion about the disadvantages and risks of bCBT ?

- What are the disadvantages/risks of bCBT vs regular CBT for you as a therapist?

- What are the disadvantages/risks of bCBT vs regular CBT for your patients?



What aspects of bCBT do you like most?



Are you planning on providing bCBT for patients with depression in the future?



What would help you as a therapist to provide bCBT more frequently in the future? Is this also applicable to your colleagues?



To what extent can you imagine that the possibilities of bCBT will be expanded?



Have your colleagues, patients or their family members your validation of bCBT reinforced? Or are you rather discouraged by them?



To what extent do you feel that patients are willing to use bCBT?



What should happen to make bCBT for depression a standard way of treatment?

[Added questions] 

What do you think is the desired proportion of online and face-to-face sessions? (e.g. 50-50, 8020)



What do you think is the desired way to ‘blend’ the online sessions in the treatment, from the beginning or at a later time point?



What is your experience with writing online feedback?

1.5. Barriers and facilitators: patient factors Now, on a patient level: 

For what kind of patients is bCBT eligible? And for who is it unsuitable?



What are reasons to not use bCBT for a patient?



How do you offer bCBT to patients?



What are the reactions of patients when they are offered bCBT?



What reactions of patients keep you from not offering bCBT?



Are there more or other barriers in the interaction with the patient or in the therapeutic relationship with bCBT?



How is the therapeutic relationship developed in bCBT?

1.6. Barriers and facilitators: organizational factors Now, on an organizational level: 

What existing resources, procedures and work routines influence the use of bCBT in your organization? And how?

104 | CHAPTER 4 -

Which are facilitating and which form a barrier for implementation?



What resources, procedures and work routines would help to offer bCBT in a standardized way in routine practice?



How can bCBT simplify the therapeutic and administrative work?

1.7. Barriers and facilitators: health insurance factors Now, on a health insurance level: 

Are the (social-economical) factors from outside of the organization that hinders the implementation of bCBT?



Are the (social-economical) factors from outside of the organization that facilitates the implementation of bCBT?



What is the role of the health insurance companies in the implementation of bCBT?



Which unmet need in the treatment of depression can be fulfilled with bCBT?

1.8. Closing question: 

Are there any issues that I didn’t ask, but are important to consider in this study?

Qualitative Study: Therapists’ Perspectives| 105

Appendix 3: Topic list semi-structured interviews non-experienced therapists 1. Introduction This focus group is being held in the context of the MasterMind research. MasterMind stands for: MAnagement of mental health diSorders Through advancEd technology and seRvices-telehealth for the MIND. MasterMind is a large European study. It is an implementation study in which cohorts of patients are followed in 11 countries who receive internet treatment for depression. Expectations are high and the results are promising, but internet treatments are still used relatively little in practice. We therefore want to investigate how this treatment can best be applied, which factors are facilitating and which ones actually get in the way of implementation.

In the Netherlands, we are investigating how blended CBT treatment is applied in basic and specialist mental healthcare for patients with depression older than 18 years. Data is collected from 300 patients, practitioners and the representatives of the mental healthcare organizations, including your organization.

With this interview we want to know wat your perspective, as a therapist, is on the implementation of internet-based interventions in the treatment of depression.

1.1. Grand tour question: 

What is your opinion about internet-based interventions in the treatment of depression?

1.2. Barriers Now, I would like to ask you about the barriers for using bCBT: 

What are the most important barriers for providing blended CBT for depression in routine practice?



Why do you think that bCBT for depression isn’t used often in routine practice?

- Are there more aspects that prevent you (or other) from using bCBT?

1.3. Facilitators Now, I would like to ask you about the facilitators for using bCBT: 

What are the most important facilitators providing blended CBT for depression in routine practice?



What would help you (or others) as a therapist to treat more patients with a depression with bCBT? - Are there more aspects that motivate you (or others) to use bCBT?

1.4. Barriers and facilitators: therapist factors Now, I would like to elaborate on the barriers and facilitators for using bCBT for depression in routine practice from the perspective of you as a therapist: 

How well do you know the different treatment elements of the online platform (e.g. platform, mobile application, treatment protocol)?



What do you know about the effectiveness of bCBT?



What kind of training is necessary for therapists to provide bCBT?



What skills does a therapist need to treat patients with a depression with bCBT? - What do you think of your own skills?



What is your opinion about the advantages and possibilities of bCBT?

106 | CHAPTER 4 -

What are the advantages/possibilities of bCBT vs regular CBT for you as a therapist?

What are the advantages/possibilities of bCBT vs regular CBT for your patients?



What is your opinion about the disadvantages and risks of bCBT? - What are the disadvantages/risks of bCBT vs regular CBT for you as a therapist?

- What are the disadvantages/risks of bCBT vs regular CBT for your patients?



Are you planning on providing bCBT for patients with depression in the future?



What would help you as a therapist to provide bCBT more frequently in the future? Is this also applicable to your colleagues?



To what extent can you imagine that the possibilities of bCBT will be expanded?



Have your colleagues, patients or their family members your validation of bCBT reinforced? Or are you rather discouraged by them?



To what extent do you feel that patients are willing to use bCBT?



What should happen to make bCBT for depression a standard way of treatment?

[Added questions] 

What do you think is the desired proportion of online and face-to-face sessions? (e.g. 50-50, 8020)



What do you think is the desired way to ‘blend’ the online sessions in the treatment, from the beginning or at a later time point?



What is your experience with writing online feedback?

1.5. Barriers and facilitators: patient factors Now, on a patient level: 

For what kind of patients is bCBT eligible? And for who is it unsuitable?



What are reasons to not use bCBT for a patient?



How would you offer bCBT to patients?



Are there more or other barriers in the interaction with the patient or in the therapeutic relationship with bCBT?



To what extent do you think that the therapeutic relationship is developed in bCBT?

1.6. Barriers and facilitators: organizational factors Now, on an organizational level: 

What existing resources, procedures and work routines influence the use of bCBT in your organization? And how?

- Which are facilitating and which form a barrier for implementation?



What resources, procedures and work routines would help to offer bCBT in a standardized way in routine practice?



How would bCBT simplify the therapeutic and administrative work?

1.7. Barriers and facilitators: health insurance factors Now, on a health insurance level:

Qualitative Study: Therapists’ Perspectives| 107 

Are the (social-economical) factors from outside of the organization that hinders the implementation of bCBT?



Are the (social-economical) factors from outside of the organization that facilitates the implementation of bCBT?



What is the role of the health insurance companies in the implementation of bCBT?



Which unmet need in the treatment of depression can be fulfilled with bCBT?

1.8. Closing question: Are there any issues that I didn’t ask, but are important to consider in this study?

108 | CHAPTER 5

CHAPTER 5

It surprises me that therapists score the platform usability higher than patients...

They don’t have to set-up the dairies Therapist Ella

Confirmatory Factor Analysis: SUS| 109

Dimensionality of the System Usability Scale among Professionals using Internet-based Interventions for Depression: A Confirmatory Factor Analysis Mayke Mol, Anneke van Schaik, Els Dozeman, Jeroen Ruwaard, Christiaan Vis, David Ebert, Anne Etzelmueller, Kim Mathiasen, Bárbara Moles, Teresa Mora, Claus Pedersen, Mette Maria Skjøth, Luisa Peleteiro Pensado, Jordi PieraJimenez, Didem Gokcay, Burçin Ünlü Ince, Alessio Russi, Ylenia Sacco, Enrico Zanalda, Ane Fullaondo Zabala, Heleen Riper & Jan Smit BMC Psychiatry 2020

5

110 | CHAPTER 5

Abstract Background: The System Usability Scale (SUS) is used to measure usability of internet-based Cognitive Behavioral Therapy (iCBT). However, whether the SUS is a valid instrument to measure usability in this context is unclear.

Objective: The aim of this study is to assess the factor structure of the SUS, measuring usability of iCBT for depression in a sample of professionals. In addition, the psychometric properties (reliability, convergent validity) of the SUS were tested.

Methods: A sample of 242 professionals using iCBT for depression from 6 European countries completed the SUS. Confirmatory Factor Analysis (CFA) was conducted to test whether a one-factor, two-factor, tone-model or bi-direct model would fit the data best. Reliability was assessed using complementary statistical indices (e.g. omega). To assess convergent validity, the SUS total score was correlated with an adapted Client Satisfaction Questionnaire (CSQ-3).

Results: CFA supported the one-factor, two-factor and tone-model, but the bi-factor model fitted the data best (Comparative Fit Index = 0.992, Tucker Lewis Index = 0.985, Root Mean Square Error of Approximation = 0.055, Standardized Root Mean Square Residual = 0.042 (respectively χ2diff (9) = 69.82, p < 0.001; χ2diff (8) = 33.04, p < 0.001). Reliability of the SUS was good (ω = 0.91). The total SUS score correlated moderately with the CSQ-3 (CSQ1 rs =.49, p < 0.001; CSQ2 rs =.46, p < 0.001; CSQ3 rs =.38, p < 0.001), indicating convergent validity.

Conclusions: Although the SUS seems to have a multidimensional structure, the best model showed that the total sum-score of the SUS appears to be a valid and interpretable measure to assess the usability of internet-based interventions when used by professionals in mental healthcare.

Keywords: Internet interventions; depression; system usability scale; psychometric evaluation; Confirmatory Factor Analysis

Confirmatory Factor Analysis: SUS| 111

Introduction Implementation of iCBT Mental healthcare in Western Europe is gradually being digitalized. Apart from administrative systems such as electronic patient records, professionals are being introduced to other digital services. These services can support or replace the delivery of regular treatment for mental disorders such as depression. Currently, one of the most studied treatments is internet-based cognitive behavioral therapy (iCBT) [1]. iCBT in a guided or blended format, has proven to be effective in the treatment of depression [2–4]. However, the translation of research findings and implementation to the complex field of routine mental healthcare practice is slow and challenging. As research showed that one of the barriers for implementation is the low usability of internet-based interventions, it is important to assess the usability of iCBT [5, 6]. In feasibility and evaluation studies on iCBT in mental healthcare, the System Usability Scale (SUS) has increasingly been applied to measure usability [7–9], however this instrument has not yet been validated in this emerging field.

System usability Although the meaning of usability is under debate [e.g. 10], usability can be seen as the perceived ‘ease of use’, ‘user-friendliness’ or ‘quality of use’ of a system, interface or product. In the international standard definition, usability is described as the extent to which a product can be used by specified users to achieve designated goals with effectiveness, efficiency, and satisfaction in a specified-context of use [11]. Satisfaction is related to system usability in the sense that satisfaction can contribute to the level of usability or where satisfaction is a consequence of usability [12]. The SUS is a popular instrument to measure the perceived usability of a wide range of products and systems. These include websites, apps, everyday products, software and hardware. Although the SUS has been presented as a ‘quick and dirty’ instrument [13], it is probably not that ‘dirty’ at all [14]. The SUS provides a single score for usability and is designed as a unidimensional (one factor) measurement [13]. In addition, it is without costs, technology agnostic, brief, reliable and valid [15]. Users are presented with ten statements that relate to various aspects of usability (i.e. need for support, complexity) on a 5-point Likert scale, ranging from strong disagreement (1) to strong agreement (5). The final score for the SUS ranges from 0 to 100, with higher scores indicating higher perceived usability.

Psychometric properties and factor structure of the SUS The psychometric properties of the SUS are sufficiently studied with reported reliabilities between 0.79 and 0.97 [e.g. 16, 17], acceptable levels of convergent validity with other measures of perceived usability [e.g. 18] and sensitivity [e.g. 14]. Normative data is available based on scores from 11.855 individual SUS assessments from 166 (unpublished) industrial usability studies [19].

The original English SUS is formally translated into different languages: Arabic, Slovene, Polish, Italian, Persian, and Portuguese [15]. In addition, informal Dutch, French and Spanish translations are available [20]. Several studies added interpretation to the SUS scores: Bangor, Kortum and Miller [21] added an 7-point rating scale to the SUS to provide a SUS score with grades ranging from A to F. A score of 70 is for example given a ‘C’ which is considered ‘good’. Sauro and Lewis [22] published a curved grading scale with a score of 68 as the center of the scale, that can be interpreted as a cut-off for above and below average usability scores.

112 | CHAPTER 5 As for the factor structure, Lewis and Sauro [14] proposed that there might be two factors in the SUS: Usability and Learnability. Since then, studies replicated inconsistent findings pointing towards this two-factor model [e.g. 23] and the one-factor model as well [24, 25]. More recent research showed that two-factor structure possibly depends on the amount of experience that users have with a given product [18]. The SUS acted as a one-factor scale with less product experience, but showed a twofactor structure when more time was spent with the product, in this case an e-learning platform. In 2017, Lewis and Sauro revisited the factor structure of the SUS and tried to replicate the two-factor structure [26]. However, they found a different two-factor structure produced by the positive and negative tone of the items. As the tone structure is of little practical and theoretical interest, their conclusion was to treat SUS as a single factor structure. They suggested that the Usability/Learnability structure can appear in certain circumstances, but that such findings require replication.

Aim The SUS has proven itself to be a useful instrument in an industrial context. However, it remains unclear whether it is a valid instrument in measuring the usability of guided and blended CBT applications, as perceived by professionals in the context of implementation within routine mental healthcare.

Therefore, the aim of this study is to assess the different factor structures of the SUS, measuring usability of iCBT. Four models will be tested: 1) a single factor model to test whether the items in the questionnaire can be summarized by one single factor score, 2) a two-factor model to test whether Usability and Learnability are two different factors, 3) a tone model to test the effect of positive and negative items and 4) a bi-factor model to confirm whether the single factor is measured by all items as well as the factors Usability and Learnability by the indicated subsets of items. In addition, the psychometric properties (reliability, convergent validity) of the SUS will be tested.

Methods Recruitment The professionals were recruited in the context of the large scale European implementation project MasterMind that aimed to provide a summative evaluation of the factors related to uptake of unguided, guided and blended iCBT in 14 regions in 10 countries [27, 28]. The project explored the role of different stakeholders that were involved in the implementation of the intervention (e.g. patients, professionals and representatives of mental healthcare organizations). For the purpose of this study, the data provided by the professionals was used (N=242). They provided guided and blended iCBT to patients with depressive symptoms in the Netherlands (n=51), Germany (n=16), Denmark (n=4), Spain (n=135), Italy (n=33) and Turkey (n=3). Data from the other MasterMind countries (Scotland, Wales, Estonia and Norway) were not suitable for the purpose of this study, as these countries evaluated unguided iCBT and the professionalsâ&#x20AC;&#x2122; interaction with the interface of iCBT was very minimal.

SUS questionnaire Via online and paper-based surveys, professionals were asked to rate the usability of iCBT interventions using ten items of the SUS on a 5-point self-report scale, ranging from 1 (I strongly disagree) to 5 (I strongly agree) after 18 months of data collection within the MasterMind study. The professionals had different levels of iCBT experience when the SUS was administered. Five statements were positively formulated (items with odd numbers) and five statements negatively (items with even numbers). See

Confirmatory Factor Analysis: SUS| 113 Box 1 for the description and response categories of the SUS items, adapted to the MasterMind study and the use of iCBT. In the countries where no translation to the local language was available, the forward and backward method was followed to translate the SUS items (i.e. first the questionnaire was translated from English into the local language by two persons who reached consensus by discussion.

Then the questionnaire was translated back to English and was compared with the original questionnaire).

To calculate the overall SUS score, the following formula was applied [13]: The item score on the positive statements was subtracted by 1 (x-1) and the item score on the negative statements was calculated by subtracting the score from 5 (5-x). The sum of these item scores was then multiplied by 2.5 to provide an overall SUS score between 0 (extremely poor usability) and 100 (excellent usability).

Box 1. Description of the SUS items and response categories in the MasterMind study.

I strongly disagree

I disagree

I don’t disagree nor agree

I agree

I strongly agree

1.

I think that I would like to provide the iCBT intervention to my clients more frequently.











2.

I found the iCBT intervention unnecessarily complex.











3.

I find the iCBT intervention easy to use in treating my clients.











4.

I think that I would need the support of a technical person to be able to use and provide the iCBT intervention to my clients.











5.

I found the various functions in the iCBT intervention were well integrated.











6.

I thought there was too much inconsistency in the iCBT intervention.











7.

I can imagine that most healthcare professionals would learn to use and provide the iCBT intervention very quickly.











8.

I found the iCBT intervention very cumbersome to use.











9.

I felt very confident using and providing the iCBT intervention to my clients.











10.

I needed to learn a lot of things before I could get going with using and providing the iCBT intervention to my clients.











SUS items

CSQ-3 questionnaire To assess convergent validity, three questions that were adapted from the Client Satisfaction Questionnaire (CSQ-3) [29] were used. See Box 2 for a description of the items and response categories.

Same as with the SUS items, in the countries where no translation to the local language was available, the back-translation method was followed to translate the CSQ items [30]. The three items of the CSQ are the main items for measuring overall satisfaction of health and human services and is frequently used as one measure among a battery of other instruments. The CSQ shows good reliability and validity

114 | CHAPTER 5 and is used across a range of services, from inpatient to forensic services, without a specific setting of care [31] and internet-based interventions [32]. Each item on the CSQ-3 is scored from 1 (low satisfaction) to 4 (high satisfaction).

Box 2. Description of the CSQ-3 items and response categories in the MasterMind study.

CSQ-3 items 1.

To what extend has the iCBT intervention met your needs in treating depressed patients?

    None of my needs have Only a few of my needs Most of my needs have Almost all of my needs been met have been met been met have been met 2. In an overall general sense, how satisfied are you with the iCBT treatment you have provided?     Quite dissatisfied Indifferently or mildly Mostly satisfied Very satisfied dissatisfied 3. If you were to provide treatment again, would you use the iCBT intervention again?     No, definitely not No, I don´t think so Yes, I think so Yes, definitely

Statistical Analyses Data of six European countries were pooled for analytic purposes. Statistical analysis was carried out using RStudio (v1.2.1335; RStudio Team, 2015) using the packages lavaan [33], psych [34] and subscore [35] To assess the factor structure, a Confirmatory Factor Analysis (CFA) was conducted. Four models were evaluated: the one-factor model, the two-factor (Usability/Learnability) model, the tone model (positive/negative) and a bi-factor model. Due to the application of the five-point Likert scale, the responses to the SUS-items were considered ordinal data. Hence, the Weighted Least Squares Mean and Variance adjusted (WLSMV) estimator was used as a method of parameter estimation as this is recommended for the analysis of ordinal data [36]. Overall model fit was assessed using a set of goodness-of-fit indices and criterion values, as suggested by Brown [36] as these indices provide an overall satisfactory performance in evaluating models: Chi-square (χ2), Comparative Fit Index (CFI, close to 0.95 or greater), Tucker Lewis Index (TLI, close to 0.95 or greater), Root Mean Square Error of Approximation (RMSEA, close to 0.06 or below) and Standardized Root Mean Square Residual (SRMR, close to 0.08 or below). These fit indices were considered in combination, as a good fit meets all the chosen criteria [37]. A scaled chi-square difference test was applied to compare the fit of the two models [36].

To further investigate the SUS structure and to assess the reliability of the SUS, more advanced statistics were calculated in order to evaluate the found factor structures in the context of finding the best solutions. The omega coefficient was calculated together with other indices: the percentage of uncontaminated correlations (PUC), the explained common variance (ECV), and omega hierarchical [38]. Omega is a reliability estimate that does not depend on the assumption of tau equivalence unlike its classic counterpart Cronbach’s alpha [39]. There is no cut-off point for omega to evaluate acceptable reliability, a minimum of .50 and values closer to .75 are recommended for satisfactory and good reliability [40]. PUC is the percentage of covariance terms which only reflect variance from the general

Confirmatory Factor Analysis: SUS| 115 dimension. ECV is the proportion of all common variance explained by the general factor. Along with ECV, PUC influences the parameter bias of the unidimensional solution. When PUC is greater than .70 and ECV greater than .70 relative bias will be slight and the common variance can be regarded as essentially unidimensional [39]. When a PUC value is lower than .80, the general ECV value greater than .60 and omega hierarchical (of the general factor) is greater than .70 it is suggested that the presence of multidimensionality is not severe enough to disqualify the interpretation of the instrument as primarily unidimensional [40]. To indicate more precisely whether the subscales has added value over and above the total SUS score Haberman introduces a methodology to qualify this added value. This is done by computing the proportional reduction in mean squared error (PRMSE) based on the total score (PRMSEtotal) and comparing that value to the proportional reduction in mean squared error based on subscale scores (PRMSEsubscale) [41]. If the ratio of these values, subscore over total score, exceed one, the subscore does not have added value and it is not recommended to use the subscore in statistical models.

To assess convergent validity, Spearman's rank-order correlations were calculated between all three items of the CSQ-3 and the total SUS score. A common criterion for the absolute magnitude of correlations that supports the hypothesis of convergent validity is a minimum of 0.30 [42].

Results Sample The respondents (n=242) completed the questionnaire between August and December 2016. Table 1 provides an overview of the respondent characteristics. Most respondents were female (71.5%), 39.3% of the respondents were GPâ&#x20AC;&#x2122;s, 34.7% were psychologists or psychiatrists and 24.8% had a different professional background (e.g. specialized health nurses, health workers). About half of the respondents had more than five years of professional experience in mental healthcare (54.5%). The majority of respondents (57%) had provided iCBT at least five times to treat depressed patients. Still, a relatively large group (36.4%) had little experience with iCBT (i.e. providing iCBT less than 5 times to patients).

116 | CHAPTER 5 Table 1. Sample characteristics of respondents.

Variable

Label

Pooled, n (%)

N

Cases

242 (100)

Gender

Female Missing GP Psychologist Psychiatrist Other Missing 0 - 2 years 3 - 4 years 5 - 9 years > 10 years Missing 1-4 times 5-9 times 10-14 times 15-19 times > 20 times Missing

173 (71.5) 2 (0.8) 95 (39.3) 76 (31.4) 8 (3.3) 60 (24.8) 3 (1.2) 69 (28.5) 36 (14.9) 39 (16.1) 93 (38.4) 5 (2.1) 88 (36.4) 48 (19.8) 23 (9.5) 24 (9.9) 43 (17.8) 16 (6.6)

Profession

Field experience

iCBT experience

SUS and CSQ scores The pooled mean total score of the SUS was 67.9 (SD 16.3; range 20-100), indicating a (just) below average score. See Table 2 and 3 for mean scores of the SUS and the CSQ-3. Fig. 1 gives a visual overview of the distribution of the item responses on the SUS. See Appendix 1 for a percentile rank of SUS items scores, a covariance matrix of SUS item scores and the distribution of the frequencies of the total SUS scores.

Table 2. Mean, standard deviation and range of the (recoded) SUS scores.

Pooled, mean (SD; range) N

242

SUS1 SUS3 SUS5 SUS7 SUS9

3.56 (0.95;1-5) 3.75 (0.96;1-5) 3.63 (0.89;1-5) 3.66 (0.94;1-5) 3.74 (0.91;1-5)

SUS total

67.85 (16.28;20-100)

SUS2 SUS4 SUS6 SUS8 SUS10

3.71 (1.04;1-5) 3.53 (1.18;1-5) 3.77 (0.95;1-5) 3.84 (0.93;1-5) 3.70 (1.08;1-5)

Confirmatory Factor Analysis: SUS| 117 Table 3. Mean, standard deviation and range of the CSQ-3 scores.

Pooled, mean (SD; range) N

241

CSQ1 CSQ2 CSQ3

2.91 (0.78;1-4) 3.10 (0.66;1-4) 3.28 (0.68;1-4)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

SUS1 SUS2 SUS3 SUS4 SUS5 SUS6 SUS7 SUS8 SUS9 SUS10

I strongly disagree

I disagree

I don't disagree nor agree

I agree

I strongly agree

Figure 1. Percent distribution of item responses on the (inversed) SUS items.

Confirmatory Factor Analysis Table 4 shows the results for the four models. The overall goodness-of-fit indices suggested that the one-factor, the two-factor and tone models have an acceptable fit (i.e. CFI ≥ 0.95, TLI ≥ 0.95, SRMR ≤ 0.08). However, further inspection revealed a better fit for the indices of the bi-factor. The scaled chisquare test confirmed that the difference between the bi-factor model and the other models was statistically significant (one-factor χ2diff = 69.82, df = 9, p < 0.001; two-factor χ2diff = 33.04, df = 8, p < 0.001; tone model χ2diff = 59.58, df = 8, p < 0.001 ). Figure 2 shows a visualization of the bi-factor model, with a positive correlation (rs = 0.70) between the factors Usability and Learnability. See Appendix 1 for an overview of the other models in diagrams and factor loadings.

118 | CHAPTER 5 Table 4. Results of the confirmatory factor analysis.

Model

Npar

Chi square

DF

CFI

TLI

RMSEA (CI)

SRMR

One-factor Two-factor Tone-model Bi-factor

50 51 51 60

124.84 82.19 109.88 44.96

35 34 34 26

0.960 0.979 0.966 0.992

0.949 0.972 0.955 0.985

0.103 (0.084-0.123) 0.077 (0.056-0.098) 0.096 (0.076-0.117) 0.055 (0.026-0.081)

0.079 0.066 0.075 0.042

Npar: number of parameters estimated in the CFA; DF: degrees of freedom; CFI: Comparative Fit Index; TLI: Tucker Lewis Index; RMSEA: Root Mean Square Error of Approximation; CI: 90% confidence interval; SRMR: Standardized Root Mean Square Residual.

Figure 2. Factor structure of the bi-factor model of the SUS.

Confirmatory Factor Analysis: SUS| 119 Reliability The reliability analysis showed that the 10 items on the SUS had acceptable reliability, ω = 0.91. The PUC was 0.36, the ECV was 0.75 and omega hierarchical was 0.78, meaning that the SUS is not entirely convincingly unidimensional and at the same time the presence of the two subscales (Usability/Learnability) not serious enough is to disqualify undimensionality. This was made clear by the PRMSE results of the total (PRMSEtotal = 0.93 and subscale scores (PRMSEusability = 0.70, PRMSE learnability = 0.55): both PRMSE ratio values exceed 1 confirming that the subscales do not have added value over the total score.

Convergent validity The total score of the SUS correlated moderately with the three items on the CSQ-3, indicating convergent validity between the two measures (See Table 5).

Table 5. Convergent validity of SUS and CSQ-3.

Item

Spearman correlation with SUS total score (CI)

CSQ1. Have the needs been met?

CSQ2. Overall satisfaction?

CSQ3. Provide treatment again?

0.49 (0.39 to 0.58)*** 0.46 (0.35 to 0.55)*** 0.38 (0.26 to 0.48)***

CI: 95% confidence interval; *** Correlation is significant at 0.001.

Discussion Usability of internet-based interventions are an important factor in successful implementation and patient engagement [6]. Findings of our study demonstrate that the System Usability Scale (SUS) is a valid measure to assess the usability of iCBT in mental healthcare.

The CFA provided support for the bi-factor model; this model fitted the data better than the onefactor model, the two-factor model or the tone model. Although this would mean that the SUS gives a score for overall usability, as well as scores for the subscales Usability and Learnability, further analysis showed that the subscales contain no information that is not already contained in the total score.

There may be several reasons why previous studies found mixed findings of the subscales Usability and Learnability. An explanation by Borsci [18] is that it depends on the level of ‘product’ experience or exposure. In our sample, the amount of iCBT experience among the professionals varied considerably. However, a large variety of product experience was also reported in studies that found a one-factor model [e.g. 24, 25]. Another explanation may be related to the complexity of the product; it can be assumed that the Learnability factor has more weight in a context that requires more learning (e.g. an e-learning or intervention platform) than a more straightforward context (e.g. microwave or mobile app). In the case of iCBT, professionals have to adapt to a new system and learn how to integrate this into their work routine. However, as Lewis also reported [26], the contexts in which Usability and Learnability (dis)appear need further investigation. Furthermore, the SUS research field could consider assessing complementary statistical indices with applying bi-factor models and its tendency to ‘overfit’, to make more informed decisions [43]. The correlations between the SUS and CSQ-scores indicated convergent validity, comparable to other studies that also found considerable evidence for the overlap

120 | CHAPTER 5 with other related questionnaires such as the Usability Metric for User Experience (UMUX) [18, 44].

Moreover, the SUS in this study had a good reliability. This is in line with previous research in other contexts as well [e.g. 26].

There are several limitations in this study that need to be discussed. First of all, the sample had an uneven distribution of professionals per country resulting in an under and over representation of the countries included within the study. This also limited the analysis of the factor structure in taking into account the different countries and the possible biased standard error. Plus, the technical formats of the iCBT applications as well as the content of the iCBT interventions differed between countries.

Hence, the representativeness of the sample might be affected by this. Secondly, as most translations of the original SUS items were informal, possible different interpretations by the professionals may have occurred. On the other hand, this risk was minimized by using a back-translation method [30]. It is possible that the factor structure of the SUS is distorted by the mixed tone of negatively and positively worded items. The mixed tone was originally used to control for acquiescence bias; the hypothesized tendency of respondents to agree with statements with a mix of positive and negative tone [19].

However, several studies encountered an unintentional SUS factor structure caused by the mixed tone [e.g. 24]. Moreover, there is evidence that the mixed tone caused respondents to make mistakes and researchers to miscode the questionnaire [45]. In 2011, Sauro and Lewis tested a positive version of the SUS and found no significant difference between the mean overall SUS scores of the positive and mixed versions [45]. In addition, they found no evidence for strong acquiescence bias or extreme response bias. To avoid problems caused by tone, future researchers could consider alternative formats of the SUS (e.g. a positive version, item specific response options, expanded scale format) [46, 47].

Conclusions This study demonstrated that the SUS had good psychometric properties, even in a heterogeneous sample of professionals in mental healthcare. Different factor structures were studied with reasonable outcomes. However the bi-factor model showed the best results in this sample indicating that researchers interested in the usability of internet-based interventions in mental healthcare can use the proposed scoring of the SUS and in particular the calculated sumscore.

Abbreviations CFA: Confirmatory Factor Analysis; CFI: Comparative Fit Index; CI: Confidence interval; CSQ: Client Satisfaction Scale; DF: Degrees of freedom; iCBT: internet-based Cognitive Behavioral Therapy; M: Mean, average; RMSEA: Root Mean Square Error of Approximation; SD: Standard Deviation; SRMR: Standardized Root Mean square Residual; SUS: System Usability Scale; TLI: Tucker Lewis Index; WLSMV: Weighted Least Square Means and Variances.

Ethical approval and consent to participate Ethical and scientific approval was granted by the Scientific and Ethical Review Board of the Faculty of Behavioral and Movement Sciences at the VU Amsterdam (file number: VCWE-2016-006) and by local medical ethical committees and relating regulatory agencies of the participating countries. All participating professionals received an written informed consent.

Confirmatory Factor Analysis: SUS| 121 Funding The MasterMind project was partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme (CIP) by the European Community (Grant Agreement number: 621000). The funding body had no influence on the design, execution, analysis, or interpretation of the results of this study.

Acknowledgements The authors would like to thank all professionals that participated in this study.

122 | CHAPTER 5

References [1] [2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10] [11] [12] [13] [14] [15] [16]

Andersson, G., Titov, N., Dear, B.F., Rozental, A., & Carlbring, P. (2019).Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry; 18, 20–8.

Carlbring, P., Andersson, G., Cuijpers, P., Riper, H., & Hedman-Lagerlöf, E. (2018). internetbased vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn Behav Ther; 47, 1–18.

Thase, M.E., Wright, J.H., Eells, T.D., Barrett, M.S., Wisniewski, S.R., Balasubramani, G.K., et al.

(2017). Improving the Efficiency of Psychotherapy for Depression: Computer-Assisted Versus Standard CBT. Am J Psychiatry; 175, 242-250.

Karyotaki, E., Ebert, D.D., Donkin, L., Riper, H., Twisk, J., Burger, S., et al. (2018). Do guided internet-based interventions result in clinically relevant changes for patients with depression?

An individual participant data meta-analysis. Clinical Psychology Review; 63, 80-92.

Folker, A.P., Mathiasen, K., Lauridsen, S.M., Stenderup, E., Dozeman, E., & Folker, M.P. (2018).

Implementing internet-delivered cognitive behavior therapy for common mental health disorders: A comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interv; 11, 60–70.

Vis, C., Mol, M., Kleiboer, A., Bührmann, L., Finch, T., & Smit, J., et al. (2018). Improving implementation of emental health for mood disorders in routine practice: Systematic review of barriers and facilitating factors. JMIR Mental Health; 20.

Kooistra, L.C., Ruwaard, J., Wiersma, J.E., van Oppen, P., van der Vaart R., van Gemert-Pijnen, J.E.W.C., et al. (2016). Development and initial evaluation of blended cognitive behavioural treatment for major depression in routine specialized mental health care. Internet Interv; 4, 61–71.

Wozney, L., Baxter, P., & Newton, A.S. (2015). Usability evaluation with mental health professionals and young people to develop an internet-based cognitive-behaviour therapy program for adolescents with anxiety disorders. BMC Pediatr; 15, 213.

Kleiboer, A., Smit, J., Bosmans, J., Ruwaard, J., Andersson, G., Topooco, N., et al. (2016).

European COMPARative Effectiveness research on blended Depression treatment versus treatment-as-usual (E-COMPARED): Study protocol for a randomized controlled, non-inferiority trial in eight European countries. Trials; 17.

Borsci, S., Federici, S., Malizia, A., & De Filippis, M.L. (2019). Shaking the usability tree: why usability is not a dead end, and a constructive way forward. Behav Inf Technol; 38, 519–532.

ISO I. 9241—11 (1998). Ergonomic requirements for work with visual display terminals (VDTs)Part 11: Guidance on usability. Brussels, BE: CEN. Tractinsky, N. (2018). The Usability Construct: A Dead End? Human-Computer Interact; 33, 131177.

Brooke, J.(1996). SUS-A quick and dirty usability scale. Usability Eval Ind; 189, 4-7.

Lewis, J.R. & Sauro, J. (2009). The factor structure of the system usability scale. Human Centered Design; 94–103.

Lewis, J.R. (2018). The System Usability Scale: Past, Present, and Future. Int J Hum Comput Interact; 34, 577–90.

Dianat, I., Ghanbari, Z., & AsghariJafarabadi, M. (2014). Psychometric properties of the persian

Confirmatory Factor Analysis: SUS| 123

[17] [18]

[19] [20] [21] [22] [23] [24] [25]

[26] [27]

[28] [29] [30] [31]

[32] [33] [34] [35] [36]

language version of the system usability scale. Heal Promot Perspect; 4, 82–9.

Finstad, K. (2010). The usability metric for user experience. Interact Comput; 22, 323–7.

Borsci, S., Federici, S., Bacci, S., Gnaldi, M., & Bartolucci, F. (2015). Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience. Int J Hum Comput Interact; 31, 484–95.

Lewis, J.R. & Sauro, J. (2018). Item Benchmarks for the System Usability Scale. J Usability Stud; 13, 158–67.

Sauro, J. (2011). A practical guide to the system usability scale: Background, benchmarks & best practices. Denver, CO: Measuring Usability LLC. Bangor, A., Kortum, P., & Miller, J. (2009). Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J Usability Stud; 4, 114-123.

Sauro, J., Lewis, J.R. (2016). Quantifying user research. In: Quantifying the User Experience: Practical statistics for user research. Morgan Kaufmann; 9–18.

Borsci, S., Federici, S., & Lauriola, M. (2009). On the dimensionality of the System Usability Scale: a test of alternative measurement models. Cognitive processing; 10, 193-197.

Kortum, P. & Sorber, M. (2015). Measuring the Usability of Mobile Applications for Phones and Tablets. Int J Hum Comput Interact; 31, 518–29.

Lewis, J.R., Brown, J., & Mayes, D.K. (2015). Psychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability Study. Int J Hum Comput Interact; 31:545–53.

Lewis, J.R., & Sauro, J. (2017). Revisiting the Factor Structure of the System Usability Scale. J Usability Stud; 12, 183-192.

Vis, C., Kleiboer, A., Prior, R., Bønes, E., Cavallo, M., Clark, S.A., et al. (2015). Implementing and up-scaling evidence-based eMental health in Europe: The study protocol for the MasterMind project. Internet Interv; 2, 399–409.

MasterMind project website. http://mastermind-project.eu/. Accessed 30 Jul 2019.

Attkisson, C.C. & Greenfield, T.K. (1995). The client satisfaction questionnaire (CSQ) scales.

Outcome assessment in clinical practice. Baltimore, MD: Williams & Wilkins.

Brislin, R.W. (1970). Back-Translation for Cross-Cultural Research. J Cross Cult Psychol; 1, 185– 216.

Miglietta, E., Belessiotis-Richards, C., Ruggeri, M., & Priebe, S. (2018). Scales for assessing patient satisfaction with mental health care: A systematic review. Journal of Psychiatric Research; 100, 33–46.

Boβ, L., Lehr, D., Reis, D., Vis, C., Riper, H., Berking, M., et al. (2016). Reliability and validity of assessing user satisfaction with web-based health interventions. J Med Internet Res; 18.

Rosseel, Y. (2012). Lavaan: an R package for structural equation modeling. Journal of Statistical Software; 48, 1-36.

Revelle, W.R. (2017). Psych: Procedures for Personality and Psychological Research.

Northwestern University, Evanston, Illinois.

Dai, S., Wang, X., & Svetina, D. (2019). Subscore: Computing subscores in Classical Test Theory and Item Response Theory. R package. Bloomington, Indiana: Indiana University.

Brown, T. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.

124 | CHAPTER 5 [37] [38]

[39] [40]

[41] [42] [43] [44] [45] [46] [47]

Hu, L. & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equ Model A Multidiscip J; 6, 1–55.

Dueber, D.M. (2017). Bifactor Indices Calculator: A Microsoft Excel-based tool to calculate various indices relevant to bifactor CFA models.

[Available at http://sites.education.uky.edu/apslab/resources/] Rodriguez, A., Reise, S.P., & Haviland, M.G. (2016). Applying Bifactor Statistical Indices in the Evaluation of Psychological Measures. J Pers Assess; 98, 223–37.

Reise, S.P., Scheines, R., Widaman, K.F., & Haviland, M.G. (2013). Multidimensionality and Structural Coefficient Bias in Structural Equation Modeling: A Bifactor Perspective. Educ Psychol Meas; 73, 5–26.

Haberman, S.J. (2008). When can subscores have value? Journal of Educational and Behavioral Statistics; 33, 204–29.

Swank, J.M. & Mullen, P.R. (2017). Evaluating Evidence for Conceptually Related Constructs Using Bivariate Correlations. Meas Eval Couns Dev; 50, 270–4.

Bonifay, W., Lane, S.P., & Reise, S.P. (2017). Three Concerns With Applying a Bifactor Model as a Structure of Psychopathology. Clinical Psychological Science; 5, 184–6.

Berkman, M.I. & Karahoca, D. (2016). Re-assessing the usability metric for user experience (UMUX) scale. J Usability Stud; 11, 89-109.

Sauro, J., & Lewis, J.R. (2011). When designing usability questionnaires, does it hurt to be positive? In: Conference on Human Factors in Computing Systems - Proceeding; 2215–23.

Zhang, X., & Savalei, V. (2016). Improving the Factor Structure of Psychological Scales: The Expanded Format as an Alternative to the Likert Scale Format. Educ Psychol Meas;76, 357–86.

Saris, W.E, Revilla, M., Krosnick, J.A., & Shaeffer, E.M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Surv Res Methods; 4, 45–59.

Confirmatory Factor Analysis: SUS| 125

Appendix 1 Table 1. Percentile rank of SUS items scores.

Percentile rank

SUS total score

5th 10th 25th 50th 75th 90th 99th

37.5 45.0 57.5 70.0 78.8 87.5 95.0

Table 2. Covariance matrix of SUS item scores, n=242.

Item

SUS1

SUS1 SUS2 SUS3 SUS4 SUS5 SUS6 SUS7 SUS8 SUS9 SUS10

0.905 0.483 0.373 0.389 0.393 0.405 0.479 0.289 0.300 0.146

SUS2 1.080 0.577 0.633 0.418 0.583 0.525 0.529 0.253 0.496

SUS3

0.911 0.525 0.341 0.368 0.401 0.499 0.305 0.434

SUS4

1.380 0.367 0.388 0.415 0.464 0.233 0.713

SUS5

0.799 0.414 0.376 0.304 0.157 0.177

SUS6

0.892 0.424 0.354 0.199 0.236

SUS7

0.877 0.307 0.312 0.209

SUS8

0.858 0.349 0.445

SUS9

0.819 0.249

SUS10

1.158

126 | CHAPTER 5

SUS total scores 25

20

Frequency

15

10

5

20 23 28 30 35 38 40 43 45 48 50 53 55 58 60 63 65 68 70 73 75 78 80 83 85 88 90 93 95 100

0

Figure 1. Distribution of frequencies of total SUS scores.

Confirmatory Factor Analysis: SUS| 127

One-factor model SUS1

0.7

0.8

SUS2

SUS3

0.7 SUS4 0.7

0.7

SUS5

SUS Total score 0.7

SUS6

0.7 SUS7 0.7 0.7

SUS8

0.6 SUS9

SUS10

Tone model

SUS1

0.7 SUS3 0.8

SUS5 0.7

SUS7

0.8

Positive items

0.5 SUS9 0.9

SUS2

0.9

SUS4

0.7

0.7

SUS6 0.8 SUS8

SUS10

0.6

Negative items

128 | CHAPTER 5

Two-factor model

SUS1

SUS2

0.7

0.8 SUS3 0.7 SUS5

0.7

Usability 0.7 SUS6 0.7

0.7 SUS7

0.7

Learnability SUS8

0.5

SUS9 0.9

SUS4

0.7

SUS10

Figure 2. Factor structure of the one-factor model, two-factor, and tone-model of the SUS.

Confirmatory Factor Analysis: SUS| 129

130 | CHAPTER 6

CHAPTER 6

Monday, 9:01 Dear Jonah, Very good that you looked at the first session of the module. You took the first step. Did you recognize yourself in the examples? You could not complete all assignments. Very good that you tried. Especially because you feel so down and tired, as you indicated in the questionnaire. Weâ&#x20AC;&#x2122;ll discuss this further in our next face-to-face session this Friday. We can also look together why it was not possible to complete the assignments.

Furthermore, I noticed that you have a clear image of the depression, namely loss of interest and energy, while you suffer from negative thoughts, worrying. You describe that it has built over the past two years, through various events. I see in your diary that your mood is always around 5, that filling this in is pleasant on the one hand and a confrontation with how you are doing on the other. Good that you dare to commit to the treatment.

If you have questions or comments in between, you can always send me a message via the mail function - I will respond to this as fast as I can.

Kind regards, Ella

Mixed-Methods Observational Study: Online Therapeutic Feedback | 131

Behind the Scenes of Online Therapeutic Feedback in Blended Therapy for Depression: Mixed-Methods Observational Study Mayke Mol, Els Dozeman, Simon Provoost, Anneke van Schaik, Heleen Riper & Jan Smit JMIR 2018

6

132 | CHAPTER 6

Abstract

Background: In Internet-delivered cognitive behavioral therapies (iCBT), written feedback by therapists is a substantial part of therapy. However, it is not yet known how this feedback should be given best and which specific therapist behaviors and content are most beneficial for patients. General instructions for written feedback are available, but the uptake and effectiveness of these instructions in iCBT have not been studied yet.

Objectives: This study aimed to identify therapist behaviors in written online communication with patients in blended CBT for adult depression in routine secondary mental health care, to identify the extent to which the therapists adhere to feedback instructions, and to explore whether therapist behaviors and adherence to feedback instructions are associated with patient outcome.

Methods: Adults receiving blended CBT (10 online sessions in combination with 5 face-to-face sessions) for depression in routine mental health care were recruited in the context of the European implementation project MasterMind. A qualitative content analysis was used to identify therapist behaviors in online written feedback messages, and a checklist for the feedback instruction adherence of the therapists was developed. Correlations were explored between the therapist behaviors, therapist instruction adherence, and patient outcomes (number of completed online sessions and symptom change scores).

Results: A total of 45 patients (73%, 33/45 female, mean age 35.9 years) received 219 feedback messages given by 19 therapists (84%, 16/19 female). The most frequently used therapist behaviors were informing, encouraging, and affirming. However, these were not related to patient outcomes.

Although infrequently used, confronting was positively correlated with session completion (ρ=.342, P=.02). Therapists adhered to most of the feedback instructions. Only 2 feedback aspects were correlated with session completion: the more therapists adhere to instructions containing structure (limiting to 2 subjects and sending feedback within 3 working days) and readability (short sentences and short paragraphs), the less online sessions were completed (ρ=−.340, P=.02 and ρ=−.361, P=.02, respectively). No associations were found with depression symptom change scores.

Conclusions: The therapist behaviors found in this study are comparable to previous research. The findings suggest that online feedback instructions for therapists provide sufficient guidance to communicate in a supportive and positive manner with patients. However, the instructions might be improved by adding more therapeutic techniques besides the focus on style and form.

Keywords: cognitive behavioral therapy; eHealth; depressive disorder

Mixed-Methods Observational Study: Online Therapeutic Feedback | 133

Introduction Internet-Delivered Cognitive Behavioral Therapy There is considerable evidence that internet-based interventions are effective for the treatment of mild, moderate, and major depression [1-3]. Therapist-guided, Internet-delivered cognitive behavioral therapy (iCBT) has been found to be more effective than unguided iCBT [1,4] and has also been found to be equally effective compared with face-to-face-delivered CBT [5]. Beyond these findings, a number of studies focused on nonspecific factors that might be effective in iCBT (e.g., therapeutic alliance, therapist competence, and placebo-expectancy effects) and especially showed interest in the role of therapist guidance in iCBT [6]. So far, mixed results have been found. In a systematic review, Richards and Richardson, e.g., found that the way guidance is given has an impact on treatment adherence in depressed patients [4]. Therapist-guided iCBT had a 72% completion rate, iCBT interventions with administrative support (support by staff to guide patients through the program in a nontherapeutic way) 65%, and interventions with no support at all 26%. For Internet-delivered problem-solving treatment (PST), there also is evidence that the level of support is important in reaching effects for patients with depression [7]. Patients who received PST with weekly support from a coach improved significantly more than the waitlist control group. In the group that received no support, completion rates were lowest (22%), and the completion rates were highest in the group that received nonspecific support (60%). Patients who received weekly support had comparable completion rates with patients who received “support on request” (33% and 31%, respectively). In a study by Titov et al, patients with depression showed significant clinical improvement after receiving iCBT, regardless of whether the support came from a therapist or a technician [8]. Findings of a recent American study indicate that iCBT with 5 hours of therapeutic face-to-face contact was noninferior to CBT that provided over 8 additional hours of therapist contact for patients with depression [9].

Therapist Behaviors However, there is much more to discover about online guidance. One point of interest is how therapists give online feedback to their patients. This can be done by looking at the communication strategies and content they use in their written support. For example, looking at therapist behaviors such as validating what patients write (e.g. “That must be very difficult for you...”) and stimulating patients to come up with their own solution (e.g. “When was the last time you felt that way? What did you think and what did you do differently?”). Written feedback is a substantial part of internet-based treatments and requires specific skills of therapists. It is therefore interesting to further explore such therapeutic micro processes in online feedback because this part of therapy may be very relevant in the adherence and also the effectiveness of iCBT [10].

Therapist Behaviors in Face-to-Face Therapy The content of feedback and its impact on treatment results have been studied in face-to-face– delivered psychotherapies, and especially in CBT. Studies have identified different therapist behaviors that are frequently used in CBT sessions with patients. These behaviors range from expressing empathy, making supportive communications (e.g., encourage, praise, or guide the patient), asking directive questions, and confronting patients with different points of view [11-13]. Self-disclosures by therapists appear to be infrequently used [11], although these are generally considered helpful by patients in the

134 | CHAPTER 6 therapeutic process [14,15]. In addition, research shows that therapist behaviors such as expressing empathy, giving positive regards, and confronting patients can have a positive impact on treatment outcome in CBT for various patients such as people with depression [11].

Therapist Behaviors in Internet-Delivered Cognitive Behavioral Therapy Therapist behaviors in iCBT have also been studied. This was done for several psychiatric diseases such as eating disorders [16], insomnia [17], anxiety [18], and depression [19,20]. Comparable to the behaviors in face-to-face therapies, the most frequently used therapist behaviors were encouraging, reinforcing, and supporting patients. When looking at the association between these therapist behaviors, patient treatment outcome, and patient online session completion, mixed results were found. Holländare et al found that encouraging, guiding, and affirming were strongly associated with session completion [19]. Encouraging, affirming, and self-disclosure were weakly to moderately associated with an improvement in depressive symptoms. The most important finding by Paxling et al was the effect of therapistsâ&#x20AC;&#x2122; task reinforcement (e.g. reinforcing completed assignments) on session completion as well as treatment outcome [18]. Interestingly, a negative association was found between deadline flexibility of therapists and treatment outcome. Thus, the more lenient therapists were with homework assignment deadlines, the fewer patients improved. In a replication study of Schneider et al, the same type of therapist behaviors were found with the addition of a few more behavior categories (e.g. asking questions) [20]. However, a different distribution of the therapist behavior frequencies was found, and the outcomes were different for patients with depression than for patients with anxiety.

Thus, the way online feedback is provided by therapists differs across studies, patients, interventions, and possibly also the instructions used for feedback.

Online Feedback Instructions In addition to more general communicative behaviors of therapists, the extent to which they follow instructions for online feedback may also influence treatment effectiveness. Research on written feedback predominantly stems from the field of education. Some of the main principles can be applied to online therapeutic feedback as well. Overall, research shows that effective written feedback is timely (provided in time), selective (commenting only on 2 or 3 things that someone can change), balanced (pointing out positive aspects as well as areas in need of improvement), forward-looking (suggesting how to improve), and understandable (written in a language that someone will understand) [21].

Instructions for training therapists in written feedback are adapted to the therapeutic process but also comparable to those used in education (e.g., beginning with a compliment, responding within 3 working days, or being careful with giving solutions). The elements in these instructions are primarily based on expert opinion rather than theory and mainly aim to motivate and support the patients, respond to the content of homework, and structure the feedback.

Study Objectives In this study, written feedback will be studied in blended CBT, in the context of the European implementation project MasterMind [22-24], with a focus on therapist behaviors and on the extent to which feedback instructions are followed. In the Netherlands, iCBT for depression is slowly but increasingly adopted in routine mental health care mostly in a blended format. Blended CBT entails one integrated, standardized CBT treatment protocol that combines face-to-face sessions and digital

Mixed-Methods Observational Study: Online Therapeutic Feedback | 135 modules to the best clinical benefit for patients and therapists [25]. The evidence of blended CBT over iCBT is unfortunately still scarce. Some first studies indicate that potential benefits of blended CBT are saving therapist time without reducing therapeutic outcome, lower treatment dropout rates, more emphasis on patient self-management, more face-to-face therapy time for deepening the CBT elements, and targeting another (often more complicated) population than iCBT [6,9,26]. The objectives of this study were to (1) identify therapist behaviors in written online communication from therapists to patients in blended CBT for adult depression in routine secondary mental health care, (2) identify the extent to which therapists adhere to feedback instructions, and (3) investigate whether therapist behaviors and therapist adherence to feedback instructions are associated with patient outcome (symptom change scores and number of completed online sessions).

Methods Design For the purpose of this observational study, the feedback messages of 45 Dutch patients that were offered blended CBT for depression by 19 therapists in routine mental health care were recruited between April 2015 and February 2017 from one outpatient clinic. This clinic was one of the participating MasterMind sites and was selected for this study because it offered a blended treatment protocol to patients within secondary health care, and the online usage information was made available for research. Patients received 219 feedback messages through a secure Web-based platform [27]. A mixed-method design was chosen to explore the content of the feedback messages: a directed qualitative content analysis [28] was used to identify therapist behaviors, and a checklist for the feedback instruction adherence of therapists was developed. To explore correlations between the frequency of therapist behaviors, scores on the checklist, and patientsâ&#x20AC;&#x2122; outcomes, an explorative quantitative approach was used.

The study was approved by a Medical Ethics Committee. They confirmed that the â&#x20AC;&#x153;Medical Research Involving Human Subjects Actâ&#x20AC;? does not apply (registration number 2014.580) because the patients in this study are not required to follow certain procedures on behalf of the research (no randomization) and routine practice was followed. An internal scientific research committee approved the research proposal (CWO 2015-005).

Participants Patients Patients were recruited through their therapists. Eligible patients received study information and an information leaflet from their therapist. After approval for telephone contact with researchers for additional information, patients received an informed consent. Patients were invited for participation in MasterMind if they 1) were aged 18 years or older; 2) had a mild, moderate, or severe depression as a primary diagnosis according to the therapist; and 3) were indicated for cognitive behavioral treatment for depression following routine secondary mental health care procedures. All patients needed to explicitly consent to take part in the study. Patients were excluded from the study if they 1) did not have a valid email address and did not have a computer with Internet access and 2) did not have adequate Dutch language skills (both verbal and written).

Therapists

136 | CHAPTER 6 Therapists who were trained in iCBT or who were motived for iCBT were invited to participate in the MasterMind study. They were recruited through team managers and eHealth attention officers of the different therapist teams. The iCBT training consisted of a 4-hour group training, provided by the outpatient clinic. During the training, the functionalities on the online platform were shown, and therapists got the chance to practice with a fictional patient. The therapists received individual instructions, access to the blended CBT treatment protocol online, and the feedback instructions. In addition, monthly 1-hour group sessions were organized where the therapists could exchange their experiences with each other.

The feedback instructions for therapists comprised general and specific elements that go in to the structure of the messages (e.g., correct greeting, limiting to 2 subjects), readability (short sentences and paragraphs), writing style (e.g., limiting abbreviations and misspellings, use of emoticons), referring to parts of the treatment (e.g., filling in the diary, referring to the next online session), and communication skills (e.g. summarizing, not providing solutions).

Intervention In the blended CBT treatment for depression of the outpatient clinic, it was agreed upon in advance that patients would receive 10 sessions online and meet with their therapist in 5 face-to-face sessions biweekly. In practice, therapists could deviate from the protocol by repeating online sessions. The online and individual face-to-face sessions were based on evidence-based treatment protocols for faceto-face CBT and are in agreement with multidisciplinary instructions for depression [29]. There were 4 core components: 1) psychoeducation, 2) cognitive restructuring, 3) behavioral activation, and 4) relapse prevention. Besides the online sessions on the treatment platform, patients were given online access to a diary and filled out questionnaires to monitor their symptoms. After each completed online session, the therapist (the same therapist as in the face-to-face sessions) wrote a feedback message to the patient. Patient and therapist could additionally communicate through a message function about practical issues (e.g., about upcoming appointments and reminders or questions about assignments).

Measures Patient information on selected demographics (e.g., age, gender, employment status) and clinical data (e.g., use of medication) were obtained by an online self-report questionnaire at baseline. Demographic and background information (e.g., treatment and iCBT experience) of the therapists were obtained by an online self-report questionnaire at the end of the study. Usage information (e.g., number of online sessions followed and number of feedback messages) was obtained from the online platform.

Session completion was defined as the number of completed online sessions per patient. Symptom improvement was measured with the 16-item Quick Inventory of Depressive Symptomatology (QIDS) [30]. The total score varies from 0 to 27, with higher scores being indicative of a higher severity of depressive symptoms. The QIDS was administered weekly on the online platform during the course of the treatment. The number of QIDS measures can vary, with up to 30 weekly measures. Of each patient, the baseline scores were included, and the last known value was used as a posttreatment score. The change score on the QIDS was calculated by subtracting the baseline measurement from the final measurement.

Mixed-Methods Observational Study: Online Therapeutic Feedback | 137 Coding of Therapist Behavior and Adherence To subtract therapist behaviors from the 219 online feedback messages, a coding matrix was developed, with 9 main categories and 13 subcategories (see Appendix 1). The coding categories were based on the directed content analysis; categories from prior research [18,19,31] were used to develop the initial coding scheme before analyzing the data [25]. To score the therapistsâ&#x20AC;&#x2122; adherence to the feedback instructions, a coding checklist was created based on the instructions that the therapists received. In total, there were 6 main categories and 19 different subcategories with a dichotomous scale (present or not present, see Appendix 2).

The coding matrix and checklist were first tested by researcher ED by coding 4 feedback messages from 2 randomly selected patients. Each of the included feedback messages was then anonymously coded and scored by researchers MM and SP. For the coding of therapist behaviors, qualitative data analysis software, ATLAS.ti 7.5.18 (ATLAS.ti Scientific Software Development GmBH, Berlin, Germany), was used.

To investigate interrater reliability, both researchers (MM and SP) coded 60 transcripts of therapeutic feedback from 10 randomly selected patients. The intraclass correlation coefficient for the therapist behaviors was .83 (95% CI 0.82-0.85) indicating good interrater reliability, based on 2-way mixed-effects agreement model [32,33]. The interrater reliability for the feedback instruction adherence categories was found to be kappa (Îş)=.84 (P<.001, 95% CI 0.80-0.87), indicating a good agreement between the raters as well. After reaching agreement, the remaining messages (n=159) were equally divided between the 2 coders. As analysis proceeded, additional codes were developed, and the initial coding matrix and checklist were revised, discussed, and refined.

The total frequency of therapist behaviors was calculated with a query tool in ATLAS.ti. A frequency score represented the total number of times the therapist displayed a behavior in the feedback messages sent to the patient (e.g., total number of informing the patient about the assignments). To correct for the number of received feedback messages (e.g., some patients received 4 messages, and the others received 8 messages), relative frequencies were used (frequency of one category divided by the total number of frequencies of all categories per patient). The percentage of therapistsâ&#x20AC;&#x2122; adherence to the instructions for each patient was calculated by the frequency of the adherence (e.g. the total number of times a therapist started with giving a compliment) divided by the total number of messages received by a patient.

Analyses Statistics were conducted using IBM SPSS (SPSS Inc., Chicago IL), version 22. First, descriptive statistics (means, SDs, percentages) were used to describe the patient and therapist sample, number of online sessions, and symptom improvement. Descriptive statistics were then used to examine the frequencies of therapist behaviors and percentages of therapist instruction adherence in the messages to the patients. Spearman correlation analyses, 2-sided, were conducted to assess the relationship between the therapist behaviors, feedback adherence scores and session completion, and symptom improvement. Spearman rho was used to avoid violation of assumptions of normality. Due to the small sample size, only explorative analysis, no missing values imputation techniques and no post-hoc correction for multiple testing (i.e., Bonferroni), were applied.

138 | CHAPTER 6

Results Patients’ and Therapists’ Characteristics A total of 45 patients (73%, 33/45 female, mean age 35.9 years) were given blended CBT in routine care by 19 therapists. Patients’ characteristics can be found in Table 1. Of the 19 therapists (84%, 16/19 female), most were licensed psychologists (53%, 10/19), others were psychologists in training under supervision for health care psychologists (26%, 5/19) or mental health nurses (21%, 4/19). Moreover, 11% (2/19) of the therapists had less than 3 years of professional experience, 26% (5/19) had between 3 and 5 years of experience, 37% (7/19) had between 5 and 10 years of experience, and 21% (4/19) had more than 10 years of experience. The experience with iCBT treatments varied among the therapists: 32% (6/19) had given less than 5 iCBT treatments, 26% (5/19) had given between 5 and 10 treatments, 21% (4/19) had given between 10 and 15 treatments, and 16% (3/19) had given more than 15 treatments.

Table 1. Characteristics of patients at baseline.

Patients’ characteristics

N=45

Gender, female, n (%) Age in years, mean (SD; range) Education level, n (%) Secondary education level Higher education level Employment, yes, n (%) Antidepressant use, yes, n (%) Duration of current depression symptoms, n (%) Duration of current depression symptoms less than 3 months Duration of current depression symptoms between 3 and 12 months Duration of current depression symptoms more than 1 year

33 (73) 35.9 (12.3; 21-64) 15 (37) 25 (61) 21 (51) 10 (24) 8 (20) 22 (54) 10 (24)

Frequencies of Therapist Behaviors and Percentages of Therapist Instruction Adherence Appendix 1 lists the categories, definitions, and examples of the therapist behaviors. In total, 1825 therapist behaviors were coded. The most frequently used therapist behaviors were informing (27.56%, 503/1825; e.g., informing the patient about the next session or specific assignments), encouraging (23.56%, 430/1825; e.g., praising past behavior), and affirming (22.25%, 406/1825; e.g., normalizing behavior, summarizing what the patient has written or said). Making self-disclosures, confronting, and emphasizing the responsibility of the patient were never or rarely used.

An overview of the percentages of the categories, descriptions, and examples of adherence to the feedback instructions can be found in Appendix 2. The therapists adhered in most cases to correct greeting and ending of messages (95.9%, 210/219). They also scored high on writing style (93.6%, 205/219; e.g., limiting of abbreviations and misspellings) and structure (87.7%, 192/219; e.g. limiting to 2 subjects and sending the feedback within 3 working days). Therapists scored the lowest on referring (34%, 74.5/219; e.g., referring to monitoring of the symptoms or reflecting on the dairy).

Within the category communication skills, therapists were very often careful with giving solutions (95.6%, 209/219) and regularly showed in their writing that they read the patients’ homework (88.6%,

Mixed-Methods Observational Study: Online Therapeutic Feedback | 139 194/219). Formulating sentences as hypotheses is something the therapists did not often apply (10.5%, 23/219).

Session Completion and Symptom Improvement The 45 patients completed, on average, 6.3 online sessions (Table 2). On average, patients received 4.9 feedback messages (SD 2.7; range 1-10). One feedback message contained an average of 139 words (SD 95.4; range 1-504), 14.2 words in one sentence (SD 4.0; range 1-26), 3.5 sentences in a paragraph (SD 1.9; 1-17), and 2.9 paragraphs (SD 1.6; 1-10).

From 7 patients, all QIDS data were missing because their therapists did not activate the online monitoring, leaving 38 patients for this exploration. Results on depressive symptoms showed that at baseline, the patients scored, on average, 15.8 points (SD 3.8) on the QIDS, and at post measurement, the patients scored, on average, 11.0 points (SD 6.0), so there was an average reduction of 4.8 points (SD 6.4). Looking at symptom severity at baseline, 8% (3/38) of the patients had mild symptoms, 34% (13/38) had moderate symptoms, and 58% (22/38) had (very) severe symptoms (Table 3). At posttreatment, 21% (8/38) of the patients had no symptoms, 29% (11/38) had mild symptoms, 24% (9/38) had moderate symptoms, and 26% (10/38) had (very) severe symptoms. In total, 63% (24/38) of the patients improved on one or more categories (Table 4). Moreover, 24% (9/38) of the patients showed no change, and 13% (5/38) deteriorated in a category. Appendix 3 contains case descriptions of 3 patients.

Table 2. Treatment completion and duration (n=45).

Treatment completion and duration

Mean (SD; range)

Completed online sessions Completed face-to-face sessions Completed face-to-face + online sessions Treatment duration in weeks Period of online activity in weeks

6.3 (2.6; 2-11) 7.1 (2.7; 2-13) 13.4 (4.4; 5-23) 26.2 (11.2; 8-52) 17.8 (10.9; 2-45)

Table 3. Severity Quick Inventory of Depressive Symptomatology scores at baseline and post measurement (n=38).

Severity Quick Inventory of Depressive Symptomatology

Baseline, n (%)

Post measurement, n (%)

None Mild Moderate Severe Very severe

0 (0) 3 (8) 13 (34) 20 (53) 2 (5)

8 (21) 11 (29) 9 (24) 5 (13) 5 (13)

140 | CHAPTER 6 Table 4. Changes in symptom severity (n=38).

Change in depressive symptom severity

n (%)

Reduction in 1 category Reduction in 2 categories Reduction in 3 categories Deterioration No change

11 (29) 10 (26) 3 (8) 5 (13) 9 (24)

Correlations of Therapist Behaviors With Session Completion and Symptom Improvement One correlation between therapist behaviors and session completion was found (Table 5): the therapist behavior confronting was positively correlated with online session completion (ρ=.342, P=.02). This indicates that more confrontations were related to completing more online sessions. No significant correlations were found with symptom improvement.

Table 5. Correlations of therapist behaviors with session completion and symptom improvement.

Therapist behavior

Session completion (n=45)

Change score Quick Inventory of Depressive Symptomatology (n=38)

Emphasizing responsibility Affirming Clarifying the framework Self-disclosure Informing Confronting Urging Encouraging Guiding Questions

.094 .074 .232 —a −.087 .342b .258 −.054 −.055 .066

.278 .035 .069 — −.249 .184 .310 −.008 .146 .115

aIndicates bP<.05,

"not applicable"; self-disclosures did not occur. a positive correlation indicates more session completion.

Correlations of Therapist Instruction Adherence With Session Completion and Symptom Improvement In Table 6, correlations of therapist instruction adherence with session completion and symptom improvement are shown. Statistically significant negative medium correlations were found between therapist instruction adherence and completed online sessions for structure (ρ=−.340, P=.02) and readability (ρ=−.361, P=.02). Meaning that the more therapists adhered to instructions containing structure (limiting to 2 subjects and sending feedback within 3 working days) and readability (short sentences and short paragraphs), the less online sessions were completed. No significant correlations were found with symptom improvement.

Mixed-Methods Observational Study: Online Therapeutic Feedback | 141 Table 6. Correlations of therapist instruction adherence with session completion and symptom improvement.

Therapist instruction adherence

Session completion (n=45)

Change score Quick Inventory of Depressive Symptomatology (n=38)

Greeting and ending Communication skills Structure Referring Readability Writing style

−.277 −.146 −.340a .170 −.361a −.150

−.064 −.212 −.214 −.085 −.185 −.139

aP<.05,

a positive correlation indicates more session completion.

Discussion Aim of This Study This observational study has uncovered several important factors in the content of online feedback messages in blended iCBT for depression. We further explored therapist behaviors and the extent to which therapists wrote their feedback according to their instructions. In addition, we wanted to know if therapist behaviors and adherence to the feedback instructions could be linked to patient adherence and treatment outcome. The study was carried out in a Dutch sample of participants of the MasterMind study, in routine practice, in a patient population with mild to (very) severe depressive symptoms and with a diverse group of trained and skilled therapists.

Principal Findings Results show that therapist behaviors in relation to the online guidance are informing the patient about the functionalities on the platform, encouraging the patient by praising past behavior or inciting future behavior, and affirming by showing interest in the thoughts, emotions, and behaviors of the patient.

Making self-disclosures, confronting, and emphasizing the responsibility of the patient are never or infrequently used. This is largely in line with the frequencies of the categories found by Holländare et al and may indicate that therapists use the same CBT principles in their written communication as in their face-to-face communication with the patient [19]. Previous research also found that more supportive therapists’ behaviors are used frequently in iCBT and that behaviors such as confronting and self-disclosures are seldom used [18,28]. However, in contrast to the findings of Holländare et al, we found that one and also a different therapist behavior correlated with module completion, and we also found that none of the therapist behaviors were related to symptom improvement. A possible explanation for this difference can be found in the patient group; in Holländare et all’s study, patients with partially remitted depression were included within the context of a randomized controlled trial.

Although therapists applied confronting in limited cases (<1%), this was positively correlated with online session completion. In face-to-face CBT, the occurrence of confrontations has been found to be somewhat higher (6%-14.3%), but is also significantly correlated with therapy outcomes [11]. Hill et al argued that “confrontation often interrupts the client’s thinking by presenting discrepancies and another point of view […]. Although confrontation feels negative at the time, such disruption may be a necessary foundation for change” [15].

142 | CHAPTER 6 Furthermore, therapists followed the feedback instructions that were used in this study on most of the defined elements, such as beginning with a compliment and being careful about providing solutions too soon. Different than expected, only half of the therapists formulated their sentences as hypotheses, and did so in only 10% of the feedback messages (e.g., “It sounds like you are not sure, is that correct?”).

Misspellings occurred regularly: in 21.5% of the feedback messages, therapists made more than 3 spelling mistakes. One of the possible explanations for this is that the treatment platform did not contain a spelling corrector, and it may have taken therapists more time to correct their own writing.

Emoticons were not used often, as only 3 therapists sometimes used an (positive) emoticon. In the “Supportive Accountability” model by Mohr and Cuijpers, it is argued that therapists may mirror the content, style, tense, and cues (e.g., emoticons) in online communication by patients to create mutual trust [31]. In this model, it is also pointed out that people pay attention to the timing and date stamps of the responses. This means responses should be timely because delays may be perceived as expressing lack of affection. In our study, the therapists sent their feedback within the limit of 3 working days in almost 80% of the cases.

Only negative associations were found with therapist instruction adherence and session completion. Providing structure and the readability was significantly negatively associated with session completion. This means that if the therapists adhered more to writing short sentences and paragraphs and the more they limited their feedback to 2 different subjects and sent the feedback back within 3 working days, the less online sessions were completed. These findings might be explained by the adaptive, and also reactive, style of the therapists to the behavior of the patient. When patients are doing well on the online platform, they are more flexible with certain elements of the instructions. On the other hand, when patients display more difficulties or when the therapist gets the feeling that he or she is losing contact with the patient, therapists may be more inclined to adhere more to some parts of the instructions. Schneider et al also found that therapists were responsive in their online feedback and that they increased some behaviors during the course of treatment when patient depressive symptoms worsened. There are similar indications in psychotherapy, where more flexibility of therapists was found related to better treatment outcomes than therapists who were less flexible [34].

Strengths and Limitations The study took place in a naturalistic setting, with routine care patients and therapists and without the restrictions of a randomized controlled trial. Patient demographic characteristics in the study sample are comparable to blended CBT research, also in routine care [9,26]. Previous studies were carried out in small samples of therapists (3-5), often trained students, who delivered treatment in a research setting [16-19]. With the use of a directed approach of the content analysis, the findings of the previous research were supported and extended. We found the same proportions of categories as Holländare et al with the addition of the category “asking questions” [19]. This was also found by Schneider et al when they replicated the study conducted by Paxling et al [18,20].

In addition, there are several limitations to this study. The generalizability of the results is limited because of the small sample size. With a greater sample size, it would have been possible to explore initial symptom severity as a predictor of the use of different therapist behaviors. The exploration of this association would be interesting for further research. Second, although this study was able to capture a group of experienced professionals, the distribution of patients over the therapist was slightly skewed. Half of the therapists treated 3 to 6 patients, and the other half treated 1 or 2 patients. Due

Mixed-Methods Observational Study: Online Therapeutic Feedback | 143 to the small sample size, it was not possible to explore potential differences in writing style or skills between the therapists. Furthermore, in face-to-face treatment, therapist characteristics such as age, gender, and ethnicity of the therapist seemed not to be related to patient treatment outcomes [35], but therapist facilitative interpersonal skills were found to be a successful predictor of treatment outcome [1]. To further explore therapist online feedback, it would be interesting to look at therapeutic skills as well. In this study, there was a high variability in the number of words in the feedback messages, and it could also be interesting to further explore this. Third, within this observational study, only prepost data and explorative analyses and, no post-hoc corrections, were used. The found correlations should be interpreted with care. Previous research showed that it is possible that therapist behaviors change over the course of treatment, with more focus on certain categories at the beginning of treatment versus the end of treatment [19,20]. Finally, this study only focused on the online part of the blended treatment and not on the content in the face-to-face sessions.

Conclusions In sum, this study showed that in blended CBT for depression, therapists primarily used supportive and positive communications like informing, encouraging, and affirming patient behavior. Therapists refrained from using therapeutic techniques, such as making self-disclosures, urging, and confronting.

This can be explained by the way the online feedback instructions were constructed. They provided the therapists guidelines that concentrate on style and form instructions, and this is also reflected in the adherence of the therapists to most of these instructions. It can be suggested that the instructions should also focus more on “disruptive” therapeutic techniques that can foster patients to address their symptoms. The blended format can give the therapist more flexibility in writing feedback because of the combination with face-to-face contact, meaning that therapists can check the interpretation of their online feedback with the patients in the face-to-face sessions. The combination with online contact gives the therapist the possibility to incorporate elements and reflect on issues that were discussed in the face-to-face sessions. On the other hand, therapists are aware that online communications can emotionally positively and also negatively affect the patient, without them being there, and are therefore careful in their communications. The therapists may miss nonverbal cues such as facial expressions and are not able to respond immediately. Writing feedback requires the therapist to assess whether the patient can correctly understand it. The extent to which this calls for specific competencies of the “online” therapist is assumed and requires further exploration. Additional research is needed to further explore the content of online feedback. With an experimental design, more causal explanations can, e.g., be made about the amount of certain therapist behaviors, the interaction with the written content of the patients, patient expectations or the timing of feedback, and also the interaction with the contact of the face-to-face sessions. With more knowledge, instructions on feedback can be enriched, and therapists can be offered more guidance in giving feedback.

144 | CHAPTER 6 Funding The MasterMind project was partially funded under the Information and Communications Technologies (ICT) Policy Support Programme as part of the Competitiveness and Innovation Framework Programme (CIP) by the European Community (Grant Agreement number: 621000).

Mixed-Methods Observational Study: Online Therapeutic Feedback | 145

References [1] [2]

[3]

[4] [5]

[6]

[7]

[8]

[9]

[10] [11]

[12]

[13]

[14] [15]

Andersson, G. & Cuijpers, P. (2009). internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther; 38 (4), 196-205.

Andrews, G., Cuijpers, P., Craske, M.G., McEvoy, P., & Titov, N. (2010). Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a metaanalysis. PLoS One; 5 (10). e13196 Konigbauer, J., Letsch, J., Doebler, P., Ebert, D., & Baumeister, H. (2017). Internet- and mobilebased depression interventions for people with diagnosed depression: a systematic review and meta-analysis. J Affect Disord; 223, 28-40.

Richards, D. & Richardson, T. (2012). Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev; 32 (4), 329-342.

Carlbring, P., Andersson, G., Cuijpers, P., Riper, H., & Hedman-Lagerlรถf, E. (2018). internetbased vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn Behav Ther; 47 (1), 1-18.

Schueller, S.M., Tomasino, K.N., & Mohr, D.C. (2016). Integrating human support into behavioral intervention technologies: the efficiency model of support. Clin Psychol Sci Pract;24 (1), 27-45.

Kleiboer, A., Donker, T., Seekles, W., van Straten, A., Riper, H., & Cuijpers, P. (2015). A randomized controlled trial on the role of support in internet-based problem solving therapy for depression and anxiety. Behav Res Ther; 72, 63-71.

Titov, N., Andrews, G., Davies, M., McIntyre, K., Robinson, E., Solley, K. (2010). Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance. PLoS One; 5 (6), e10939.

Thase, M.E., Wright, J.H., Eells, T.D., Barrett, M.S., Wisniewski, S.R., Balasubramani, G.K., et al.

(2018). Improving the efficiency of psychotherapy for depression: computer-assisted versus standard CBT. Am J Psychiatry; 175 (3), 242-250.

Johansson, R. & Andersson, G. (2012). internet-based psychological treatments for depression.

Expert Rev Neurother; 12 (7), 861-870.

Keijsers, G.P., Schaap, C.P., Hoogduin, C.A. (2000). The impact of interpersonal patient and therapist behavior on outcome in cognitive-behavior therapy. A review of empirical studies.

Behav Modif; 24 (2), 264-297.

Watson, J.C. & McMullen, E.J. (2005). An examination of therapist and client behavior in highand low-alliance sessions in cognitive-behavioral therapy and process experiential therapy.

Psychother Theor Res Pract Train; 42 (3), 297-310.

Watzke, B., Rueddel, H., Koch, U., Rudolph, M., & Schulz, H. (2008). Comparison of therapeutic action, style and content in cognitive-behavioural and psychodynamic group therapy under clinically representative conditions. Clin Psychol Psychother; 15 (6), 404-417.

Knox, S. & Hill, C.E. (2003). Therapist self-disclosure: research-based suggestions for practitioners. J Clin Psychol; 59 (5), 529-539.

Hill, C.E., Helms, J.E., Tichenor, V., Spiegel, S.B., O'Grady, K.E., & Perry, E.S. (1988). Effects of therapist response modes in brief psychotherapy. J Couns Psychol; 35 (3), 222-233.

146 | CHAPTER 6 [16]

[17] [18]

[19]

[20]

[21] [22]

[23]

[24] [25] [26]

[27] [28] [29]

[30]

Sánchez-Ortiz, V.C., Munro, C., Startup, H., Treasure, J., & Schmidt, U. (2011). The role of email guidance in internet-based cognitive-behavioural self-care treatment for bulimia nervosa. Eur Eat Disord Rev; 19 (4), 342-348.

De Bruin, E.J. & Meijer, A.M. (2017). The impact of online therapeutic feedback on outcome measures in Internet-CBTI for adolescents with insomnia. Sleep Med; 29, 68-75.

Paxling, B., Lundgren, S., Norman, A., Almlöv, J., Carlbring, P., Cuijpers, P., et al. (2013). Therapist behaviours in internet-delivered cognitive behaviour therapy: analyses of e-mail correspondence in the treatment of generalized anxiety disorder. Behav Cogn Psychother; 41 (3), 280-289.

Holländare, F., Gustafsson, S.A., Berglind, M., Grape, F., Carlbring, P., Andersson, G., et al.

(2016). Therapist behaviours in internet-based cognitive behaviour therapy (ICBT) for depressive symptoms. Internet Interv; 3, 1-7.

Schneider, L.H., Hadjistavropoulos, H.D., & Faller, Y.N. (2016). Internet-delivered cognitive behaviour therapy for depressive symptoms: an exploratory examination of therapist behaviours and their relationship to outcome and therapeutic alliance. Behav Cogn Psychother; 44 (6), 625-639.

McKeachie, W.J. & Svinicki, M.D. (2010). McKeachie's Teaching Tips: Strategies, Research, and Theory for College and University Teachers (College Teaching Series). Belmont: Wadsworth.

Vis, C., Kleiboer, A., Prior, R., Bønes, E., Cavallo, M., Clark, S.A., et al. (2015). Implementing and up-scaling evidence-based eMental health in Europe: the study protocol for the MasterMind project. Internet Interv; 2 (4), 399-409.

Mol, M., Dozeman, E., van Schaik, D.J., Vis, C.P., Riper, H., & Smit, J.H. (2016). The therapist's role in the implementation of internet-based cognitive behavioural therapy for patients with depression: study protocol. BMC Psychiatry; 16 (1), 338 MasterMind Homepage URL: http://mastermind-project.eu/ [WebCite Cache ID 6wF4PBAQj] Triple-E. Definition blended treatment Heleen Riper, 2017 URL: https://www.tripleehealth.nl/en/projects-cat/depressie/ [WebCite Cache ID 6wZFBY9pX] Kenter, R.M., van de Ven, P.M., Cuijpers, P., Koole, G., Niamat, S., Gerrits, R.S., et al. (2015).

Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: results from a naturalistic study. Internet Interv; 2 (1), 77-83.

Minddistrict platform URL: https://www.minddistrict.com/ [WebCite Cache ID 6wF4sBjHm] Hsieh, H.F., & Shannon, S.E. (2005). Three approaches to qualitative content analysis. Qual Health Res; 15 (9), 1277-1288.

Spijker, J., Bockting, C., Meeuwissen, J.A., Van Vliet, I.M., Emmelkamp, P.M., Hermens, M.L., et al. (2013). Multidisciplinaire richtlijn Depressie (Derde revisie): Richtlijn voor de diagnostiek, behandeling en begeleiding van volwassen patiënten met een depressieve stoornis [Multidisciplinary guideline Depression (Third revision): Guideline for the diagnosis, treatment and counseling of adult patients with a depressive disorder]. Utrecht: Trimbos Institute.

Rush, A.J., Trivedi, M.H., Ibrahim, H.M., Carmody, T.J., Arnow, B., Klein, D.N., et al. (2003). The 16-Item quick inventory of depressive symptomatology (QIDS), clinician rating (QIDS-C), and self-report (QIDS-SR): a psychometric evaluation in patients with chronic major depression. Biol Psychiatry; 54 (5), 573-583.

Mixed-Methods Observational Study: Online Therapeutic Feedback | 147 [31]

[32] [33] [34] [35]

Mohr, D.C., Cuijpers, P., & Lehman, K. (2011). Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res; 13 (1), e30.

Landis, J.R. & Koch, G.G. (1977). The measurement of observer agreement for categorical data.

Biometrics; 33 (1), 159.

Portney, L. & Watkins M. (2000). Foundations of Clinical Research: Applications to Practice.

Michigan: Prentice Hall Health.

Owen, J. & Hilsenroth, M.J. (2014). Treatment adherence: the importance of therapist flexibility in relation to therapy outcomes. J Couns Psychol; 61 (2), 280-288.

Beutler, L.E., Malik, M.L., Alimohamed, S., Harwood, T.M., Talebi, H., Noble, S., et al. (2004).

Therapist variables. In: Lambert MJ, editor. Bergin and Garfield's Handbook of Psychotherapy and Behavior Change. New York: Wiley; 227-306.

148 | CHAPTER 6 Appendix 1. Mainᵃ and subcategories therapist behaviors, definitions, examples and percentages out of 219 feedback messages.

Category therapist behaviors

Definition

Example

%

Emphasizing responsibility

Emphasize the responsibility of the patient

It is useful for yourself to get insight in this by filling in the diary

0.1

Affirming

Give attention/recognize/show interest in thoughts, emotions and behaviors of the patient and to consider them valid Confirm by interpreting and validating what the patient has written Confirm by stating that the behavior of the patient often occurs/is normal Confirm by summarizing and repeating what the patient has written Confirm by summarizing and repeating what the patient has said

Validating and interpreting Normalizing

Summarizing online

Summarizing f2f

22.2

That must be very difficult for you I see that you are struggling with it It is not easy to …. /It is very common that you feel this way

10.9

You write that you…./You are able to focus and enjoy your activities This morning we discussed that you….

7.4

1.4

2.5

Clarifying the framework

Clarify/emphasize/remind the patient of the protocol/framework and give practical information

You gave now finished the first part of the treatment, in the second part we will focus on….

3.9

Self-disclosure

Use experiences and personal examples from the therapist's life

This exercise has helped me with my sleeping problems

0

Informing

Inform or refer to different functionalities on the online platform Inform or refer to the next session and/or its content Inform or refer to the diary and/or its content

Informing about the next session Informing about the diary

Informing about the monitoring Informing about the assignments Informing about the f2f session Confronting

Inform or refer to the monitoring/ questionnaire and/or its content Inform or refer to the assignments in the previous session and/or its content Inform or refer to face-to-face session Express a different opinion or disagree with the patient

27.5

You can now proceed with session 5: Structure and planning It is helpful to set a reminder in your phone to fill in the dairy everyday Your score on the QIDS is 7, so your symptoms are mild now You kept track of your activities during the weekend, I’m curious about your week-days See you next week, at 14.00!

10.4

I do not think that your activities are useless, but I think the opposite!

0.4

2.8

1.5 3.7

9.1

Mixed-Methods Observational Study: Online Therapeutic Feedback | 149 Urging

Urge to let the patient do something

Encouraging

Encourage/motivate past and future behavior of the patient Praise something the patient has done in the past Incite something that the patient is planning to do

Praising past behavior Inciting future behavior Guiding Psychoeducation

Give advice, information or suggestions Give information on psychological processes

Giving suggestions

Give suggestions about alternative behavior/provide advice on how it can be addressed

Questions

Ask or answer questions

Asking questions to clarify

Ask questions to better understand the behavior or emotions of the patient Ask questions to encourage the patient to think further Respond to questions of the patient

Asking ‘thought’ questions Answering questions

ᵃMain categories are in bold.

It is important that you try this

3.9 23.4

Even though you found it difficult, you did it, very good!

Continue to keep up this good work! Good luck!

14.5 8.9 11.3

The fact that it doesn’t immediately give more pleasure is something we see often in a depression Try to think of something that’s easy to do and cannot easily go wrong

5.5

5.8

6.9 Is that something you already do?/ … is that correct?

3.4

How would you like to feel and what should your life look like?

Yes, I will repeat this session for you, so that you can practice more

3.2 0.3

150 | CHAPTER 6 Appendix 2. Mainᵃ and subcategories feedback instructions, descriptions, examples and percentages out of 219 feedback messages.

Category feedback instructions Greeting/ending Correct greeting Correct ending Communication skills Begin with compliment Summarize

Reading homework Hypotheses Giving no solutions

Structure 2 subjects Within 3 working days

Referring Referring diary

Referring monitoring

Referring next session

Referring next f2f session Readability Short sentences

Description in instructions

Example

Use a greeting Use a correct ending

Dear [name patient] Good luck with the next session!

Greetings, [name therapist]

Begin the message with a compliment/positive approach Summarize the assignments of the patient, use his/her own words

From your answers I can see it was difficult, but you did very good!

You clearly described your depression, the loss of interest/energy, negative thoughts and worrying.

I see that you…/You say that you…./If I understand you correctly…

That sounds like….. , is that correct?..

If I understand you correctly…..

When was the last time you felt that way? What did you think and what did you do differently then?

Show that you have read the homework by using their examples Formulate sentences as hypotheses Give no solutions, let patients think of their own solutions. You can give them a starter in the right direction

Confirm the next f2f appointment

Write short, clear sentences.

95.9 95.4 96.3 69.4 79.0 73.1

88.6 10.5 95.6

87.7 95.4

Give feedback on max two subjects in order to keep the structure clear Give feedback within three working days. Make an agreement when the patient should finish the session and when you as a therapist can write the feedback Reflect on the diary or when the patient did not succeed yet, address (again) the utility of the diary Refer to the monitoring questionnaire or when the patient did not succeed yet, ask to fill this in next time End the message with a reference to the content of the next session

Yes %

79.9

34.0 Good that you used the diary. I can 26.9 see that your mood is different in the mornings than in the evenings, it that something you also experience?

Your score on the QIDS is 7, so right 11.4 now your symptoms are mild.

I will open the next session for you.

There you can further practice with the schemas We will see each other again on May 19.

41.6

56.2 68.0 61.6

Mixed-Methods Observational Study: Online Therapeutic Feedback | 151 Short paragraphs Writing style Limit abbreviations Limit misspellings Limit emphasis

Correct emoticons

Write short, clear paragraphs Limit your abbreviations Be aware of misspellings Limit text in caps, exclamation marks, underline text or make text bold Use emoticons only when you know the meaning is clear for the patient and use it as a complement and not as a substation

áľ&#x192;Main categories are in bold.

74.4 93.6 97.3 78.5 98.6

And do not forget the fun activities ď &#x160; 100

152 | CHAPTER 6 Appendix 3. Case descriptions of three patients Case 1: patient K. Background information K. is a 40 year old father of one child. He works as a chef. He suffers from fatigue, feelings of exhaustion, anhedonia, emotional instability and excessive guilt. His symptoms seem to be caused by an accumulation of stressors over the last few years (high workload, illness of his mother, job change of partner and worrying about his daughter), in combination with a habit of making high demands on himself and a need to always be strong. This is the third time he is experiencing these symptoms. There seems to be an overload of stress for some time now, which has now led to the development of a depressive disorder.

Treatment course K. completed 11 online sessions (repeated online session 3) and had 5 face-to-face sessions with his therapist. At the beginning of treatment K. scores 17 on the QIDS, which is considered severe. The treatment was successful, after 15 weeks of treatment the depression was in remission (QIDS score 7, severity mild). Most used therapist behaviors were Affirming (relative frequency 24.7), Informing (relative frequency 19.7) and Encouraging (relative frequency 18.3).

Graph 1. QIDS scores of patient K. 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Table 1. Quotes and therapist behavior codes of online sessions of patient K. Online sessions Quotes from feedback messages Online session 2 ‘Clarifying your symptoms’

Online session 3 ‘Motivation and setting goals’

Online session 10 ‘Looking at the future’

Feeling guilty, is that something you always do (to a greater or lesser extent)? Or do you suffer from it now, during your depression? (In any case, it is also one of the symptoms of depression, but it can also be something that you do anyway, and then it is probably a longstanding thought pattern.

We have just discussed that you will repeat this session again. You will briefly describe the day, so that you can get concrete goals for the coming period. What do you want to achieve with the therapy?

Make that concrete and think about one (or more) steps that you can take. We just discussed that it might be wise, for example, that you give your child more responsibility. You do not have to do everything for her. Children are allowed to do tasks, that is also part of parenting.

You have achieved good results with the therapy. You can enjoy time with your child and friends again and you realize how this happened by structurally taking too much hay on your fork. You now know how it feels to be rested. You have gained many insights by critically examining your own thoughts. Keep challenging your thoughts, so that your new thoughts / rules of life will become increasingly credible. Your score on QIDS = 7, so still mild complaints (0-5 = no depressive symptoms). This means that you experience a lot of improvement, the symptoms you have are still light and I expect if you persevere, that you can maintain the improvement.

Codes asking questions to clarify, psychoeducation

summarizing f2f, asking ‘thought’ questions, giving suggestions, normalizing

summarizing online, inciting future behavior, informing about the monitoring, validating/ interpreting

Mixed-Methods Observational Study: Online Therapeutic Feedback | 153 Case 2: patient A. Background information A. is 25 years old and experiences problems with her work. Her symptoms have been there for two to three years now. She has a negative self-image and shows a significant amount of avoidance behavior. She sometimes thinks of suicide, but says she doesn’t want to die. Her family lives in Turkey.

Treatment course A. completed 8 online sessions (repeated online session 5) and had 5 face-to-face sessions with her therapist. At the beginning of treatment she scores 18 on the QIDS, which is considered severe. After 17 weeks of treatment A. is still experiencing severe symptoms of depression and she scores 21 on the QIDS. Despite this, she indicates at the end of treatment that she has gained a lot of insight from the treatment. Apart from the online sessions, she also received guidance from a self-help book about a negative self-image.

After treatment she wants to continue to use the online thought-schemes. Most used therapist behaviors were Informing (relative frequency 31.6), Affirming (relative frequency 21.1) and Encouraging (relative frequency 19.3).

Graph 2. QIDS scores of patient A. 25 20 15 10 5 0 1 2 3 4 5 6 7 8 9 1011121314151617

Table 2. Quotes and therapist behavior codes of online sessions of patient A. Online sessions Quotes from feedback messages Online session 2 ‘Clarifying your symptoms’

Online session 5 ‘Pleasure and fulfilment’

Online session 6 ‘Structure and planning’

Very nice that you managed to go through the first session and that you succeeded in completing the exercise. Good to hear that you have experienced it as helpful to put everything in order. Good that you filled in the questionnaires! We will discuss them later. I see that you also kept the diary for one day. It is important to do this consistently so that we get an idea of what the link is between your activities and the symptoms you have.

I see that thinking of enjoyable activities goes well, but that you find it difficult to implement them. We have already discussed this in the face to face sessions several times. It's good that you've tried it. You say that you want to do the session again.

Shall we agree that you will do it one more time and then actually carry out the intended activity? Try to think of something that is easy to carry out and that cannot go wrong.

Whatever happens, allow yourself this activity. Maybe you can also think of a way to reward yourself when you succeed?

You indicated earlier that you benefited from previous structures and planning for work. So it is worth to try to incorporate this in your private activities as much as possible as well. Try to make a planning for each day by using the steps: Step 1: Plan your every-day activities, Step 2: Plan your mustactivities, Step 3: Plan your enjoyable activities. Good luck with the next module about the thought-scheme!

Codes praising past behavior, informing about the monitoring, diary, urging

validating/interpreting, summarizing f2f, praising past behavior, giving suggestions, urging, asking ‘thought’ questions

summarizing f2f, informing about the assignments, giving suggestions, inciting future behavior

154 | CHAPTER 6 Case 3: patient M. Background information M. is 25 years old. His depressive symptoms are associated with emotional neglect by his parents. M. is very driven and ambitious, in combination with a vulnerable and low self-esteem. He supports his parents wherever he can, especially financially, but doesn’t get recognition for this. M. also suffers from concentration and reading problems following an accident.

Treatment course M. completed 4 online sessions and had 8 face-to-face sessions with his therapist. At the beginning of treatment he scores 19 on the QIDS, which is considered severe. At the last known QIDS he scores 11. M. indicates that the therapy has helped him to get more insight into the dynamics within de family. He feels less responsible for his parents. His mood has improved. His self-image and knowledge about his needs remain diffuse. Most used therapist behaviors were Encouraging (relative frequency 28.6), Affirming (relative frequency 21.4) and Questions (relative frequency 21.4).

Graph 3. QIDS scores of patient A. 20 15 10 5 0 1

2

Table 3. Quotes and therapist behavior codes of online sessions of patient A. Online sessions Quotes from feedback messages Online session 2 ‘Clarifying your symptoms’

Online session 3 ‘Motivation and setting goals’

Online session 4 ‘How active are you?’

I see you've been working well on the assignment. What was it like to write about your depressive symptoms? It is indeed difficult if you are not in a good mood to do things. And that if you manage to do things, that this will have a positive influence on your mood.

Thank you for completing the assignments. Well done. You write that is the first step is very difficult, since you want to be less busy, but that means more free time and peace and that turns into thinking and worrying. But is this then a good reason not to take a rest? What could you do about the rumination?

Good that you have found a way to deal with this exercise.

Has it helped to get better insight between the relationship of activity and mood? You can find more helpful questions in the assignment. Good luck.

3

4

Codes praising past behavior, asking ‘thought’ questions, validating/interpreting praising past behavior, summarizing online, confronting, asking ‘thought’ questions

praising past behavior, asking questions to clarify, informing about the assignments, inciting future behavior

Mixed-Methods Observational Study: Online Therapeutic Feedback | 155

156 | CHAPTER 7

GENERAL DISCUSSION| 157

CHAPTER 7

7

GENERAL DISCUSSION

I would like to use bCBT in the future, but I also would like to have more ideas about this; to find out when it works, for whom it doesn't work, figure out what you can add. Of course you have to be careful in creating a kind of store in which you will go shopping without having properly figured out for whom it fits and when it fits. It would be nice if more precision is given about this.

Therapist Tom

158 | CHAPTER 7 Previous research has shown that internet-based Cognitive Behavioral Therapy (iCBT) is promising in the treatment of depression, but the uptake in routine practice is slow. The MasterMind project aimed to monitor and upscale the use of iCBT for depression in fifteen European regions, in order to close the gap between research and practice (www.mastermind-project.eu). A successful implementation of interventions in mental health care can be influenced by many factors on multiple levels such as patient, therapist and organizational levels. The central aim of this thesis was to gain a deeper insight into the level of the therapists regarding the barriers and facilitators when implementing iCBT for depression in routine mental health care practice.

Summary of main findings This PhD’s starting point consisted of an exploration of existing barriers and facilitators when implementing eMental health (eMh) interventions for mood disorders in routine practice. We systematically reviewed the literature on patient, therapist, organizational, and health care system level to address the implementation gap between evidence-based interventions in research settings and their actual implementation in routine practice (Chapter 2). In total, we screened 13,147 articles, of which 48 studies were included in the review. Through thematic analyses, 37 determinants were identified, of which three were reported most frequently: 1) expectations and preferences of patients and therapists about receiving and providing eMh in routine care, 2) the appropriateness of eMh in addressing patients’ mental health disorders, and 3) its availability, reliability, and interoperability with other existing technologies such as digital patient health record systems. These were important factors for therapists to consider using iCBT or to remain engaged in providing it to their patients. This review made us aware that the majority of the studies were of a descriptive nature, focusing on the patient and professional level. Given the status of this emerging field, this was not unexpected.

Research findings showed that the eMh implementation factors that affected therapists were mostly based on expectations and preferences of therapists about eMh and not on actual experiences with internet-based interventions for depression. Therefore, qualitative research among therapists who actually use eMh might add more in-depth insight into the underlying reasons for the low uptake of iCBT in routine practice. We carried out this qualitative research among therapists within the context of the European Mastermind study. In Chapters 3 and 4 the study protocol and results of this part of the MasterMind study in the Netherlands were described. We analyzed qualitative interview data of 36 therapists working at 7 mental health care organizations where blended CBT (a combination of online and face-to-face sessions) had been implemented. Therapists with bCBT experience found the usability sufficient and were generally satisfied with providing bCBT to their patients. The thematic analysis showed three main themes on promoting and hindering factors when working with bCBT: 1) therapists’ needs for bCBT uptake (e.g. absence of ongoing training with a focus on therapeutic content), 2) therapists’ role in motivating patients for bCBT (e.g. uncertainty about eligibility of patients) and 3) therapists’ experiences with bCBT (e.g. helping to contain therapist drift). Overall, therapists found that bCBT is a good format for patients they treat in routine practice; they assumed it can be offered by most regular CBT therapists and they expect the uptake to grow in the future. In particular, the pre-set structure of bCBT was found to be beneficial. That said, therapists did not experience promised time savings in their daily practice; it even took them more time.

In addition to the qualitative interviews, we used the System Usability Scale (SUS) as an instrument to measure usability of iCBT and bCBT. The SUS is an often used validated questionnaire to measure

GENERAL DISCUSSION| 159 usability of applications in industrial contexts. However, this instrument was not yet validated for testing usability of eMh interventions. In Chapter 5 psychometric properties such as the factor structure of the SUS was evaluated in a sample of professionals using iCBT and bCBT for depression from six European countries in the MasterMind project. Reliability was assessed using complementary statistical indices. In addition, the convergent validity of the SUS was tested with three questions on satisfaction (Client Satisfaction Questionnaire-3). Based on a Confirmatory Factor Analysis we concluded that although the SUS appears to have a multidimensional structure, the best model showed that the total sum-score of the SUS is a valid, reliable and interpretable measure to assess the usability of internetbased interventions, when used by health professionals in routine mental health care.

In Chapter 6 we zoomed in on therapistsâ&#x20AC;&#x2122; behaviors in the online messages they send to their patients. The online bCBT platform offers a unique opportunity to gather this information by extracting the messages from the online platform and studying their content in detail. Online feedback by therapists is a substantial part of internet-based interventions, but it is unknown how feedback is given, and how therapists make use of the instructions for online feedback. We therefore investigated therapist behaviors in online feedback and therapistsâ&#x20AC;&#x2122; adherence to feedback instructions. We also explored if the behaviors and feedback adherence were associated with patient outcomes (e.g. completed online sessions and depression symptoms). A total of 19 therapists sent 219 feedback messages to 45 patients in routine secondary mental health care. The most frequently used therapist behaviors in the online feedback were informing, encouraging, and affirming. These were, however, not related to patient outcomes. Although infrequently used, confronting (e.g. the therapist expressed a different opinion or disagreed with what the patient wrote) was positively correlated with session completion. Therapists adhered to most of the feedback instructions. Only 2 feedback aspects were correlated with session completion: the more strictly therapists adhered to instructions containing structure (limiting to 2 subjects and sending feedback within 3 working days) and readability (short sentences and short paragraphs), the fewer online sessions were completed. We found no associations with depression symptom change scores.

Reflection on the main findings Therapistsâ&#x20AC;&#x2122; barriers to iCBT uptake The findings from the systematic review taught us that there are several factors that prevented therapists from providing iCBT to their patients with depressive symptoms (Chapter 2). The most frequently mentioned therapist barriers for uptake in the literature we encountered were: preferences for face-to-face treatments, negative expectations about extra workload and concerns regarding the establishment of a positive therapeutic relationship. Yet, these preferences, expectations and concerns were predominantly based on attitudes of therapists who did not have experience with iCBT themselves. Another major barrier for uptake was the limited awareness about the availability, possibilities and benefits of iCBT among therapists in routine care. From the studies that included therapists with (some) iCBT experience, other barriers were raised, with iCBT training, usability and interoperability with other existing technology (e.g. digital patient health records) mentioned most.

Therapists encountered, for example, a lack of practical training to integrate iCBT into routine practice and they found that limited time was offered to explore the internet-based interventions, which resulted in inadequate knowledge.

160 | CHAPTER 7 Studies on perspectives from therapists with iCBT experience in routine care are however limited, especially concerning bCBT. In our qualitative study (Chapter 4) with bCBT trained therapists we found similar barriers to the above-mentioned barriers: therapists were hindered by a lack of knowledge on patient eligibility, knowledge on clinical effectiveness or by negative expectations about the therapeutic relationship. Another major corresponding barrier was training of the therapists. Therapists in our study felt that being offered one training was not enough to increase their uptake of bCBT. The ‘train and pray’ has long been recognized as an unlikely effective implementation strategy in training therapists [1]. Moreover, other research showed that there is little to no evidence that asking therapists to read treatment manuals or to attend one training session will result in a positive, sustained outcome (i.e., increased skill and competence) [2]. The training set-up was also a barrier to therapists in our study: training was perceived as too technical and not sufficient to enable therapists to use bCBT in their daily workflow because they missed ongoing support after the training. Similar research on bCBT confirmed this [3]. Furthermore, it must not be underestimated that working with new protocols and guidelines demands ongoing facilitation for therapists from the organization. Similar to face-to-face psychotherapy protocols, patients often receive a modified version of the original intervention that is not always in line with existing evidence and is thought to be related to the organization context [4].

To increase adherence to treatment protocols, therapists are in need of regular supervision, team meetings and a shared treatment vision [4].

Once therapists started offering bCBT to patients with depression in their daily practice, other barriers were encountered. They had difficulty integrating bCBT into their workflow. Not only because of the known barrier of technical disconnection with existing system for patients’ administration, but also because of uncertainties regarding the bCBT protocol and logistic integration into their daily therapeutic and administrative schedules. The main challenge for the therapists was perhaps not experiencing the promised time savings; for some, it even generated a higher experienced workload, especially in the beginning.

Another important factor is that the combination of lack of adequate knowledge about the benefits and possibilities of bCBT, (ongoing) therapeutic training and guidelines for bCBT, may also have caused different interpretations of the protocol. Individual uptake of bCBT partly resulted for some in using the online platform on top of the face-to-face treatment instead of replacing a portion of the face-toface sessions.

Therapists’ facilitators for iCBT uptake In the literature we found several factors that can facilitate therapists’ iCBT uptake (Chapter 2). Uptake can be increased by raising awareness about iCBT, by allowing therapists to gain experience with iCBT and providing them a specific training can change their knowledge about, attitudes towards and confidence in delivering iCBT for depression. A survey among 124 Australian mental health workers explored the role of knowledge and change of negative attitudes towards iCBT in general [5]. It was found that even a short presentation could change negative attitudes into a more positive view regarding iCBT. The results also suggested that more knowledge can positively impact the perceived disadvantages of iCBT and the circumstances in which iCBT is perceived to be advantageous. The authors concluded that in disseminating iCBT to therapists, information on iCBT should therefore not only focus on effectiveness, but also include disconfirmation of common misconceptions as well (e.g. ineffectiveness for severe depressive symptoms or fear of being replaced by technology).

GENERAL DISCUSSION| 161 In our qualitative study (Chapter 4) we found indications that gaining experience with internetbased interventions might indeed be key in changing the perceptions, expectations and concerns of the therapists. Therapists who lacked bCBT experience had different concerns and views on, for example, patient eligibility, personalization of the protocol and therapeutic relationship than therapists with bCBT experience. We found that experienced therapists said there was no typical patient for bCBT, or that patients drop out more, or that it was impossible to develop a positive therapeutic relationship; that the same influencing factors applied to face-to-face CBT. One of the main reported advantages of bCBT in our study was the focus and pre-set structure that made therapists and also their patients more adherent to the protocol and helped therapists to avoid therapistâ&#x20AC;&#x2122; drift. The structure of the bCBT protocol could potentially also positively impact treatment duration in comparison to regular face-to-face CBT. A recent pilot RCT on bCBT for depression showed that bCBT in comparison to face-to-face CBT may contribute to shorter treatment duration (19 versus 33 weeks) by facilitating the delivery of treatment sessions (online and face-to-face) [6]. And after 30 weeks of treatment in this study, 81% of the patients who received bCBT completed 75% or more of the protocol versus 45% in the group who received face-to-face CBT. Measuring therapistsâ&#x20AC;&#x2122; iCBT satisfaction and usability We found that trained therapists who used bCBT in routine practice were in general satisfied and rated the usability of the online platform as sufficient (Chapter 4). For measuring satisfaction, we used the three questions from the CSQ, adapted to the therapists (e.g. CSQ-3: In an overall general sense, how satisfied are you with the bCBT treatment you have provided?). The CSQ has shown good reliability (omega = 0.95) and validity in a sample of internet-based depression intervention users [7]. Usability was measured using the SUS, a popular instrument to measure usability of websites and products in industrial contexts. Although the SUS is increasingly used as a measure for usability of internet-based interventions in mental health care, it is unknown whether it is a valid instrument to measure usability in this new context. We found that in a sample of health professionals from 6 European countries, the SUS appeared to be a reliable (omega = 0.91) and valid measure to assess the usability of internetbased interventions (Chapter 5). Results from the factor analysis seemed to indicate that the SUS had a multidimensional structure, meaning that the SUS gives a score for overall usability, as well as scores for two subscales: Usability and Learnability. Further analysis showed that the subscales had no information that is not already contained in the total sum-score.

The factor structure of the SUS could unintentionally be distorted by the mix of positive and negative items. The mixed tone was originally used to control for acquiescence bias; the hypothesized tendency of respondents to agree with statements with a mix of positive and negative tone [8]. In addition to distorting the factor structure, the risk of mistakes by respondents and miscoding by researchers may be increased. In 2011, a positive version of the SUS was tested and there was no significant difference between the mean overall SUS scores of the positive and mixed versions [9].

Therefore, it would be advisable to use an all-positive version of the SUS. It is interesting however that despite the fact that many other studies encountered an unintentional factor structure and with the availability positive version, most researchers are still using the original mixed version.

In addition to usability and satisfaction there are many other factors that might be relevant to measure regarding the uptake of iCBT, such as the quality of the content of the online sessions or online feedback. More importantly, standardized and theory-based questionnaires are underdeveloped in the

162 | CHAPTER 7 field of implementation in mental health [10]. Many existing questionnaires lack good psychometric properties and instruments measuring feasibility, appropriateness, and sustainability are especially needed. According to Lewis and colleagues (2015), the field will struggle to identify which implementation strategies work best, for which organizations, and under what conditions, until psychometric strength is established [10].

Therapistsâ&#x20AC;&#x2122; online feedback Providing online feedback is a substantial part of the treatment format of iCBT and bCBT. A key problem in bCBT uptake in routine care, might be that providing online feedback demands a new skill set of the therapists (Chapter 4). Writing feedback takes more time than is generally assumed, especially in the beginning. In the study of Kooistra et al. (2019), it was estimated in advance that providing feedback per patient would take about 30 minutes per session in bCBT [6]. However, this may cost much more time in the exploratory phase when therapists need to grow accustomed to a new way of working. We found that in training, therapists are in need of more instructions and tools, and examples on online feedback should be included; e.g. how to address something negative (e.g. misunderstanding of an assignment), how to handle or prevent miscommunication, and how to adapt to different patient types.

In an explorative study (Chapter 6) we looked in depth at the online feedback instructions therapists received in bCBT for depression and how they actually provided this online feedback. The manual with feedback instructions for therapists comprised general and specific elements important to the structure of the messages (e.g. correct greeting, limiting to 2 subjects), readability (short sentences and paragraphs), writing style (e.g. limiting abbreviations and misspellings, use of emoticons), referring to parts of the treatment (e.g. diary entries, referring to the next online session), and communication skills (e.g. summarizing, not providing solutions). Our findings suggested that the instructions provided sufficient guidance to communicate in a supportive and positive manner with patients. We identified the most frequently used therapist behaviors to be informing, encouraging, and affirming. These were in line with other studies on online feedback in iCBT [11â&#x20AC;&#x201C;13]. Although we know that more therapeutic techniques, such as confronting or making self-disclosures are significantly correlated to treatment outcomes in face-to-face CBT [14], in our study therapists used these techniques to a very limited extent or not at all. Although therapists applied confronting in a few cases, we found an indication that confronting might be related to patients completing more sessions. We also found that if the therapists adhered more to the instructions regarding structure and readability, the less online sessions were completed. These findings might be explained by an adaptive style of the therapists to the behavior of the patient. Therapists might be more flexible with certain instructions, when the patients are doing well on the platform. On the other hand, therapists may be more inclined to adhere more to some of the instructions when patients display more difficulties or when the therapist gets the feeling that he or she is losing contact with the patient. Moreover, in training and in the manual, therapistsâ&#x20AC;&#x2122; instructions for written online feedback are mainly focused on style and form. Therapists in routine care may be reluctant to use more therapeutic techniques in their online feedback. Online feedback instructions might therefore be improved by adding more therapeutic techniques besides the focus on style and form. In addition, to provide therapists more support on this subject in supervision. A recent study used a deep learning model to automatically categorize therapist online feedback from approximately 90,000 hours of iCBT [15]. It was found that increased therapist use of cognitive and behavioral techniques (e.g. cognitive restructuring, psychoeducation) was positively associated with

GENERAL DISCUSSION| 163 reliable improvement in patient symptoms and engagement. In addition, it was found that the more therapists addressed non-therapy related content such as technical/practical issues or non-therapeutic conversations, the less patients improved. Exploring the content of online feedback and how therapists are trained in providing this, could be an interesting area of further research.

Methodological considerations In this thesis, we took a wide perspective on the role of the therapist in the implementation of iCBT and bCBT for depression in routine practice by including review, explorative, psychometric and qualitative studies. Results of this thesis should, however, be interpreted in the context of the following methodological considerations.

In the protocol paper, we described the RE-AIM framework, which we intended to apply to structure therapist data in terms of Reach, Effectiveness, Adoption, Implementation and Maintenance [16]. We used the framework to help us guide the formulation of the interview questions and structure the analyses. We additionally intended to use the framework in reporting the results. However, it turned out to be difficult to apply all RE-AIM dimensions, given constraints such as restricted time, data availability and data accessibility to answer these questions. We therefore wanted to focus on the most important factors for therapist uptake. Attempting to take account of all dimensions turned out to be too extensive and would have compromised the level of depth. In addition, we intended to report on the RE-AIM dimensions with existing data from administrative systems, Routine Outcome Measurements (ROM) and the online platform (log-data) to keep the burden on therapists (and patients) as low as possible. However, we could not report on several quantitative measures (e.g. reach of therapists, therapist treatment fidelity) because of several encountered problems and important lessons learned that go beyond the processes involved around iCBT and bCBT. 1) There was a transitional period of tightening privacy legislation that made it more difficult to gain access to patient treatment data from the electronic patient dossier (EPD) because access rights changed. 2) There also was a lack of predetermined agreement and ownership of the log-data between the mental health care organizations and the online platform provider. A part of the log-data turned out to be inaccessible. 3) Moreover, both the EPD and the platform-data were not equipped to extract data for research purposes, and lacked interoperability. 4) Guidelines on how to report the online activities within blended treatment in the EPD were unclear to the therapists, which resulted in unreliable data because it was done a different ways by individual therapists. Therefore, we chose to adapt the methodology by reporting the qualitative part that focused on therapistsâ&#x20AC;&#x2122; perspectives. It would benefit future research and mental health care organizations to facilitate, manage and stimulate therapists to consistently register their treatment activities and monitor patients progress by creating a unified userfriendly administrative system, and by providing therapists with routine (visual) feedback on their activities and caseload. Combining data from administrative systems, ROM and log-data from the online platforms could not only help therapists to improve their treatment, it also could give researchers more access and insight into the â&#x20AC;&#x2DC;black boxâ&#x20AC;&#x2122; of (i)CBT. We considered it a strength to include a heterogenic sample of therapists, with different professional backgrounds, differences in experiences with bCBT and exposure to various implementation strategies from a number of mental health care organizations. This enabled us to ensure a broad exploration of their perspectives. However, a relatively small sample group prevented us from making specific statements about possible determinants, especially about the therapists who

164 | CHAPTER 7 are less open to iCBT. We know for example that the intention to use iCBT can be related to influencing factors such as professional background. Cognitive-behavior-oriented therapists tend to be more positive than therapists with a psychoanalytic background [17] and knowledge can positively impact their attitude [18]. Maybe in contrast to general expectation, age seems unrelated to therapistsâ&#x20AC;&#x2122; intention to use iCBT [19]. In health care, age and also gender were also found to be unrelated to attitudes of professionals towards the use of technology [20]. It would be interesting to further explore therapist characteristics such as previous amount of experience with iCBT or bCBT with selecting, tailoring and adapting implementation strategies.

Implications for clinical practice The implementation of bCBT in routine care in the Netherlands has, like many other technological innovations, been disruptive through all levels in mental health care organizations: From the moment patients enter treatment, through administrative services, management levels, organizational policy, ICT staff, connections with primary care, to patient aftercare. In the long run, the disruption may bring many potential benefits for the entire system, and specifically for the constrained work field of the therapists. However, this point has not yet been reached. This is also related to the fact that despite bCBT being considered in many ways to be very similar to CBT, it may have been underestimated that for therapists this is a different way of providing treatment and it entails many more substantive changes than initially assumed by the mental health care organizations. It is possibly just as different for an online therapist to give treatment face-to-face.

The perceived and encountered therapistsâ&#x20AC;&#x2122; barriers and facilitators might not be solely applicable to bCBT for depression. There are common barriers to other technological innovations (e.g. mobile apps, videoconferencing) in other contexts (primary care, inpatient care, health care) as well; there is also limited awareness and knowledge of the innovations among patients and professionals, and there are concerns or misconceptions about the appropriateness, technical usability and reliability of the innovations [e.g. 21,22]. A meta-review of 44 systematic reviews on eHealth implementation in health care settings found that multiple factors were important for implementation, and no single factor was identified as a key barrier or facilitator [23]. Another meta-review of implementing interventions in primary care showed that there might be factors that are particularly relevant for eHealth [24]. The expectation of a more efficient workflow was an important factor only for the implementation of ehealth interventions and was not present for other types of interventions (e.g. guidelines or evidencebased practice, integration of collaborative working). Also, adaptability and costs seem to be eHealthspecific factors. All this teaches us that the implementation of technological innovations in care is complex. We can learn from the different approaches and effective strategies for implementation.

However, every context is different, has its own relevant stakeholders, and moreover there are no magic bullets in effective implementation strategies [25]. So far, it seems that mental health care organizations have used ad hoc strategies and traditional ways to implement bCBT; they have mainly focused on building a technical infrastructure and training therapists in the technical basics; and they have underestimated the disruptiveness of implementing bCBT in all processes of the mental health care system and in the way therapists normally work, combined with a greater priority for face-to-face treatments.

To support the therapist in the process of increasing uptake, several specific strategies can be taken into account. A first strategy would be to return to informing therapists about bCBT. The main promise

GENERAL DISCUSSION| 165 of bCBT has been its cost-effectiveness and efficiency; however this has not been experienced by all therapists yet (Chapter 4). As therapists think that bCBT can be beneficial for delivering good quality CBT and can help them to structure the treatment, which is considered an advantage for both patients and therapists, the future main message of bCBT could focus on treatment quality and structure. There is evidence that the pre-set structure of bCBT can increase treatment adherence, and could potentially speed up patient flow through the treatment process in routine specialized mental health care [6]. In addition, attitudes of therapists might be positively influenced by providing them with information targeting the misconceptions about bCBT. Furthermore, the dissemination of bCBT may have caused therapists to have concerns about being replaced by technology and being controlled by higher organizational levels (Chapter 4). It could help mental health care organizations to remove the ‘i’ from iCBT or ‘b’ from bCBT, and to offer it as a standard when delivering CBT instead of merely as part of a project and/or being dependent on research initiatives. Making bCBT mandatory for all therapists could work counterproductively. It may instead be helpful to adopt a ‘blended, unless’ strategy, whereby bCBT is be offered to every patient, unless there are compelling reasons not to. This could help therapists to make it a routine part of their work. It is probably crucial to get them to experience the benefits by themselves; we know that the uptake can be facilitated by allowing therapists to gain experience (Chapter 4). Considering that this process may take some time and will be different for every therapist, organizations should find ways to induce therapists to do this, without creating a higher workload for them. For example, assign the role of introducing patients to the platform to support staff; this way patients will have already received login instructions and are acquainted with the platform and its possibilities.

A second strategy would be to improve the way therapists are educated and trained, as bCBT demands a specific skillset: to integrate bCBT in the education of therapists in order to normalize the use of internet-based treatment for the next generation; to aim to train all therapists, even if therapists only use the platform for online (secure) communication with patients; and to improve therapist training. In training, therapists need more guidelines beyond the technical and general platform instructions on how to use the protocol for depression. They need to learn how to motivate their patients for bCBT, how to combine the face-to-face and online part and importantly, how to provide online feedback. Some studies have already developed theory-based iCBT training focusing on enhancing the therapists skills, including deciding when to use iCBT and providing online feedback [26,27].

Third, an overall strategy is to tailor and adapt strategies to the bCBT experience level of the therapists. In addition, to focus first on the most experienced group as the most negative group of therapists is likely to shrink in the future. Once a good foundation is built for the most experienced group, the rest of the therapists can be targeted based on their bCBT experience and skills, supported by the lessons learned from the experienced group.

The aforementioned strategies may only be effective with the preconditions that 1) therapists receive ongoing support from a well-equipped accessible helpdesk for technical issues, as online services will always be subjected to updates and technical glitches, 2) ongoing treatment content support is additionally available from a core team of experienced therapists, 3) the administrative systems and the online platform are fully technically integrated and redundant systems are deimplemented, 4) the mental health care organizations invest in a shared treatment vision on bCBT.

166 | CHAPTER 7 It is sometimes expressed that bCBT is just CBT in a different format. However, it might be key to value the online platform as more than just a tool. It is a tool that generates a different workflow and demands a new skillset from the therapists. Moreover, even little technological elements (e.g. reminders, rewards or updates) in bCBT or other eHealth services can create a big impact on the adherence of patients [28]. Within mental health care organizations, therapists actually work fairly autonomously. It is therefore perhaps not surprising that they hold back when they feel that a new way of working is imposed from above. Often, they feel that organizations are doing it for political reasons, rather than treatment-related reasons. Furthermore, one of the biggest problems they encounter is that working with bCBT does not solve their time constraints and relieve them from administrative tasks. But it potentially could. Although it might be CBT in a different format, it takes time to learn to work with bCBT, but in the end it could make therapistsâ&#x20AC;&#x2122; work easier.

Suggestions for future research Research on dissemination and implementation of iCBT and bCBT for depression is a young area of empirical investigation. We are currently in the phase of initial implementation. Before discussing the suggestions for future research, there is an inevitable question that needs to be answered: should iCBT be implemented in routine depression care? Although it was not the focus of this thesis, we argue that iCBT should be implemented routinely. It should be taken into account that the increasing attention for iCBT is derived from different motives. Policymakers, health care insurance companies and mental health care organizations may promote iCBT to increase the cost-effectiveness, creating an unintentionally negative emphasis among therapists. Nevertheless, there is compelling evidence that iCBT can be effective in reducing depressive symptoms; it can improve treatment quality and eventually will support the therapist in doing their job more efficiently. In order to move to full implementation and sustainability of iCBT in routine care the following suggestions for future research are put forward: 1) The optimal versus personalized protocol use: in theory there is a high probability that iCBT and bCBT could be cost-effective [29,30]. However, in practice this has not yet been demonstrated [6,31].

One explanation is that therapists have a tendency to provide patients with online sessions on top of the face-to-face sessions, resulting in increased costs [31]. This was previously also identified by Kenter et al. [32]. Based on a-priori patient and therapists preferences [33] the common standard ratio of online and face-to-face sessions is set at 50-50%. For therapists in routine practice this ratio might not always be desirable. A recent study by Kemmeren et al. [34] showed a large variability in the usage of bCBT and that therapists and patients together created a more personalized blended care approach.

However, there can be a challenge in balancing personalization and therapist drift. Therefore, research could provide therapists with guidelines on what would facilitate protocol adaptations without losing efficacy, by measuring the extent to which therapists adhere to bCBT protocol and how well they implement the treatment. In addition, future research could focus on further crystallization of the combination between online and face-to-face sessions.

2) To improve the uptake in routine practice, we need more information from comparative trials between iCBT and CBT, not solely focusing on symptom reduction, but on other important factors such as therapist feedback, therapist time, treatment quality, CBT principles and treatment fidelity. These kinds of studies could help to improve and standardize the guidelines for providing iCBT and facilitate the uptake of iCBT. A recently developed scale, the iCBT Therapist Rating Scale (ICBT-TRS), could be further explored [35]. The Normalization MeAsure Development (NoMAD) questionnaire may be a

GENERAL DISCUSSION| 167 useful generic implementation instrument for measuring implementation processes in mental health care and the extent to which therapists have ‘normalized’ iCBT in their daily practice [36], although the responsiveness and predictive validity have yet to be proven.

3) To close the gap between research and practice, we need shorter cycles between research and practice in the form of repeated evaluations to inform research and organization on iCBT and bCBT implementation. This is only possible if we reorganize the way relevant routine treatment data is processed and collected. The way this is structured now, is not sufficiently valid to construct a scientific database. With online data we can open the ‘black box’ of CBT, but this requires better agreements around data from the different sources on data privacy, data ownership and technical set-up.

Therapists should be able to input relevant treatment into the systems easily, and others should be to able to extract research and organization data in a reliable way. Moreover, technological solutions (Ecological Momentary Assessments; EMA) can return the treatment data (e.g. continuous symptoms assessments, treatment progress) to therapists (e.g. crisis plan) and patients (e.g. patterns visualization), to provide them with beneficial information that could possibly positively influence treatment outcomes [37].

4) In addition to shorter research cycles, research should focus more on the maintenance/sustainability of iCBT which typically receives little attention [38]. But before looking at long-term sustainability, the next step in research is to move beyond the barriers and facilitators and to test tailored implementation strategies in comparison with other tailored strategies [39].

Implementation strategies are highly specified, theory-based methods that target known barriers to improve uptake at different levels [40]. We know that the most effective strategies are multifaceted, multilevel and adaptive [25]. Where training therapists is concerned, the training could consist of an adaptive multicomponent package (e.g. addressing technical and therapeutic issues, practice with fictional patients and video instructions, adapted to their skill set) with a follow-up training, also targeting their managers. Testing strategies for uptake would be a fruitful area for further research.

The European project, ImpleMentAll (www.implementall.eu) might provide routine practice with essential knowledge for Internet interventions in mental health care [41]. In ImpleMentAll, personalized implementation strategies will be tested to facilitate the use of internet-based and blended interventions in routine care in several different countries.

Closing remarks It is very likely that providing internet-based treatments (or other digital innovations) by therapists in mental health care will be normalized in the near future. However, promising results from iCBT effectiveness research are no guarantee for uptake in routine practice by therapists. This thesis underlines the difficulty in implementing an innovation in a complex context such as mental health care and discusses the lessons learned. Complex contexts require complex solutions that have yet to be further explored. One of the high potential solutions is centered around the therapist; to reach more patients with iCBT, the implementation of iCBT has to be led by therapists, and patients will follow. The studies in this thesis contribute to the growing body of literature on iCBT for depression and implementation in routine care; they raise questions for future researchers and recommend further steps in the widespread implementation of internet-based interventions.

168 | CHAPTER 7 Last but not least, because of the outbreak of the Coronavirus disease-2019 (COVID-19) many mental health care organizations in the Netherlands and around the world are currently closing their doors to patients who receive face-to-face therapy. To enable patients to continue treatment, therapists are now trying to replace their face-to-face sessions with internet interventions and videoconferencing. This means that therapists, ICT-staff and managers instantly have to overcome several barriers for uptake. The virus seems therefore an unforeseen but powerful catalyst for the implementation eMh in routine practice [42]. It would be interesting to see if this response influences therapists to normalize the use of iCBT when mental health care organizations reopen their doors to patients.

GENERAL DISCUSSION| 169

References [1]

[2]

[3]

[4]

[5]

[6]

[7]

[8] [9] [10]

[11]

[12]

[13]

Beidas, R.S. & Kendall, P.C. (2010). Training Therapists in Evidence-Based Practice: A Critical Review of Studies From a Systems-Contextual Perspective. Clinical Psychology: Science and Practice; 17 (1), 1–30.

Herschell, A.D., Kolko, D.J., Baumann, B.L., and Davis, A.C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review; 30 (4), 448–466.

Titzler, I., Saruhanjan, K., Berking, M., Riper, H., and Ebert, D.D. (2018). Barriers and facilitators for the implementation of blended psychotherapy for depression: A qualitative pilot study of therapists’ perspective. Internet Interventions; 12, 150–164.

Bruijniks, S.J.E., Franx, G., and Huibers, M.J.H. (2018). The implementation and adherence to evidence-based protocols for psychotherapy for depression: The perspective of therapists in Dutch specialized mental healthcare. BMC Psychiatry; 18 (1),.

Donovan, C.L., Poole, C., Boyes, N., Redgate, J., and March, S. (2015). Australian mental health worker attitudes towards cCBT: What is the role of knowledge? Are there differences? Can we change them? Internet Interventions; 2 (4), 372–381.

Kooistra, L.C., Wiersma, J.E., Ruwaard, J., Neijenhuijs, K., Lokkerbol, J., van Oppen, P., et al.

(2019). Cost and Effectiveness of Blended Versus Standard Cognitive Behavioral Therapy for Outpatients With Depression in Routine Specialized Mental Health Care: Pilot Randomized Controlled Trial. Journal of Medical Internet Research; 21 (10), e14261.

Bo, L., Lehr, D., Reis, D., Vis, C., Riper, H., Berking, M., et al. (2016). Reliability and validity of assessing user satisfaction with web-based health interventions. Journal of Medical Internet Research; 18 (8).

Lewis, J.R. (2018). The System Usability Scale: Past, Present, and Future. International Journal of Human-Computer Interaction; 34 (7), 577–590.

Sauro, J. and Lewis, J.R. (2011). When designing usability questionnaires, does it hurt to be positive? in: Conf. Hum. Factors Comput. Syst. - Proc.; 2215–2223.

Lewis, C.C., Fischer, S., Weiner, B.J., Stanick, C., Kim, M., and Martinez, R.G. (2015). Outcomes for implementation science: An enhanced systematic review of instruments using evidencebased rating criteria. Implementation Science; 10 (1).

Paxling, B., Lundgren, S., and Norman, A. (2013). Therapist behaviours in internet-delivered cognitive behaviour therapy: Analyses of E-mail correspondence in the treatment of generalized anxiety disorder. Behavioural and Cognitive Psychotherapy; 41(3), 280-289.

Schneider, L.H., Hadjistavropoulos, H.D., and Faller, Y.N. (2016). Internet-delivered cognitive behaviour therapy for depressive symptoms: an exploratory examination of therapist behaviours and their relationship to outcome and therapeutic alliance. Behavioural and Cognitive Psychotherapy; 44 (6), 625–639.

Holländare, F., Gustafsson, S.A., Berglind, M., Grape, F., Carlbring, P., Andersson, G., et al.

(2016). Therapist behaviours in internet-based cognitive behaviour therapy (ICBT) for depressive symptoms. Internet Interventions; 3 1–7.

170 | CHAPTER 7 [14]

[15]

[16]

[17] [18] [19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28]

Keijsers, G., Schaap, C., and Hoogduin, C. (2000). The Impact of Interpersonal Patient and Therapist Behaviour on Outcome in Cognitive-Behaviour Therapy. Behaviour Modification; 24 (2), 264–297.

Ewbank, M.P., Cummins, R., Tablan, V., Bateup, S., Catarino, A., Martin, A.J., et al. (2020).

Quantifying the Association between Psychotherapy Content and Clinical Outcomes Using Deep Learning. in: JAMA Psychiatry, American Medical Association; 35–43.

Glasgow, R.E., Vogt, T.M., and Boles, S.M. (1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health; 89 (9), 1322–1327.

Mora, L., Nevid, J., and Chaplin, W. (2008). Psychologist treatment recommendations for Internet-based therapeutic interventions. Computers in Human Behavior; 24 (6), 3052–3062.

Casey, L.M., Joy, A., and Clough, B.A. (2013). The impact of information on attitudes toward emental health services. Cyberpsychol Behav Soc Netw; 16 (8), 593–598.

Carper, M.M., McHugh, R.K., and Barlow, D.H. (2013). The dissemination of computer-based psychological treatment: a preliminary analysis of patient and clinician perceptions. Adm Policy Ment Health; 40 (2), 87–95.

Ward, R., Stevens, C., Brentnall, P., and Briddon, J. (2008). The attitudes of health care staff to information technology: A comprehensive review of the research literature. Health Information and Libraries Journal; 25 (2), 81–97.

Hennemann, S., Beutel, M.E., and Zwerenz, R. (2017). Ready for eHealth? Health Professionals’ Acceptance and Adoption of eHealth Interventions in Inpatient Routine Care. Journal of Health Communication; 22 (3), 274–284.

Ariens, L.F., Schussler-Raymakers, F.M., Frima, C., Flinterman, A., Hamminga, E., Arents, B.W., et al. (2017). Barriers and Facilitators to eHealth Use in Daily Practice: Perspectives of Patients and Professionals in Dermatology. Journal of Medical Internet Research; 19 (9), e300.

Ross, J., Stevenson, F., Lau, R., and Murray, E. (2016). Factors that influence the implementation of e-health: A systematic review of systematic reviews (an update). Implementation Science; 11 (1).

Lau, R., Stevenson, F., Ong, B.N., Dziedzic, K., Treweek, S., Eldridge, S., et al. (2016). Achieving change in primary care-causes of the evidence to practice gap: Systematic reviews of reviews.

Implementation Science; 11 (1), 1–39.

Powell, B.J., Fernandez, M.E., Williams, N.J., Aarons, G.A., Beidas, R.S., Lewis, C.C., et al. (2019).

Enhancing the impact of implementation strategies in healthcare: A research agenda. Frontiers in Public Health; 7.

Hadjistavropoulos, H.D., Thompson, M.J., Klein, B., and Austin, D.W. (2012). Dissemination of therapist-assisted internet cognitive behaviour therapy: development and open pilot study of a workshop. Cogn Behav Ther; 41 (3), 230–240.

Terpstra, J.A., van der Vaart, R., Spillekom-van Koulil, S., van Dam, A., Rosmalen, J.G., Knoop, H., et al. (2018). Becoming an eCoach: Training therapists in online cognitive-behavioral therapy for chronic pain. Patient Education and Counseling; 101, 1702–1707.

Kelders, S.M., Kok, R.N., Ossebaard, H.C., and Van Gemert-Pijnen, J.E.W.C. (2012). Persuasive system design does matter: A systematic review of adherence to web-based interventions.

Journal of Medical Internet Research; 14 (6), e152.

GENERAL DISCUSSION| 171 [29]

[30]

[31]

[32]

[33]

[34]

[35]

[36]

[37]

[38]

[39]

[40]

Donker, T., Blankers, M., Hedman, E., Ljotsson, B., Petrie, K., and Christensen, H. (2015).

Economic evaluations of Internet interventions for mental health: A systematic review.

Psychological Medicine; 45 (16), 3357–3376.

Hedman, E., Ljótsson, B., and Lindefors, N. (2012). Cognitive behavior therapy via the Internet: a systematic review of applications, clinical efficacy and cost-effectiveness. Expert Review of Pharmacoeconomics & Outcomes Research; 12 (6), 745–64.

Kolovos, S., van Dongen, J.M., Riper, H., Buntrock, C., Cuijpers, P., Ebert, D.D., et al. (2018). Cost effectiveness of guided Internet-based interventions for depression in comparison with control conditions: An individual–participant data meta-analysis. Depression and Anxiety; 35 (3), 209– 219.

Kenter, R.M.F., van de Ven, P.M., Cuijpers, P., Koole, G., Niamat, S., Gerrits, R.S., et al. (2015).

Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: Results from a naturalistic study. Internet Interventions; 2 (1), 77–83.

Van der Vaart, R., Witting, M., Riper, H., Kooistra, L., Bohlmeijer, E.T., and van Gemert-Pijnen, L.J. (2014). Blending online therapy into regular face-to-face therapy for depression: content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry; 14, 355.

Kemmeren, L.L., van Schaik, A., Smit, J.H., Ruwaard, J., Rocha, A., Henriques, M., et al. (2019).

Unraveling the Black Box: Exploring Usage Patterns of a Blended Treatment for Depression in a Multicenter Study. JMIR Mental Health; 6 (7), e12707.

Hadjistavropoulos, H.D., Schneider, L.H., Klassen, K., Dear, B.F., and Titov, N. (2018).

Development and evaluation of a scale assessing therapist fidelity to guidelines for delivering therapist-assisted Internet-delivered cognitive behaviour therapy. Cognitive Behaviour Therapy; 47 (6), 447–461.

Vis, C., Ruwaard, J., Finch, T., Rapley, T., de Beurs, D., van Stel, H., et al. (2019). Toward an Objective Assessment of Implementation Processes for Innovations in Health Care: Psychometric Evaluation of the Normalization Measure Development (NoMAD) Questionnaire Among Mental Health Care Professionals. Journal of Medical Internet Research; 21 (2), e12376.

Colombo, D., Fernández-Álvarez, J., Patané, A., Semonella, M., Kwiatkowska, M., GarcíaPalacios, A., et al. (2019). Current State and Future Directions of Technology-Based Ecological Momentary Assessment and Intervention for Major Depressive Disorder: A Systematic Review.

Journal of Clinical Medicine; 8 (4), 465.

Proctor, E., Luke, D., Calhoun, A., McMillen, C., Brownson, R., McCrary, S., et al. (2015).

Sustainability of evidence-based healthcare: Research agenda, methodological advances, and infrastructure support. Implementation Science; 10 (1), 1–13.

Baker, R., Camosso-Stefinovic, J., Gillies, C., Shaw, E.J., Cheater, F., Flottorp, S., et al. (2015).

Tailored interventions to address determinants of practice. Cochrane Database of Systematic Reviews; 4.

Powell, B.J., Waltz, T.J., Chinman, M.J., Damschroder, L.J., Smith, J.L., Matthieu, M.M., et al.

(2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science; 10 (1).

172 | CHAPTER 7 [41]

[42]

Bührmann, L., Schuurmans, J., Ruwaard, J. et al. Tailored Implementation of internet-based Cognitive Behavioural Therapy in the Multinational Context of the ImpleMentAll Project: A Study Protocol for a Stepped Wedge Cluster Randomized Trial. Submitted for Publication.

Wind, T.R., Rijkeboer, M., Andersson, G., and Riper, H. (2020). The COVID-19 pandemic: The ‘black swan’ for mental health care and a turning point for e-health. Internet Interventions; 100317.

173

174

SAMENVATTING| 175

SAMENVATTING Eerder onderzoek heeft aangetoond dat online cognitieve gedragstherapie (iCGT) (zie Box 1) veelbelovend is bij de behandeling van depressie (zie Box 2), maar in de dagelijkse praktijk wordt deze vorm van eMental health (eMh) nog relatief weinig toegepast. Het MasterMind-project had het doel om het gebruik van iCGT voor depressie in vijftien Europese regio's te volgen en op te schalen om de kloof tussen onderzoek en praktijk te dichten (www.mastermind-project.eu). Een succesvolle implementatie van interventies in de geestelijke gezondheidszorg kan worden beïnvloed door vele factoren op meerdere niveaus, zoals patiënt-, therapeut- en organisatieniveau. Het centrale doel van dit proefschrift was om meer inzicht te krijgen in het implementeren van iCGT voor depressie op het niveau van de therapeut in de dagelijkse praktijk.

Box 1: Wat is depressie?

Depressie, ofwel een depressieve stoornis, is een veelvoorkomende psychiatrische stoornis. Volgens het classificatiesysteem van de DSM-5, the Diagnostic and Statistical Manual of Mental Disorders, moet aan een aantal criteria worden voldaan om een depressieve stoornis vast te stellen: een sombere stemming en/of verlies van plezier in alle of meeste activiteiten, gedurende de hele dag voor tenminste twee weken achter elkaar en dit leidt tot aanzienlijke beperkingen in het functioneren op verschillende domeinen. Daarnaast kunnen ook nog andere symptomen aanwezig zijn, zoals aanzienlijk gewichtsverlies of gewichtstoename, slapeloosheid of veelvuldig slapen, vermoeidheid of verlies van energie, verminderd vermogen om te concentreren of beslissingen te nemen, gevoelens van waardeloosheid of buitensporige schuld, en terugkerende gedachten over de dood of gedachten over zelfmoord.

Box 2: Wat is iCGT? iCGT is een gedigitaliseerde vorm van de cognitieve gedragstherapie. CGT leent zich goed voor een digitaal format, vanwege de gestructureerde aanpak en psycho-educatiecomponent, evenals huiswerkopdrachten en registratie-oefeningen die online kunnen worden aangeboden. Hoewel er verschillende iCGT formats worden aangeboden door verschillende providers, is het algemene idee achter iCGT om patienten een online platform aan te bieden waarop ze 24/7 toegang hebben tot (depressie) modules. Patiënten kunnen online sessies doorlopen in hun eigen tempo en binnen de privacy van hun eigen huis. Met internettoegang in 89% van de huishoudens in Europa en 98% in Nederland, is het potentiële bereik van iCBT aanzienlijk en wordt de toegang tot een depressiebehandeling hiermee uitgebreid. iCBT heeft de potentie om de druk op de huidige geestelijke gezondsheidszorg te verlagen door de hoeveelheid therapeutentijd per patient te verminderen. Bovendien wordt aangenomen dat iCBT gemakkelijk kan worden voorgeschreven, aangeboden met consistent hoge kwaliteit en lage vaste kosten voor patiënten. Over het algemeen kan iCBT worden onderverdeeld in drie formats: er zijn onbegeleide, begeleide en meer recentelijk ook blended formats ontwikkeld. In dit proefschrift wordt ingegaan op de implementatie van begeleide en blended iCBT, waarbij de patienten online contact hebben met hun therapeut. Patiënten en therapeuten communiceren meestal a-synchroon via beveiligde digitale schriftelijke communicatie, waarbij therapeuten bijvoorbeeld geschreven feedback geven over de huiswerkopdrachten van de patiënten. De communicatie kan ook synchroon verlopen, zoals in het geval van ‘chat’ ondersteuning. Bovendien bestaat er voor therapeuten vaak de mogelijkheid om online de patiënt te volgen via symptoomvragenlijsten en dagboekaantekeningen. Het type begeleiding binnen iCGT varieert per instelling en is afhankelijk van de ernst van de symptomen en kan variëren van 5 minuten wekelijkse telefonische ondersteuning tot uitgebreide tweewekelijkse asynchrone e-mailondersteuning. Bij blended CGT worden online en face-to-face sessies met elkaar gecombineerd in een geprotocoleerde behandeling.

176 | SAMENVATTING Voordat we ons concentreerden op factoren op therapeutniveau, zijn we begonnen we een breed perspectief op de belemmerende en bevorderende factoren bij het implementeren van eMh voor depressie in de dagelijkse praktijk. We hebben systematisch de literatuur onderzocht op het niveau van patiënten, therapeuten, organisatie en gezondheidszorg om de implementatiekloof tussen onderzoek en praktijk te adresseren (hoofdstuk 2). In totaal hebben we 13.147 artikelen gescreend, waarvan 48 studies in de review zijn opgenomen. We hebben 37 determinanten geïdentificeerd door thematische analyse, waarvan drie het vaakst werden gerapporteerd: 1) verwachtingen en voorkeuren van patiënten en therapeuten met betrekking tot het ontvangen en aanbieden van eMh in de dagelijkse praktijk, 2) de geschiktheid van eMh voor het verminderen van klachten bij de patiënten, en 3) de beschikbaarheid, betrouwbaarheid en interoperabiliteit van de eMh toepassingen met andere bestaande technologieën zoals digitale patiëntendossiersystemen. Dit waren belangrijke factoren voor therapeuten om te overwegen iCBT te gaan gebruiken of te blijven gebruiken in de routine depressie zorg. Deze review maakte ons ervan bewust dat het merendeel van de studies beschrijvend van aard was, gericht op de patiënt en op therapeutenniveau. Gezien de huidige status van dit opkomende onderzoeksgebied was dit niet onverwacht.

Met betrekking tot implementatiefactoren die van invloed kunnen zijn op de therapeuten, hebben we geleerd dat de meeste onderzoeksresultaten gebaseerd waren op verwachtingen en voorkeuren van de therapeuten. De meeste therapeuten hadden beperkte of geen daadwerkelijke ervaringen met iCGT voor depressie in de dagelijkse praktijk. Kwalitatief onderzoek met therapeuten die al ervaring hebben met eMh kan meer diepgaand inzicht geven in de onderliggende redenen voor de lage acceptatie van iCBT in de reguliere praktijk. We hebben dit kwalitatieve onderzoek onder therapeuten uitgevoerd in het kader van de Mastermind-studie. In hoofdstuk 3 en 4 zijn het studieprotocol en de resultaten van dit deel van de MasterMind-studie in Nederland beschreven. Therapeuten die met een blended format van iCBT (bCBT) hadden gewerkt (waarbij online en face-to-face gesprekken worden gecombineerd), vonden de bruikbaarheid voldoende en waren over het algemeen tevreden met het geven van bCBT aan hun patiënten. Met een thematische analyse vonden we drie hoofdthema's met de belemmerende en bevorderende factoren: 1) de behoeften van therapeuten om bCBT te gaan gebruiken (bijv. een doorlopende training met focus op therapeutische inhoud), 2) de rol van therapeuten in het motiveren van patiënten voor bCBT (bijv. onzekerheid over de geschiktheid van de patiënten) en 3) ervaringen van therapeuten met bCBT (bijv. de structuur helpt het beperken van de therapeuten drift). Over het algemeen stemden therapeuten ermee in dat bCBT een goed format is voor patiënten die ze in de dagelijkse praktijk behandelen; ze gingen ervan uit dat bCBT door de meeste reguliere therapeuten kan worden aangeboden en ze verwachten ook dat het gebruik in de toekomst zal groeien. Vooral de structuur van bCBT bleek gunstig in de praktijk. Desondanks hebben therapeuten geen beloofde tijdsbesparing ervaren in hun dagelijkse praktijk; het kostte hen zelfs meer tijd.

Naast kwalitatieve interviews hebben we de System Usability Scale (SUS) vragenlijst gebruikt om de gebruiksvriendelijkheid van iCBT te meten. De SUS is een populaire vragenlijst die vaak gebruikt is in eMh onderzoek. Dit instrument is echter alleen gevalideerd om de bruikbaarheid in industriële contexten te meten, nog niet in de context van eMh. In hoofdstuk 5 hebben we de psychometrische eigenschappen, zoals de factorstructuur, van de SUS onderzocht in een steekproef van professionals die iCBT en bCBT voor depressie gebruikten uit zes Europese landen binnen het MasterMind-project.

De betrouwbaarheid werd gemeten met behulp van aanvullende statistische indices (onder andere omega). De convergente validiteit van de SUS werd getest met drie vragen over de tevredenheid (Client

SAMENVATTING| 177 Satisfaction Questionnaire-3). Op basis van een confirmatieve factoranalyse concludeerden we dat hoewel de SUS een multidimensionale structuur lijkt te hebben, het beste model aantoonde dat de totale somscore van de SUS een geldige, betrouwbare en interpreteerbare maat is om de gebruiksvriendelijkheid van iCGT te beoordelen, wanneer gebruikt door professionals in de dagelijkse geestelijke gezondheidszorg.

In hoofdstuk 6 hebben we ingezoomd op het gedragingen van de therapeuten in de online berichten die ze naar hun patiënten stuurden in bCGT. Het online platform biedt een unieke mogelijkheid om informatie te verzamelen door de berichten van het online platform te extraheren en hun inhoud in detail te bestuderen. Feedback van therapeuten is een substantieel onderdeel van de online therapie, maar het is onbekend hoe de feedback wordt gegeven en hoe therapeuten gebruik maken van de instructies voor de online feedback. We wilden weten of de gedragingen van therapeuten en het naleven van de instructies verband hielden met de uitkomsten van de patiënten (het aantal voltooide online sessies en de ernst van de depressiesymptomen). In totaal stuurden 19 therapeuten 219 feedbackberichten naar 45 patiënten. Het meest gebruikte gedragingen van de therapeuten waren informatief, aanmoedigend en bevestigend. Deze waren niet gerelateerd aan de uitkomsten van de patiënten. Hoewel het weinig werd gebruikt, was het confronteren (bijv. de therapeut sprak een andere mening uit of het niet eens met wat de patiënt schreef) positief gecorreleerd met het aantal voltooide sessies. Verder hielden de therapeuten zich aan de meeste feedbackinstructies (bijv. de feedback beginnen met het geven van een compliment). Slechts twee feedbackaspecten waren gecorreleerd met het aantal voltooide sessies: hoe meer therapeuten zich hielden aan instructies met betrekking tot structuur (beperken tot 2 onderwerpen en de feedback versturen binnen 3 werkdagen) en leesbaarheid (gebruik van korte zinnen en korte alinea's), hoe minder online sessies werden voltooid. We hebben geen associaties gevonden met scores voor depressiesymptomen. De bevindingen suggereerden dat feedbackinstructies voor therapeuten voldoende begeleiding gaven om op een ondersteunende en positieve manier met patiënten online te communiceren. We denken echter dat de instructies kunnen worden verbeterd door meer specifieke therapeutische technieken toe te voegen naast alleen de focus op stijl en vorm.

Het is zeer waarschijnlijk dat het aanbieden van iCGT (of andere digitale innovaties) door therapeuten in de geestelijke gezondheidszorg in de nabije toekomst zal worden genormaliseerd.

Veelbelovende resultaten van iCBT-effectiviteitsonderzoek zijn echter geen garantie voor het gebruik door therapeuten in de dagelijkse praktijk. Dit proefschrift onderstreept de moeilijkheid om een innovatie in een complexe context als geestelijke gezondheidszorg te implementeren en bespreekt de geleerde lessen. Complexe contexten vereisen complexe oplossingen die nog verder moeten worden onderzocht. Een van de oplossingen met hoog potentieel is gecentreerd rond de therapeut; om meer patiënten met iCBT te bereiken, moet de implementatie van iCBT via therapeuten verlopen en zullen patiënten volgen. De studies in dit proefschrift dragen bij aan de groeiende hoeveelheid literatuur over iCBT voor depressie en implementatie in routinematige zorg, roepen vragen op voor toekomstige onderzoekers en raden verdere stappen aan in wijdverbreide implementatie van online interventies.

178 | DANKWOORD

DANKWOORD| 179

DANKWOORD Middels deze woorden wil ik graag iedereen bedanken die betrokken is bij mijn promotie, direct of zijdelings. Dankzij jullie heb ik met heel veel plezier aan dit project gewerkt.

Bij de start stapte ik niet alleen in een rijdende trein, het voelde ook nog eens als warm bad. Heleen en Jan. Ik heb ontzettend veel gehad aan jullie kritische en methodologische steun. Ik heb veel geleerd van jullie feedback. Voor jullie betrokkenheid en inzet voor het werk heb ik veel bewondering. Heleen, met jouw visie op technologische innovatie in de zorg en jouw ‘out of the box’ manier van denken, ben jij voor mij een grote bron van inspiratie op vele vlakken. Jan, jij hebt een geheugen om jaloers op te zijn. Hoe jij het overzicht houdt is voor mij nog altijd een raadsel. Privé of werk gerelateerd, ik kon altijd bij je binnen lopen. Bovendien bezit je ook nog eens de gave om problemen, klein of groot, binnen een dag op te lossen.

Het andere duo dat mijn werk naar een hoger niveau heeft gebracht: Anneke en Els, Els en Anneke.

Ik heb ontzettend veel van jullie geleerd; van jullie visie op onderzoek en jullie blik op de praktijk vanuit verschillende en complementerende onderzoeksbrillen en behandelaarspetten. Onze wekelijkse overleggen vond ik altijd iets om naar uit te kijken. Anneke, ik vind het bewonderingswaardig hoe jij alle ballen in de lucht weet te houden; hoe jij je werk structureert, nuchter, weloverwogen en alles met chirurgische precisie aanpakt en opruimt. Ik probeer nu voortaan mijn teksten met de Anneke blik in mijn achterhoofd te schrijven. Els, wie kent jou niet? Ik ben heel blij dat ook ik jou heb leren kennen.

Jouw aanpak is heel erg betrokken en super pragmatisch, no-nonsense, laten we het gewoon doen, niet morgen maar vandaag. Het was absoluut een voorrecht om jou als begeleider te mogen hebben.

Christiaan, voor een groot deel ben jij verantwoordelijk geweest voor het opstarten van de trein en voor het scannen van 13,147 studies in de zomer van 2015. Dank je wel voor al jouw netwerkvaardigheden, management skills en dat ik hierop kon meeliften.

Lieve Lise en Lisa, oftewel de harde kern van de ‘eHealthmiepjes’. Mijn prachtige slimme paranimfen. Dit traject was zonder jullie ondenkbaar. Zo fijn om alles met elkaar te kunnen delen, van data tot drankjes. Lieve Renske, collega en meer (alwetend pragmatisch wikiwonder, EngelsNederlands woordenboek, en vriendin). We zijn tegelijkertijd begonnen, zwanger, en nu beiden in de afrondende fase. Ik zal jou als wandel- en kamergenoot ontzettend missen.

Mijn andere kamer- en ganggenoten Anna, Carisha, Carolien, Gerard, Mandy, Melany, Merijn, Richard, Wouter, de dames uit de M-vleugel: Adrie, Claire, Esther, Josien, Laura, Rosa, Ruth, Trees en alle andere promovendi en post-docs bij GGZ inGeest en de VU wil ik ook heel erg bedanken voor alle gezelligheid, babypraat en natuurlijk onderzoek-gerelateerde ondersteuning in de afgelopen jaren.

Hierbij mag ook zeker de gezelligheid en ondersteuning van Carla, Dana, Jolanda, Margie, Rianne en Susan niet ontbreken. Lieve Rianne, ook al was het soms een godswonder, als spin in het wiel kreeg jij het toch altijd weer voor elkaar om de agenda’s te managen. Mijn dank is groot.

Alle (ex)datamanagement en veldwerk mensen wil ik ook graag bedanken voor alle nodige hulp, waaronder met name Bep, Bianca, Denise, Ho Ming, Nanda (Holmes), Tim en Thuur. Lieve Stasja, mijn moderator en bij tijden redder in nood, dank voor alles! Lieve Jeroen, dank je wel dat ik even gebruik mocht maken van jouw brein en R-skills. Het is een groot gemis voor velen dat je er niet meer bent.

Lieve Isabella, dankzij jouw inzet en doorzettingsvermogen geloof ik er heilig in dat iedereen bij GGZ inGeest straks aan eHealth doet.

180 | DANKWOORD Met regelmaat denk ik terug aan alle mooie plekken die ik binnen dit traject heb mogen bezoeken voor congressen of MasterMind consortium meetings: Auckland (de wijntoer op elektrische fietsen op Waiheke Island met Esther), Barcelona (de frisse duik in het hotelzwembad op het dak met Els), Berlijn, Brussel, Hamburg (magische cocktails in de hotelbar met Anne), Odense (de toerist uithangen in Kopenhagen met Anneke), Stockholm (het Abba museum met Rosa en Patricia), Turijn en Warschau.

De leden van de beoordelingscommissie, prof.dr. A.W.M. Evers, prof.dr. V. Kaldo, prof.dr. J.E.W.C. van Gemert-Pijnen, Dr. N.M. Batelaan, Dr. A.M. Kleiboer onder voorzitterschap van prof.dr. P. van Oppen, wil ik graag bedanken voor het beoordelen van dit proefschrift.

Natuurlijk mogen ook alle GGZ instellingen, projectleiders, behandelaren en hun patiĂŤnten niet ontbreken. Dank jullie wel voor jullie tijd en betrokkenheid bij het onderzoek.

Ernst, dank u wel dat ik mocht blijven plakken na mijn stage. Ik denk nog vaak terug aan onze gesprekken en uw (soms wat exentrieke) kijk op het leven. Lieve Janine, dank je wel voor alle kansen die jij mij hebt geboden. Jij hebt mij warm gemaakt voor (kwalitatief) onderzoek. Het contact dat wij hebben is mij heel erg dierbaar.

Lieve familie en vrienden, dank jullie wel dat jullie er onvoorwaardelijk voor mij zijn. De laatste woorden zijn voor Sjoerd en Ben. Sjoerd, jij maakt mij een beter persoon. Jouw positieve instelling en jouw liefde voor gadgets zijn besmettelijk. Ja, het is clichĂŠ, Ben. Ik ken jou het kortst, maar jij maakt het leven van mij en Sjoerd nog rijker, waardevoller en mooier dan het al is.

CURRICULUM VITAE| 181

CURRICULUM VITAE 1985

Geboren, Naarden

1997-2004

Gymnasium Gemeentelijk Gymnasium, Hilversum

2004-2011

Bachelor Klinische Ontwikkelingspsychologie Universiteit van Amsterdam

2008-2011

Masteropleiding Klinische Ontwikkelingspsychologie Universiteit van Amsterdam

2008-2014

Basispsycholoog, Dyslexie- en beroepskeuzecentrum Ergoselect, Amsterdam

2010-2015

Onderzoeker, Sociaalwetenschappelijk onderzoeksbureau Impact R&D, Amsterdam

2015-2020

Promovendus, GGZ inGeest, Amsterdam

2020

Psycholoog in opleiding, GGZ inGeest, Amsterdam

182 | PUBLICATIONS AND PRESENTATIONS

PUBLICATIONS AND PRESENTATIONS| 183

PUBLICATIONS AND PRESENTATIONS 2020

2020

2019

2018

2018

2016

2015

2016

2015 2014

SCIENTIFIC PUBLICATIONS Mol, M., van Schaik D.J. , Dozeman, E., J. Ruwaard, C. Vis, DD. Ebert, A. Etzelmueller, K. Mathiasen, B. Moles, T. Mora, CD. Pedersen, MM Skjøth, L. Peleteiro Pensado, J. PieraJimenez, D. Gokcay, B. Ünlü Ince, A. Russi, Y. Sacco, E. Zanalda, H. Riper, JH. Smit.

Dimensionality of the System Usability Scale among professionals using internet-based interventions for depression: a confirmatory factor analysis. BMC Psychiatry, 20, 218.

Mol, M. C. van Genugten, E. Dozeman, A. van Schaik, S. Draisma, H. Riper, Smit, J. Why uptake of blended internet-based interventions is challenging: a qualitative study on therapist perspective. Journal of Clinical Medicine, 9(1), 91.

Vis, C., Ruwaard, J., Finch, T., Rapley, T., Mol, M., Kleiboer, A., ... & Smit, J. Toward an Objective Assessment of Implementation Processes for Innovations in Health Care: Psychometric Evaluation of the Normalization Measure Development (NoMAD) Questionnaire Among Mental Health Care Professionals. Journal of medical Internet research, 21(2), e12376-e12376.

Mol, M., Dozeman, E., Provoost, S., van Schaik, A., Riper, H., & Smit, J. H. Behind the Scenes of Online Therapeutic Feedback in Blended Therapy for Depression: Mixed-Methods Observational Study. Journal of medical Internet research, 20(5).

Vis, C., Mol, M., Kleiboer, A., Bührmann, L., Finch, T., Smit, J., & Riper, H. Improving Implementation of eMental Health for Mood Disorders in Routine Practice: Systematic Review of Barriers and Facilitating Factors. JMIR mental health, 5(1).

Mol, M., Dozeman, E., van Schaik, D. J., Vis, C. P., Riper, H., & Smit, J. H. The therapist’s role in the implementation of internet-based cognitive behavioural therapy for patients with depression: study protocol. BMC psychiatry, 16(1), 338.

Vis, C., Kleiboer, A., Prior, R., Bønes, E., Cavallo, M., Clark, S. A., Dozeman, E., Ebert, D., Etzelmueller, A., Favaretto, G., & Zabala, A. F., Kolstrup, N., Mancin, S., Mathiassen, K., Myrbakk, V. N., Mol, M., Jimenez, J. P., Power, K., van Schaik, A., Wright, C., Zanalda, E., Pederson, C. D., Smit, J., Riper, H. Implementing and up-scaling evidence-based eMental health in Europe: the study protocol for the MasterMind project. Internet Interventions, 2(4), 399-409.

NON-SCIENTIFIC PUBLICATIONS Plaisier, J., Mol, M. Gural, J., Baalman, J., Hoetjes, V., Lenssen, D., & Pollaert, H. Ebegeleiding. Evaluatie van een pilot gericht op bruikbaarheid en uitvoerbaarheid van deze nieuwe methode voor gedragsverandering. Amsterdam: Impact R&D. Dozeman, E., Mol, M., & van Schaik, D. J. F. E-Mental health: blended behandeling voor patiënten met een depressie. Nurse academy GGZ, 4, 41-46.

Plaisier, J., & Mol, M. Waarom de gedragsbeïnvloedende maatregel weinig wordt opgelegd. In opdracht van de directie Justitieel Jeugdbeleid van het ministerie van Veiligheid en Justitie.

184 | PUBLICATIONS AND PRESENTATIONS 2014

2013

2013

2013

2012 2012

2012

2011

2019 2019

2018 2018

2018 2017 2017

Pröpper, I., Plaisier, J., Mol, M., & Rouw, M. De maatregel ‘Inrichting Stelselmatige Daders’ voor jongvolwassen veelplegers: de uitvoeringspraktijk en lessen voor de toekomst.

Amsterdam: Impact R&D. In opdracht van het ministerie van Veiligheid & Justitie, in samenwerking met Partners+Pröpper.

Mol, M., Hoetjes, V., & Plaisier, J. De training Kies voor Verandering. Eerste indrukken van de ervaringen van gedetineerden en personeel in zes gevangenissen. Amsterdam: Impact R&D. In opdracht van de Dienst Justitiële Inrichtingen.

Nelen, H., Mol, M., Plaisier, J., & Peters, M. Pilot Dreigingsmanagement. De implementatie en wijze van uitvoering onder de loep. Amsterdam: Impact R&D. In opdracht van het ministerie van Veiligheid & Justitie, in samenwerking met Universiteit Maastricht.

Plaisier, J., & Mol, M. Procesevaluatie van de werkwijze voor nazorg van jongeren die uit een Justitiële Jeugdinrichting komen. Amsterdam: Impact R&D. In opdracht van het ministerie van Veiligheid & Justitie.

Mol, M., & Plaisier, J. De aanpak van criminele jeugdgroepen. Quickscan internationale onderzoeksliteratuur. Amsterdam: Impact R&D. Plaisier, J., Ditzhuijzen, J. van, Mol, M., & Giessen, M. van der. De mogelijke invloed van de eigen bijdrage op de instroom van patiënten in de forensische zorg en het gevangeniswezen. Amsterdam: Impact R&D. In opdracht van de Penitentiair Psychiatrische Centra van de Dienst Justitiële Inrichtingen, in samenwerking met GGZ Nederland.

Plaisier, J., Wiersema, D. V., Mol, M., & Giessen, M. van der. Psychiatrische zorg in detentie: een literatuuroverzicht. Den Haag/Amsterdam: Dienst Justitiële Inrichtingen & Impact R&D. Ditzhuijzen, J. van, Mol, M., & J. Plaisier. Trainershandleiding en wetenschappelijke onderbouwing van de Fysiek-Mentale Weerbaarheidstraining. Amsterdam: Impact R&D. PRESENTATIONS Minor Geneeskunde (VUmc), Amsterdam: college ‘Implementation of eHealth’.

Congres International Society for Research on Internet Interventions (ISRII), Auckland: presentatie ‘Behind the scenes of online therapeutic feedback in blended therapy for depression’.

Minor Geneeskunde (VUmc), Amsterdam: college ‘Implementation of eHealth’.

Conferentie Nederlands Kenniscentrum Angst en Depressie (NedKad), Amersfoort: posterpresentatie ‘Behind the scenes of online therapeutic feedback in blended therapy for depression’.

VUmc Science Exchange day, Amsterdam: posterpresentatie ‘Behind the scenes of online therapeutic feedback in blended therapy for depression’.

Annual meeting Amsterdam Public Health (APH), Amsterdam: presentatie ‘Implementing and up-scaling eMental health in Europe: perspectives of the professionals’.

Conferentie Vereniging voor Gedrags- en Cognitieve therapieën (VGCt), Veldhoven: presentatie ‘De rol van de behandelaar bij implementatie van eHealth voor depressie’.

PUBLICATIONS AND PRESENTATIONS| 185 2017

2017 2017 2016 2016

2016

2016

2015

Congres International Society for Research on Internet Interventions (ISRII), Berlijn: posterpresentatie ‘Exploring online therapist feedback in blended treatment for depression in routine mental health care: an update’; presentatie ‘Blended care and the Professionals’. eHealth bijeenkomst GGZ inGeest, Amsterdam: presentatie ‘Het geven van online feedback binnen de blended depressie behandeling’.

Congres Innovating Health care, Amsterdam: workshop ‘eHealth in routine mental health care: lessons learnt and future challenges’.

Congres Vereniging voor Gedrags- en Cognitieve therapieën (VGCt), Veldhoven: presentatie ‘De rol van de behandelaar bij implementatie van eHealth voor depressie’.

Conferentie Nederlands Kenniscentrum Angst en Depressie (NedKad), Amersfoort: posterpresentatie ‘Evaluation of the implementation of eHealth solutions for depression in mental health care in the Netherlands’.

Congres European Association for Behavioural and Cognitive Therapies (EABCT), Stockholm: presentatie ‘Implementation of internet-based cognitive behavioural therapy for patients with depression in routine care- MasterMind study’.

Congres International Society for Bipolar Disorders/International Society for Affective Disorders (ISDB/ISAD), Amsterdam: presentation ‘Evaluation of a blended cognitive therapy for patients with depression in multiple Dutch health care settings’.

Congres European Society for Research on Internet Interventions (ESRII), Warschau: poster presentatie ‘Evaluation of the implementation of e-health solutions for depression in mental health care in the Netherlands’.

186 | DISSERTATION SERIES

DISSERTATION SERIES| 187

DISSERTATION SERIES Department of Psychiatry, Amsterdam University Medical Centers N.M. (Neeltje) Batelaan (2010). Panic and Public Health: Diagnosis, Prognosis and Consequences. Vrije Universiteit Amsterdam. ISBN: 978-90-8659-411-5.

G.E. (Gideon) Anholt (2010). Obsessive-Compulsive Disorder: Spectrum Theory and Issues in Measurement. Vrije Universiteit Amsterdam.

N. (Nicole) Vogelzangs (2010). Depression & Metabolic Syndrome. Vrije Universiteit Amsterdam. ISBN: 978-90-8659-447-4.

C.M.M. (Carmilla) Licht (2010). Autonomic Nervous System Functioning in Major Depression and Anxiety Disorders. Vrije Universiteit Amsterdam. ISBN: 978-90-8659-487-0.

S.A. (Sophie) Vreeburg (2010). Hypothalamic-Pituitary-Adrenal Axis Activity in Depressive and Anxiety Disorders. Vrije Universiteit Amsterdam. ISBN: 978-90-8659-491-7.

S.N.T.M. (Sigfried) Schouws (2011). Cognitive Impairment in Older Persons with Bipolar Disorder. Vrije Universiteit Amsterdam. ISBN: 978-90-9025-904-8.

P.L. (Peter) Remijnse (2011). Cognitive Flexibility in Obsessive-Compulsive Disorder and Major Depression â&#x20AC;&#x201C; Functional Neuroimaging Studies on Reversal Learning and Task Switching. Vrije Universiteit Amsterdam. ISBN: 978-90-6464-449-8.

S.P. (Saskia) Wolfensberger (2011). Functional, Structural, and Molecular Imaging of the Risk for Anxiety and Depression. Vrije Universiteit Amsterdam. ISBN: 978-90-8659-536-5.

J.E. (Jenneke) Wiersma (2011). Psychological Characteristics and Treatment of Chronic Depression.

Vrije Universiteit Amsterdam. ISBN: 978-94-9121-150-8.

P.D. (Paul David) Meesters (2011). Schizophrenia in Later Life. Studies on Prevalence, Phenomenology and Care Needs (SOUL Study). Vrije Universiteit Amsterdam. ISBN: 978-90-8659-563-1.

R. (Ritsaert) Lieverse (2011). Chronobiopsychosocial Perspectives of Old Age Major Depression: a Randomized Placebo Controlled Trial with Bright Light. Vrije Universiteit Amsterdam. ISBN: 978-908570-858-2.

A. (Adrie) Seldenrijk (2011). Depression, Anxiety and Subclinical Cardiovascular Disease. Vrije Universiteit Amsterdam. ISBN: 978-94-6191-052-3.

188 | DISSERTATION SERIES Y. (Yuri) Milaneschi (2012). Biological Aspects of Late-life Depression. Vrije Universiteit Amsterdam.

ISBN: 978-90-8659-608-9.

L. (Lynn) Boschloo (2012). The Co-occurrence of Depression and Anxiety with Alcohol Use Disorders.

Vrije Universiteit Amsterdam. ISBN: 978-94-6191-327-2.

D. (Didi) Rhebergen (2012). Insight into the heterogeneity of depressive disorders. Vrije Universiteit Amsterdam. ISBN: 978-94-6191-387-6.

T.M. (Michiel) van den Boogaard (2012). The Negotiated Approach in the Treatment of Depressive Disorders: the impact on patient-treatment compatibility and outcome.

Vrije Universiteit Amsterdam. ISBN: 978-90-8891-495-9.

M. (Marjon) Nadort (2012). The implementation of outpatient schema therapy for borderline personality disorder in regular mental healthcare. Vrije Universiteit Amsterdam. ISBN: 978-94-6191463-7.

U. (Ursula) Klumpers (2013). Neuroreceptor imaging of mood disorder related systems. Vrije Universiteit Amsterdam. ISBN: 978-94-6191-575-7.

E. (Ethy) Dorrepaal (2013). Before and beyond. Stabilizing Group treatment for Complex posttraumatic stress disorder related to child abuse based on psycho-education and cognitive behavioral therapy.

Vrije Universiteit Amsterdam. ISBN: 978-94-6191-601-3.

K. (Kathleen) Thomaes (2013). Child abuse and recovery. Brain structure and function in child abuse related complex posttraumatic stress disorder and effects of treatment. Vrije Universiteit Amsterdam.

ISBN: 978-94-6191-600-6.

K.M.L.(Klaas) Huijbregts (2013). Effectiveness and cost-effectiveness of the implementation of a collaborative care model for depressive patients in primary care. Vrije Universiteit Amsterdam. ISBN: 978-90-9027404-1.

T.O. (Tessa) van den Beukel (2013). Ethnic differences in survival on dialysis in Europe. The role of demographic, clinical and psychosocial factors. Vrije Universiteit Amsterdam. ISBN: 978-94-6108410-1.

A. (Agnes) Schrier (2013). Depression and anxiety in migrants in the Netherlands. Population studies on diagnosis and risk factors. Vrije Universiteit Amsterdam. ISBN: 978-94-6191-719-5.

B. (Barbara) Stringer (2013). Collaborative Care for patients with severe personality disorders.

Challenges for the nursing profession. Vrije Universiteit Amsterdam. ISBN: 978-94-6191-809-3.

C.M. (Caroline) Sonnenberg (2013). Late life depression: sex differences in clinical presentation and medication use. Vrije Universiteit Amsterdam. ISBN: 978-94-6191-866-6.

DISSERTATION SERIES| 189

Z. (Zsuzsika) Sjoerds (2013). Alcohol dependence across the brain: from vulnerability to compulsive drinking. Vrije Universiteit Amsterdam. ISBN: 978-90-8891-695-3.

V.J.A. (Victor) Buwalda (2013). Routine Outcome Monitoring in Dutch Psychiatry: Measurement, Instruments, Implementation and Outcome. Vrije Universiteit Amsterdam. ISBN: 978-94-6191-905-2.

J.G. (Josine) van Mill (2013). Sleep, depression and anxiety: an epidemiological perspective. Vrije Universiteit Amsterdam. ISBN: 978-94-6108-525-2.

S. (Saskia) Woudstra (2013). Framing depression in a SN[a]Pshot: Imaging risk factors of MDD. Vrije Universiteit Amsterdam. ISBN: 978-90-8891-751-6.

N.C.M. (Nicole) Korten (2014). Stress, depression and cognition across the lifespan. Vrije Universiteit Amsterdam. ISBN: 978-94-6108-748-5.

M.K. (Maarten) van Dijk (2014). Applicability and effectiveness of the Dutch Multidisciplinary Guidelines for the treatment of Anxiety Disorders in everyday clinical practice. Vrije Universiteit Amsterdam. ISBN: 978-94-92096-00-5.

I.M.J. (Ilse) van Beljouw (2015). Need for Help for Depressive Symptoms from Older Persons Perspectives: The Implementation of an Outreaching Intervention Programme. Vrije Universiteit Amsterdam. ISBN: 978-94-6259-496-8.

A.M.J. (Annemarie) Braamse (2015). Psychological aspects of hematopoietic stem cell transplantation in patients with hematological malignancies. Vrije Universiteit Amsterdam. ISBN: 978-94-6259-594-1.

A. (Annelies) van Loon (2015). The role of ethnicity in access to care and treatment of outpatients with depression and/or anxiety disorders in specialised care in Amsterdam the Netherlands. Vrije Universiteit Amsterdam. ISBN: 978-94-90791-34-6.

C. (Chris) Vriend (2015). (Dis)inhibition: imaging neuropsychiatry in Parkinsonâ&#x20AC;&#x2122;s disease. Vrije Universiteit Amsterdam. ISBN: 978-94-6295-115-0.

A.M. (Andrea) Ruissen (2015). Patient competence in obsessive compulsive disorder. An empirical ethical study. Vrije Universiteit Amsterdam. ISBN: 978-90-6464-856-4.

H.M.M. (Henny) Sinnema (2015). Tailored interventions to implement guideline recommendations for patients with anxiety and depression in general practice. Vrije Universiteit Amsterdam. ISBN: 978-946169-653-3.

T.Y.G. (Nienke) van der Voort (2015). Collaborative Care for patients with bipolar disorder. Vrije Universiteit Amsterdam. ISBN: 978-94-6259-646-7.

190 | DISSERTATION SERIES

W. (Wim) Houtjes (2015). Needs of elderly people with late-life depression; challenges for care improvement. Vrije Universiteit Amsterdam. ISBN: 978-94-6108-985-4.

M. (Marieke) Michielsen (2015). ADHD in older adults. Prevalence and psychosocial functioning. Vrije Universiteit Amsterdam. ISBN: 978-90-5383-132-8.

S.M. (Sanne) Hendriks (2016). Anxiety disorders. Symptom dimensions, course and disability. Vrije Universiteit Amsterdam. ISBN: 978-94-6259-963-5.

E.J. (Evert) Semeijn (2016). ADHD in older adults; diagnosis, physical health and mental functioning.

Vrije Universiteit Amsterdam. ISBN: 978-94-6233-190-7.

N. (Noera) Kieviet (2016). Neonatal symptoms after exposure to antidepressants in utero. Vrije Universiteit Amsterdam. ISBN: 978-94-6169-794-3.

W.L. (Bert) Loosman (2016). Depressive and anxiety symptoms in Dutch chronic kidney disease patients. Vrije Universiteit Amsterdam. ISBN: 987-94-6169-793-6.

E. (Ellen) Generaal (2016). Chronic pain: the role of biological and psychosocial factors. Vrije Universiteit Amsterdam. ISBN: 978-94-028-0032-6.

D. (Dóra) Révész (2016). The interplay between biological stress and cellular aging: An epidemiological perspective. Vrije Universiteit Amsterdam. ISBN: 978-94-028-0109-5.

F.E. (Froukje) de Vries (2016). The obsessive-compulsive and tic-related brain. Vrije Universiteit Amsterdam. ISBN: 978-94-629-5481-6.

J.E. (Josine) Verhoeven (2016). Depression, anxiety and cellular aging: does feeling blue make you grey?

Vrije Universiteit Amsterdam. ISBN: 978-94-028-0069-2.

A.M. (Marijke) van Haeften-van Dijk (2016). Social participation and quality of life in dementia: Implementation and effects of interventions using social participation as strategy to improve quality of life of people with dementia and their carers. Vrije Universiteit Amsterdam. ISBN: 978-94-6233-341-3.

P.M. (Pierre) Bet (2016). Pharmacoepidemiology of depression and anxiety. Vrije Universiteit Amsterdam. ISBN: 978-94-6299-388-4.

M.L. (Mardien) Oudega (2016). Late life depression, brain characteristics and response to ECT. Vrije Universiteit Amsterdam. ISBN: 978-94-6295-396-3.

H.A.D. (Henny) Visser (2016). Obsessive-Compulsive Disorder; Unresolved Issues, Poor Insight and Psychological Treatment. Vrije Universiteit Amsterdam. ISBN: 978-94-028-0259-7.

DISSERTATION SERIES| 191

E.C. (Eva) Verbeek (2017). Fine mapping candidate genes for major depressive disorder: Connecting the dots. Vrije Universiteit Amsterdam. ISBN: 978-94-028-0439-3.

S. (Stella) de Wit (2017). In de loop: Neuroimaging Cognitive Control in Obsessive-Compulsive Disorder.

Vrije Universiteit Amsterdam. ISBN: 978-90-5383-225 7.

W.J. (Wouter) Peyrot (2017). The complex link between genetic effects and environment in depression.

Vrije Universiteit Amsterdam. ISBN: 978-94-6182-735-7.

R.E. (Rosa) Boeschoten (2017). Depression in Multiple Sclerosis: Prevalence Profile and Treatment. Vrije Universiteit Amsterdam. ISBN: 978-94-028-0474-4.

G.L.G. (Gerlinde) Haverkamp (2017). Depressive symptoms in an ethnically DIVERSe cohort of chronic dialysis patients: The role of patient characteristics, cultural and inflammatory factors. Vrije Universiteit Amsterdam. ISBN: 978-94-6233-528-8.

T.J. (Tjalling) Holwerda (2017). Burden of loneliness and depression in late life. Vrije Universiteit Amsterdam. ISBN: 978-94-6233-598â&#x20AC;&#x201C;1.

J. (Judith) Verduijn (2017). Staging of Major Depressive Disorder. Vrije Universiteit Amsterdam. ISBN: 978-94-6299-597-0.

C.N. (Catherine) Black (2017). Oxidative stress in depression and anxiety disorders. Vrije Universiteit Amsterdam. ISBN: 978-94-6299-672-4.

J.B. (Joost) Sanders (2017). Slowing and Depressive Symptoms in Aging People. Vrije Universiteit Amsterdam. ISBN: 978-94-6233-650-6.

W. (Willemijn) Scholten (2017). Waxing and waning of anxiety disorders: relapse and relapse prevention. Vrije Universiteit Amsterdam. ISBN: 978-94-6299-606-9.

P. (Petra) Boersma (2017). Person-centred communication with people with dementia living in nursing homes; a study into implementation success and influencing factors. Vrije Universiteit Amsterdam.

ISBN: 978-94-6233-725-1.

T.I. (Annet) Bron (2017). Lifestyle in adult ADHD from a Picasso point of view. Vrije Universiteit Amsterdam. ISBN: 978-94-6299-685-4.

S.W.N. (Suzan) Vogel (2017). ADHD IN ADULTS: seasons, stress, sleep and societal impact. Vrije Universiteit Amsterdam. ISBN: 978-94-6299-673-1.

192 | DISSERTATION SERIES R. (Roxanne) Schaakxs (2018). Major depressive disorder across the life span: the role of chronological and biological age. Vrije Universiteit Amsterdam. ISBN: 978-94-6299-819-3.

J.J. (Bart) Hattink (2018). Needs-based enabling- and care technology for people with dementia and their carers. Vrije Universiteit Amsterdam. ISBN: 978-94-6295-880-7.

F.T. (Flora) Gossink (2018). Late Onset Behavioral Changes differentiating between bvFTD and psychiatric disorders in clinical practice. Vrije Universiteit Amsterdam. ISBN: 978-94-6295-899-9.

R. (Roxanne) Gaspersz (2018). Heterogeneity of Major Depressive Disorder. The role of anxious distress.

Vrije Universiteit Amsterdam. ISBN: 978-94-028-1076-9.

M.M. (Marleen) Wildschut (2018). Survivors of early childhood trauma and emotional neglect: who are they and whatâ&#x20AC;&#x2122;s their diagnosis? Vrije Universiteit Amsterdam. ISBN: 978-94-6332-401-4.

J.A.C. (Jolanda) Meeuwissen (2018). The case for stepped care. Exploring the applicability and costutility of stepped-care strategies in the management of depression. Vrije Universiteit Amsterdam. ISBN: 978-90-5383-359-9.

D.S. (Dora) Wynchank (2018). The rhythm of adult ADHD. On the relationship between ADHD, sleep and aging. Vrije Universiteit Amsterdam. ISBN: 978-94-6375-034-9.

M.J. (Margot) Metz (2018). Shared Decision Making in mental health care: the added value for patients and clinicians. Vrije Universiteit Amsterdam. ISBN: 978-94-6332-403-8.

I. (Ilse) Wielaard (2018). Childhood abuse and late life depression. Vrije Universiteit Amsterdam. ISBN: 978-94-6380-072-3.

L.S. (Laura) van Velzen (2019). The stressed and depressed brain. Vrije Universiteit Amsterdam. ISBN: 978-94-6380-062-4.

S. (Sonja) Rutten (2019). Shedding light on depressive, anxiety and sleep disorders in Parkinsonâ&#x20AC;&#x2122;s disease. Vrije Universiteit Amsterdam. ISBN: 978-94-6380-176-8.

N.P.G. (Nadine) Paans (2019). When you carry the weight of the world not only on your shoulders.

Factors associating depression and obesity. Vrije Universiteit Amsterdam. ISBN: 978-94-6380-141-6.

D.J. (Deborah) Gibson-Smith (2019). The Weight of Depression. Epidemiological studies into obesity, dietary intake and mental health. Vrije Universiteit Amsterdam. ISBN: 978-94-6380-144-7.

C.S.E.W. (Claudia) Schuurhuizen ( 2019). Optimizing psychosocial support and symptom management for patients with advanced cancer. Vrije Universiteit Amsterdam. ISBN: 978-94-6323-468-9.

DISSERTATION SERIES| 193 M.X. (Mandy) Hu (2019). Cardiac autonomic activity in depression and anxiety: heartfelt afflictions of the mind. Vrije Universiteit Amsterdam. ISBN: 978-94-6380-206-2.

J..K. (Jan) Mokkenstorm (2019). On the road to zero suicides: Implementation studies.

Vrije Universiteit Amsterdam. ISBN: 978-94-6361-224-1.

S.Y. (Sascha) Struijs (2019). Psychological vulnerability in depressive and anxiety disorders. Vrije Universiteit Amsterdam. ISBN: 978-94-6380-244-4.

H.W. (Hans) Jeuring (2019). Time trends and long-term outcome of late-life depression: an epidemiological perspective. Vrije Universiteit Amsterdam. ISBN: 978-94-6380-228-4.

R. (Ruth) Klaming Miller (2019). Vulnerability of memory function and the hippocampus: Risk and protective factors from neuropsychological and neuroimaging perspectives. Vrije Universiteit Amsterdam. ISBN: 978-94-6182-955-5.

P.S.W. (Premika) Boedhoe (2019) The structure of the obsessive-compulsive brain â&#x20AC;&#x201C; a worldwide effort.

Vrije Universiteit Amsterdam.ISBN: 978-94-6380-329-8.

C.S. (Carisha) Thesing (2020). Fatty acids in depressive and anxiety disorders: fishing for answers. Vrije Universiteit Amsterdam. ISBN: 978-94-6375-846-8.

R.D. (Richard) Dinga (2020). Evaluation of machine learning models in psychiatry.

Vrije Universiteit Amsterdam.

M. (Mayke) Mol (2020). Uptake of internet-based therapy for depression: the role of the therapist. Vrije Universiteit Amsterdam. ISBN: 978-94-6416-150-2