Evaluating provider training in stepped care 2.0 and one-at-a-time services among mental health and addiction providers

Abstract

Background

See More

Stepped Care 2.0 (SC2.0) and One-at-a-Time (OAAT) approaches can help address challenges related to accessing effective addiction and mental health (A&MH) services. OAAT services, available by walk-in or appointment, were implemented in New Brunswick (NB) as the first step in developing a provincial stepped care framework in alignment with NB’s A&MH action plan. This study sought to evaluate the impact of online training courses in SC2.0 and OAAT service delivery on providers’ knowledge, readiness, and capabilities to implement OAAT services in A&MH clinics, within the broader context of the provincial SC2.0 model.

Methods

Providers employed with A&MH services (e.g., social workers, nurses, psychologists) across NB completed asynchronous training courses in SC2.0 and OAAT services as part of a provincial implementation initiative. Over 400 providers volunteered to complete questionnaires related to this training (N = 401). Knowledge acquisition questionnaires were developed based on SC2.0 course content and administered pre- and post-training. Providers also completed a post-training knowledge acquisition questionnaire on OAAT services. Providers completed questionnaires on acceptability, appropriateness and feasibility of training courses, and self-efficacy post-training. Qualitative interviews were conducted with 28 providers to further understand their experiences with training courses in SC2.0 (n = 12) and OAAT services (n = 16).

Results

Mean percentage of correct responses at post-course for SC2.0 and OAAT services was 67.2% (SD = 15.9%) and 75.7% (SD = 15.7%), respectively. A modest, but significant, increase in knowledge of SC2.0 was observed post-training. Courses were deemed acceptable, appropriate and feasible, and resulted in favorable outcome expectancies. Moreover, providers reported modest self-efficacy to enact SC2.0 following training. Providers made recommendations to receive additional resources and training in SC2.0 and OAAT services to further enhance confidence to integrate key principles into practice.

Conclusions

Asynchronous training courses in SC2.0 and OAAT services supported the provincial practice change initiative in NB. In line with the COM-B model of behavior change, course barriers and facilitators were identified and provide insights into ways in which these courses, and related implementation projects involving training healthcare professionals, could be adapted to help create and sustain change.

Historically, mental health has not been prioritized in Canadian health policy [6], and the delivery of mental health services has been described as a silent crisis [45]. Similar to other jurisdictions across North America [52], addiction and mental health (A&MH) services across Canada are overburdened, with Canadians of all ages facing long wait times for services [10, 13, 34] and uncertainty surrounding where to receive care [40]. Mental healthcare needs are often unmet, as evidenced through the well-documented mental health treatment gap, which indicates that many individuals do not receive the mental healthcare that they feel they need [9]. As such, it is important that financial investments in mental healthcare [22], often aimed at increasing access to mental healthcare, begin to address these barriers to meet the needs of Canadians more effectively and efficiently.

Addressing barriers to accessing A&MH care requires increasing the integration of needs-based service delivery to better manage service demand, which would also help address service fragmentation issues reported by Canadian mental healthcare providers [56]. Arranging available resources across a broad and integrated continuum of care and facilitating open access drop-in clinics allows for more timely access to services, which is one such way to meet user demand [54]. A stepped care approach, which was originally developed in the United Kingdom, holds potential to address the challenges associated with accessing effective A&MH services. Adapted from the original model which favored a staging approach, Stepped Care 2.0 (SC2.0) is an approach to A&MH care that emphasizes service user needs, preferences, choice, and readiness, and aims to better balance service demand with available resources [15]. SC2.0 is an open-access model that has been imagined as a 9-step framework comprising low intensity (e.g., guided self-help, peer support), medium intensity (e.g., group psychotherapy) and high intensity (e.g., specialist consultation) resources (Refer to Fig. 1). SC2.0 is unique as clients, in collaboration with care providers, can decide to enter, exit and transition to any level of care based on preferences, readiness, and needs [15]. Shared-care decision-making is prioritized within the SC2.0 model, and used to determine next steps and the appropriate level of care when discrepancies arise between clients and providers. Clients are provided fully informed consent (e.g., potential risks and benefits, anticipated wait times, resources most likely to lead to positive outcomes), and their preferences are privileged when appropriate. Refer to Fig. 2 for a description of the Core Components underlying the SC2.0 model.

Fig. 1

Description of SC2.0 9 Step Framework, from Stepped Care Solutions

Fig. 2

Description of SC2.0 Core Components, from Stepped Care Solutions [11]

Several Canadian provinces and territories are in the process of incorporating SC2.0 into A&MH services, including New Brunswick (NB), Prince Edward Island, and the Northwest Territories. For example, a 5-Year Interdepartmental Addiction and Mental Health Action Plan was launched by NB in 2021 [42]. This action plan aims to achieve five goals: (1) improve population health, (2) intervene early; (3) match individuals to effective care; (4) improve access to services; and (5) reduce drug-related impacts. A cornerstone of this plan includes the implementation of a Stepped Care framework for the delivery of A&MH Services that began in 2021 (Refer to [24] for a structural overview of NB’s A&MH services). Using a phased approach, the province implemented One-at-a-Time (OAAT) services, available by walk-in (a.k.a., Single Session Therapy [SST]) or appointment, within Horizon and Vitalité Health Networks as a first step towards a provincial SC2.0 model (refer to [24]). These services were offered at 14 Community Addiction and Mental Health Centres (CAMHCs) serving adult clientele across the province and through 44 child and youth teams [24, 25]. The availability of walk-in services delivered using an approach that addresses top of mind concerns represents one of the SC2.0 core components—facilitating rapid access to care—which has been associated with reduced wait times across NB [24], and Newfoundland and Labrador (NL) [39]. OAAT/SST is a service delivery model that aims to increase access to care by supporting clients as they address their top-of-mind concern one session at a time [18, 28,29,30, 51, 53]. Each session is treated as if it may be the last contact point, where clients have the autonomy to decide if, and when, to avail of another session [30]. Young [61] described SST as a way of thinking about therapy that accepts three robust research findings: (1) “the most common number of service contacts that clients attend is one, followed by two, followed by three, irrespective of diagnosis, complexity, or the severity of their problem”; (2) “the majority (often about 70–80%) of those people who attend only one session, across a range of therapies, report that the single session was adequate given their current circumstance”; and (3) “it seems impossible to accurately predict who will attend only one session and who will attend more … the first session should logically be conducted “as if” it may also be the last.” (p 44). The first session has also been shown to be important in establishing the therapeutic alliance [17], which has a medium effect size (d = 0.57) on clinical outcomes [43] and can help build client readiness. Within the OAAT approach, continuity of service with the same provider is not guaranteed. As such, OAAT services privilege the alliance by drawing heavily on collaboration and goal consensus, which have medium sized effects on therapeutic outcomes [43].

Implementation of new services, such as OAAT services, on a province wide scale is a complex process that involves facilitating behavior change among healthcare professionals.

Drawing from the implementation of clinical practice guidelines, provider adoption of evidence-based therapeutic practices for the management of mental health concerns are low with adherence rates ranging between 24 and 52% [48, 60]. A recent systematic synthesis of qualitative research identified several factors that reliably influence the uptake of evidence into practice, including: (1) knowledge and self-efficacy, (2) anticipation of client outcomes; (3) client preference and readiness to engage in treatment; and (4) availability of professional supports [7]. These factors are remarkably similar to those reported across the field of medicine [35] which is of particular relevance to the delivery of OAAT services which align closely with primary care walk-in clinics. One systematic review of 20 studies reported that the most frequent barriers to integrating mental health services into primary care included: (1) attitudes regarding the acceptability, appropriateness, and credibility of the service, (2) knowledge and skills to implement the practice; and (3) motivation to change [55]. As such, it is important to design and evaluate robust approaches to training that: (1) are deemed acceptable, appropriate, and credible, (2) enhance knowledge and self-efficacy; and (3) shape attitudes towards positive outcome expectancies. One recent systematic review was conducted that reported on the effect of eLearning for capacity building among healthcare professionals [2]. This review identified 44 studies that reported on more than 40,000 providers and concluded that various types of eLearning (e.g., synchronous, blended, self-learning were effective in advancing healthcare professionals’ knowledge, attitudes, and practices [2]. Similarly, one systematic review of therapist training in mental health suggested that workshops and more intensive online training hold promise to influence knowledge, and attitudes [21]. It is important to note that changing knowledge and attitudes are necessary but often insufficient to result in meaningful behaviour change [21, 49].

Current study

OAAT services, available by walk-in or appointment, were implemented in NB as the first step in developing a provincial SC2.0 framework in alignment with their A&MH action plan. Providers completed asynchronous training courses in SC2.0 and OAAT services, as well as virtual face-to-face training in OAAT services delivered over Zoom by recognized experts. The primary objective of the current study was to evaluate the impact of the online training courses on providers’ knowledge, readiness, and capabilities to implement OAAT services in A&MH clinics. Completion of an applied training course can enhance motivation and provide innovation specific capacities (e.g., knowledge and self-efficacy) which are crucial precursors to appropriately and competently enact SC2.0. The secondary objective of this study was to assess perceived acceptability of the online training courses, and experiences completing them. Interviews were included to better capture providers’ perceptions and experiences completing the courses.

Methods

Context and methodological approach

This research was part of a multi-provincial evaluation of the implementation of core components of SC2.0 in several Atlantic Canadian provinces, and was conducted in collaboration with A&MH Services in NB and Stepped Care Solutions, a not-for-profit mental health system consultancy group. A mixed-methods design was used that included surveys and interviews to evaluate the impacts of completing online asynchronous courses in SC2.0 and OAAT services on providers’ knowledge and attitudes. The online training courses included text and videos to relay key principles and concepts, and exercises for providers to apply their knowledge and understanding of the content using a graded task approach (i.e., multiple choice responses and providing responses to open-text responses to facilitate self-reflective practice through journaling). Refer to Tables 1 and 2 for an outline of course content. Completion time for each course was 3–5 h, and providers received organizational support to complete the courses, including time. A subset of providers were purposively selected to complete an interview to better understand their experiences with the training courses. At the time of data collection, this provincial practice change was primarily in the installation and initial implementation stages. Refer to Harris-Lane et al. [24] and Harris-Lane et al. [25] for additional details relating to the implementation process. This research study was approved by ethics committees in NL (NL Health Research Ethics Board Ref# 2021.094) and NB (Horizon Health Network Ethics Board: Ref# 2021-3015, Vitalité Health Network Ethics Board: Ref# 2957).

Table 1 Introduction to Stepped Care 2.0 Course Outline

Table 2 One-at-a-Time Services Course Outline

Participants and recruitment

Healthcare providers (e.g., social workers, nurses, psychologists, counsellors) working across NB’s regional health authorities, Horizon and Vitalité, and school districts were eligible to participate in research associated with the implementation of OAAT services and a provincial SC2.0 model. These health networks are responsible for the provision of A&MH services in hospitals, schools, clinics and other community settings, and are overseen by NB’s Department of Health. Although only a subset of providers were expected to use OAAT services, all providers were expected to complete the training courses as it was deemed important for all providers to understand the upcoming practice change given the anticipated impact on operating procedures, such as referrals. Providers were recruited through team meetings and targeted emails initiated by the management and Directors of A&MH Services. Emails to providers described the provincial practice change initiative and online courses, and offered providers the opportunity to participate in optional research to understand their attitudes and perceptions. Providers were directed to Qualtrics, where they reviewed the informed consent form and provided consent to participate. Providers did not need to participate in research to receive access to the training.

Procedures

Surveys

As outlined in Table 3, providers who consented to participate completed the SC2.0 and OAAT services courses, interspersed by three surveys. Courses were delivered in a counterbalanced manner, with adult providers beginning with the OAAT services course, and child and youth providers beginning with the SC2.0 course, refer to Table 3. Pre- and post-course surveys contained questionnaires assessing: (1) knowledge of SC2.0; (2) post-course knowledge of OAAT services; (3) perceptions of the acceptability, appropriateness, and feasibility of implementing SC2.0 into one’s practice; (4) stage of change to implement SC2.0 into practice; and (5) acceptability of the online courses. Given that cut offs are not currently available for measures of acceptability, appropriateness, and feasibility [23], no a-priori metric was used to evaluate these outcomes. Rather, qualitative interviews were completed with a purposefully selected sample to further understand providers’ perceptions of the courses. Other studies have also used similar mixed methods approaches to evaluate acceptability [23, 27].

Table 3 Schedule of Study Assessments for Providers Working with Adult Populations

Interviews

A sample of providers who consented to being contacted for future research were invited to participate in an interview on their experiences completing the online courses. Providers were sampled in a purposeful manner to ensure representation across setting (i.e., health networks and education districts), population served (i.e., adult, and children & youth), and discipline (e.g., nursing, social work). A 60-min virtual interview was arranged with each consenting provider. Semi-structured interviews were conducted using the WebEx or Zoom platform depending on the provider’s preference.

Surveys and interviews were offered in English and French. Participation was incentivised through $20 gift cards offered for completing the first survey, third survey, and interview for a total possible incentive of $60.

Measures

Surveys

Demographics. Information pertaining to the organization, practice setting and location in which participants worked was collected. Participants’ education level, current profession and role within the organization were also obtained, in addition to the clinical populations they served.

Knowledge

SC2.0 Knowledge Acquisition Questionnaires. Knowledge of SC2.0 core components, guiding principles, and the continuum of care was measured using questionnaires developed by our team, in consultation with Stepped Care Solutions. Questionnaires were 30-items in length and in a multiple-choice format with 4 response options to select from. Two parallel versions were developed to allow for pre- and post-test administrations with a lower likelihood of carry-over effects.

OAAT Knowledge Acquisition Questionnaire. Knowledge of fundamental principles of OAAT services was measured at post-course using one questionnaire developed by our team, in consultation with Stepped Care Solutions and international experts in OAAT services. This 25-item questionnaire used a multiple-choice format with 4 response options to select from.

Acceptability, appropriateness, feasibility and stage of change

Acceptability, Appropriateness and Feasibility of Intervention Measure (AAFI). Acceptability, appropriateness, and feasibility of SC2.0 as a model of care delivery was evaluated using a 12-item questionnaire [57]. The AAFI is a psychometrically strong measure that is commonly used within implementation projects [37, 57]. Items were rated on a 5-point Likert scale, with anchors at 1-“Completely disagree” and 5-“Completely agree”, with higher mean scores suggestive of greater acceptability, appropriateness, and feasibility.

Acceptability of Training Questionnaire (AoTQ-8). Provider acceptability of online courses was evaluated using 8-items taken from the Acceptability of Training Questionnaire (AoTQ-8) [47]. The AoTQ was developed to capture pertinent facets of the Theoretical Framework of Acceptability, such as attitude towards the intervention and intention to adopt the intervention [50]. The AoTQ was administered following completion of each online course. Items were presented as 5-point Likert scales, ranging from 1- “Strongly disagree” to 5- “Strongly agree”, with higher mean scores suggestive of greater acceptability, appropriateness, and feasibility.

Self-Efficacy to Enact Stepped Care (SEESC-8). A scale measuring Self-Efficacy to Enact Stepped Care (SEESC-8) was created in accordance with Bandura’s guideline for constructing self-efficacy scales [5]. This 8-item scale measures a provider’s confidence to enact SC2.0 in difficult situations or with difficult clients and was administered after completing the SC2.0 asynchronous course. Items were rated on a 10-point Likert scale with anchors at 10%—“Cannot do at all” and 100%—“Highly certain can do”.

Stages of Change–Stepped Care (SOC-SC). Providers’ stage of change to adopt SC2.0 (e.g., precontemplation, contemplation, preparation, action, maintenance) was measured using the SOC-SC scale. This 2-item measure was adapted from Willey et al.’s [59] measure of stage of change for medication adherence and aligns with the transtheoretical model [46].

Interviews

The study team created a semi-structured interview template based on the Theoretical Framework of Acceptability [50] to further understand providers’ experiences completing the asynchronous training courses. Interviews focused on course effectiveness as well as successes and struggles involved in completing the courses.

Statistical analysis

Analyses were performed using IBM SPSS Statistics v27 [31]. A small number of participants (less than 5%) did not complete the full survey. Missing data was not imputed given the descriptive nature of the study and proportion of missing data, refer to Table 5.

Characterizing the sample

Descriptive statistics (e.g., frequencies, means, and standard deviations) were computed to characterize the sample.

Psychometric evaluation of knowledge acquisition questionnaires

Discriminability (a-par) and difficulty (b-par) of items used in knowledge acquisition questionnaires were evaluated using a 2-parameter-logistic (2PL) item response theory model estimated in a marginal maximum likelihood framework using jMetrik [38]. The 2PL model includes the log-odds of endorsement, and additionally, a discrimination parameter, which is the slope of the S-shaped logistic curve at its inflection point [32]. Values of a-par typically range between 0 and 2 with higher values indicating that the item has better ability to tell the difference between different levels of latent ability. Values of a-par in excess of 0.3 indicate adequate discrimination. Values of b-par range from − 3.0 to 3.0 with larger positive values indicating greater difficulty.

Knowledge

Change in knowledge of SC2.0 from pre- to post-test was measured using paired samples t-tests. Mean correct response rate following OAAT training was also reported, given that comparable data was not available at pre-test.

Acceptability, appropriateness, feasibility, and stage of change

Descriptive statistics were conducted to evaluate acceptability, appropriateness and feasibility of courses and implementing SC2.0 into practice, and to understand stage of change and self-efficacy to enact SC2.0 in practice.

Interviews

Thematic analysis was performed using an inductive, descriptive approach to develop individual codes and overarching themes from consistent patterns found in collected data [8]. After familiarization with transcripts, 10 reviewers, including AK, LHL, and JAR, synthesized data in an iterative process, discussed emerging themes and continuously refined definitions [16].

Results

Sample characteristics

As part of a provincial implementation initiative, providers employed within A&MH services in NB (approximately 900 in total) were invited to complete training. In total, 775 providers completed the training courses, and 401 (52%) participated in the research. The majority of the 401 participating providers worked within NB’s two regional health authorities, Horizon Health Network (48.6%), followed by Vitalité Health Network (35.2%), and within the school districts (15.5%) with 0.7% reporting an “other” organization. Over half of providers worked in urban settings (55.9%) and served child and youth populations (54.1%), with over 36% working in community mental health and addictions clinics. Almost 80% of providers reported their professional role as “provider”. Providers were employed within A&MH Services and delivered services to communities across NB, with a small proportion employed within primary care. Although training backgrounds were varied, most providers were trained in social work (55.4%) or nursing (16.0%), and over half of providers held a Baccalaureate (59.6%), refer to Table 4.

Table 4 Sample Characteristics

Psychometric evaluation of knowledge acquisition questionnaires

The results of the 2PL item response theory model for pre- and post-SC2.0 knowledge acquisition questionnaires are located in Supplemental Table A1. One item (#14) obtained a low correct response rate of < 5% on pre- and post-test assessment and was removed prior to running the 2PL model. Items that fell below the 0.30 threshold, thus demonstrating poor discriminability during pre-test (item #9, 30, 22), and post-test (item #26), were also removed, along with items deemed overly easy (item #17) or difficult (item #2), as indicated by difficulty parameters outside the typical range. Removing these items resulted in a 24-item knowledge acquisition questionnaire. Of the remaining items, most had a difficulty parameter below 0 and were judged as relatively easy. Most items were less difficult during post-test relative to pre-test as indicated by a shift in the difficulty parameter to greater negative values.

The results of the 2PL item response theory model for the post-course OAAT services knowledge acquisition questionnaire are located in Supplemental Table A2. One item exhibited poor discriminability (item #22) while item #1, 4, 5, 7 and 17 were considered very easy items based on b-par values. These items were removed and resulted in a 19-item questionnaire. Many of the remaining items had a difficulty parameter below 0 and were judged as relatively easy.

Knowledge

SC2.0

Knowledge of SC2.0 increased significantly following the completion of the SC2.0 training course (M_Pre-SC = 61.4%, SD = 16.9%; M_Post-SC = 67.2%, SD = 15.9%, t(392) = 8.08, SEM = 0.00709, p ≤ 0.001) with a small to medium effect size, d = 0.41. Reliable knowledge gains were observed on items that pertained to Core Component 8 (i.e., data-informed decision-making) and matching resources to the corresponding intensity within the care continuum, refer to Supplemental Table A1.

OAAT therapy

Mean correct response rate at post-course was 75.7% (SD = 15.7%).

Acceptability, appropriateness, feasibility, and stage of change

SC2.0

Providers reported SC2.0 to be acceptable (AAFI_{Acceptability}: M = 4.25, SD = 0.64) and appropriate (AAFI_{Appropriateness}: M = 4.13, SD = 0.68), and feasible to implement into practice (AAFI_Feasibility: M = 3.91, SD = 0.72). Providers also endorsed modest self-efficacy to enact principles of SC2.0 into practice (SEESC-8: M = 66.3, SD = 15.04; AoTQ-8_{Self Efficacy}: M = 3.94, SD = 0.73). Providers reported the SC2.0 training course to be enjoyable (AoTQ-8_Attitude: M = 4.08, SD = 0.74), and did not report the course to be troublesome (AoTQ-8_Burden: M = 2.56, SD = 1.00). The course content was considered to be digestible and provided an understanding of SC2.0 principles as well as techniques to use in practice (AoTQ-8_{Intervention Coherence}: M = 3.99, SD = 0.64). Providers also reported that SC2.0 would be effective in addressing client concerns (AoTQ-8_{Effectiveness}: M = 3.97, SD = 0.83) with minimal opportunity costs (AoTQ-8_{Opportunity Cost}: M = 2.18, SD = 0.98). Over 10% of providers reported using stepped care principles in practice post-training, while 55.9% of providers were planning to incorporate principles of SC2.0 into their practice. Of note, 5.8% of providers indicated they were not considering using principles of SC2.0 in their practice post-training, refer to Table 5.

Table 5 Variables impacting uptake of SC2.0 and OAAT services into practice post-training

OAAT services

Providers endorsed a positive attitude towards the OAAT services course (AoTQ-8_Attitude: M = 4.30, SD = 0.64), and did not find the course to be burdensome (AoTQ-8_Burden: M = 2.21, SD = 0.97). Providers reported that the course content gave them an understanding of OAAT principles as well as techniques to use in practice (AoTQ-8_{Intervention Coherence}: M = 4.25, SD = 0.53). Moreover, providers endorsed that they were capable of applying OAAT principles into practice following course completion (AoTQ-8_{Self-Efficacy}: M = 4.10, SD = 0.65). OAAT services was also endorsed by providers to be effective in addressing client concerns (AoTQ-8_{Effectiveness}: M = 4.05, SD = 0.73) with minimal opportunity costs (AoTQ-8_{Opportunity Cost}: M = 2.23, SD = 1.01).

Interviews

Qualitative interviews were conducted with a convenience sample of 28 providers to understand their experiences with training courses. The majority of providers interviewed worked within Horizon Health Network (71.4%), 17.9% worked within Vitalité Health Network, 7.1% worked within the school districts, and 3.6% worked for other organizations. Over half of providers worked as part of child and youth teams within healthcare (53.6%), 35.7% worked within adult community A&MH clinics, and 10.7% worked as part of child and youth teams within school districts.

Perceptions of stepped care 2.0 course

Interviews were conducted that focused on perceptions and experiences with the SC2.0 training course with 12 providers. Table 6 presents the summary of findings from these interviews.

Table 6 Summary of Findings for Qualitative Interviews of Stepped Care 2.0

Successes in SC2.0 Course Development. The majority of providers felt that the training course was well-developed, providing a range of content that increased focus and engagement:

“I thought it was perfect because it was a nice combination. So sometimes when you take training and you’re just reading and reading and reading it kind of gets monotonous and the way that the training went, it switched so you were reading and then you were answering questions, and then you were giving feedback or writing in the journal, so it kept you thinking and focused on the training that you did.” (P54).

The majority of providers noted that the depiction of the SC2.0 model clearly broke the service delivery model into its nine components. Some providers reported that the online course was easy to navigate and helped them better understand the SC2.0 model, with some endorsing that they developed the necessary skills to implement this type of model into their clinical practice. The value of SC2.0 to providers’ professional development was also reported in interviews, with some providers indicating they would recommend the course to colleagues.

Critiques of Course Content. Providers noted that the training course was information-heavy, which may impact engagement in learning the course content. Some providers noted that the course was ideal for individuals who prefer to learn through reading and writing, but not for those who may have diverse learning needs (e.g., hands-on practice).

Barriers and Facilitators of Course Completion. Logistical challenges, such as accessing course passwords were reported as barriers to completing the training course by some providers. Additionally, competing work demands and insufficient protected time at work to complete courses were cited as barriers to course completion:

“Sometimes it was hard just to carve out the time, like, just from an operational perspective... Like I said, I made time to do it. So, I think just making sure once I had time carved out it wasn’t hard to complete.” (P1014)

Similarly, providers reported that incorporating natural breaks into the training modules helped them complete the course flexibly. Similarly, providers indicated that it was beneficial to have dedicated time at work allocated to course completion:

“.... we were given the invitation to focus on the training instead of our regular work duties to get the training done, which was a big help…” (P1013)

Suggestions for Course Improvement. Providers suggested that it would be beneficial to incorporate more practical elements to help with learning the material as well as more interactive components. They also suggested that a refresher course, or an advanced training course, would enable providers to continue developing their skillset in SC2.0. Similarly, some providers noted that additional resources (e.g., workbooks and printed material) may aid in the implementation of SC2.0 into their practices.

Perceptions of OAAT services course

Interviews were conducted with 16 providers that focused on perceptions and experiences with the OAAT services training course. Tables 7 and 8 present the summary of findings from these interviews about the asynchronous OAAT services course and live OAAT training, respectively.

Table 7 Summary of Findings for Qualitative Interviews of One-at-a-Time Services Online Training

Table 8 Summary of Findings for Qualitative Interviews of One-at-a-Time Services Live Training

Successes in OAAT Course Development. Some providers appreciated the flexibility and self-paced design of the course and reported that the course content and layout was clear:

“I think One-at-a-Time was explained really well. I liked the format and everything, and I thought that was pretty clear.” (P13)

Most providers reported that course activities (e.g., self-reflective exercises, knowledge checks) enhanced engagement. The inclusion of videos appeared to further increase engagement in course material as noted by most providers and may have been particularly beneficial for providers who would describe themselves as visual and auditory learners. Moreover, some providers appreciated the inclusion of summaries and found them to be beneficial in understanding OAAT services, along with the use of journal entries.

Critiques of Course Content. A minority of providers interviewed reported that the course content was repetitive, and some noted that the course was lengthy. In addition, half of providers felt that the examples provided in course videos were simplistic and may not be representative of their clinical practice. Not surprisingly, most providers reported a need for additional resources, such as handouts, forms, etc.:

“I would’ve loved to see… other issues tackled or some additional links. I would have loved to [see] that, and I would have clicked on every single one of them. Some higher range things like people coming with trauma… or even seeing a session that didn’t work would’ve been cool. Or how to redirect clients. I think I heard that a lot through the clinic too.” (P194)

Barriers and Facilitators of Course Completion. Many providers reported motivation to complete the training course and half reported it would positively impact their work. However, approximately 31% of providers reported feeling apprehensive towards training due to: (1) a lack of communication of the rationale for training; and (2) uncertainty about how OAAT services aligns with differing clinical populations, such as children and youth. Addressing these concerns may provide opportunities for an increase in uptake among providers, as knowledge on SC2.0 and OAAT services alone will not result in behavioral change. Thirty eight percent of providers cited that protected time in their schedule made it easier to complete the course without distraction. Moreover, some providers noted that the modularized layout of the course enabled them to complete the training at their preferred pace.

Suggestions for Future Course Development. A minority of providers proposed incorporating more concrete, application-based questions into the training course, and highlighted the potential benefit of including sample diary responses so providers could determine if they correctly applied course content:

“The only thing was that you would write what you would say [as your diary response], and then [the trainers] didn’t do anything with that after … Is [my entry] right? Is it wrong? Is it not? And then it’s your own diary, but it’s probably a little wrong. So I’m like, I’m not printing that– It’s probably all wrong.” (P193)

Perceptions of live OAAT course content

Online training in OAAT services was also complemented by an in-person training. Half of providers who completed interviews commented on the live training in OAAT therapy. Providers appeared to have mixed opinions on the value added by live trainings. Few providers who serve adult populations indicated that the live training mirrored the online training in terms of content, making it repetitive, while some providers who serve child and youth populations found the live training to be engaging and well-organized. Moreover, some child and youth providers enjoyed the collaborative, interactive training components (e.g., use of online breakout rooms), and felt that the live training complemented the online training course.

Discussion

The province of NB conducted a provincial practice change initiative to implement OAAT services across the province as part of a provincial stepped care model for the delivery of A&MH services. Providers employed in Horizon and Vitalité Health Networks, and school districts across the province received training in SC2.0 and OAAT services as part of this initiative. To understand the impact of training on providers’ knowledge, confidence, and attitudes pertaining to SC2.0 and OAAT services, a mixed methods approach was undertaken. Approximately 400 providers participated in this research initiative, with 28 completing interviews. The results indicated that providers were knowledgeable in SC2.0 and OAAT services, and perceived both training courses positively.

Questionnaires were developed to quantify factual and applied knowledge of the key principles of SC2.0 and OAAT services and then further refined using item-response theory. Knowledge of SC2.0 was 67% following completion of the asynchronous training course in SC2.0, representing a change of practical significance [19] with room for focused improvement. Although providers demonstrated a good fund of knowledge in key SC2.0 concepts before course completion, training resulted in a modest, but significant, increase with an effect size that met the criteria for minimal practical significance [19]. Thus, it appears that the SC2.0 training course added some practical knowledge of SC2.0 that reliably built on the intuitive fund of knowledge that providers in A&MH services obtained during their careers. Knowledge gains in SC2.0 following training was consistent with previous research reviews, which indicated that online training and education meetings (e.g., courses) where education was a primary component of the intervention resulted in a probable slight improvement in compliance with desired practice when compared with no intervention [20]. These findings corroborate current results that online training courses are a suitable medium to facilitate provider learning, although likely inadequate to produce clinical practice change alone [21]. This research project identified aspects of the courses that may help build skill (e.g., use of knowledge checks) as well as factors that may need to be addressed to help facilitate the transition from knowledge to practice change (e.g., rationale for training, how training can be used with specific clinical populations). As part of the implementation and communication plan, several information sessions briefly describing SC2.0 were delivered to A&MH providers across the province to create initial awareness of the upcoming change process, which may have influenced providers’ knowledge level prior to training. However, additional emphasis on the role of OAAT services may be needed prior to engaging in training given some apprehension among providers.

Of interest, items with the greatest increase in accuracy from pre- to post-training pertained to data-informed decision-making, and the representation of resources along the continuum of care. This suggests that courses were particularly effective at improving provider understanding of measurement-based care, and the care continuum. Similarly, providers demonstrated 76% correct responses on the questionnaire pertaining to knowledge of OAAT services following completion of the asynchronous training course. Unfortunately, change in knowledge following training in OAAT services could not be measured given that the questionnaires originally developed were not parallel. However, based on post-course knowledge, most providers appeared to understand the key principles of OAAT therapy, along with the evidence supporting it, and also appeared knowledgeable regarding the role of setting realistic expectancies. This suggests that, despite room for continued learning, providers understand many facets of OAAT services that are essential to its effective delivery.

Courses in SC2.0 and OAAT services were associated with favorable attitudes towards these models of care. Providers agreed that the SC2.0 model and OAAT services were acceptable, appropriate, and feasible to implement in their practices. Moreover, they agreed that SC2.0 and OAAT services would be effective for addressing concerns that they typically see in clinical practice, and with minimal opportunity costs. Providers reported modest self-efficacy to enact SC2.0 following training with 10% reporting use in clinical practice post training and an additional 56% planning to incorporate it into their practice. Just over 5% of providers indicated no plan to use SC2.0, suggesting that additional efforts and resources are needed to help shape the organizational culture and rationale for this practice shift, the latter of which was highlighted through interviews. The results from this study may have important implications for facilitating provider behavior change to support the adoption of SC2.0 and OAAT services in practice. The training courses had a beneficial influence on knowledge, outcome expectations, and self-efficacy, all of which are thought to be important psychological capabilities that facilitate the initiation of behavior change [3, 26]. Psychological capabilities, such as outcome expectancies, are thought to have an important influence on motivation. A substantial number of providers endorsed their readiness to implement SC2.0 into practice, suggesting that the training courses may have increased reflective motivation to adopt a new practice, likely due to consistency of practices with professional goals and values [41]. A recent systematic review of 18 studies reported that knowledge, self-efficacy, and outcome expectancies were among the factors that most reliably influenced uptake of evidence-based treatments among mental health care providers [7], further highlighting the importance of training courses such as those evaluated in the present investigation. The acceptability of training courses in the current study adds to the literature on eLearning within healthcare and provides support for the use of asynchronous training of healthcare providers working within A&MH settings, as effectiveness studies of asynchronous eLearning programs have focused on aspects of physical health to date [2].

The goal of these training courses was ultimately to encourage the adoption of a complimentary clinical practice among providers. One way in which this was thought to be achieved was through incorporating instruction on how to perform the desired behavior, a behavior change technique which helps foster skill development [11]. Although both training courses provided general instruction on how to use SC2.0 and OAAT services in clinical practice, many providers’ responses suggested that more specialized instruction would improve the implementation of these skills into their practice, which might lead to sub-optimal self-efficacy following training. It may also be beneficial to include graded tasks to help facilitate learning of the skill being taught [11]. A graded task approach was employed in both of the training courses by having providers answer a multiple-choice question following the presentation of a concept, and then completing journal entries as a way to tie the various content areas together. Behavioral practice and rehearsal can also aid skill development. Indeed, a link between behavioral practice and rehearsal, and skill development was reported in a literature synthesis study of 277 articles [11], with 95% consensus of this link among at least 20 international behavior change experts [14]. Although practice and rehearsal were not built into the current training courses, the virtual OAAT services training sessions provided an opportunity for practitioners’ consolidation of learning, and was used to enhance skills in delivering OAAT services.

With respect to the COM-B model of behavior change, creating behavior change requires capability (C), opportunity (O) and motivation (M) [58]. The COM-B model was used to support the implementation team in maximizing the role of training courses in promoting behavioral change. Interviews revealed that delivering content through multiple modalities (i.e., written text, audio and video) fostered engagement and helped providers commit course information to long-term memory. Multimodal content represents a psychological capability acting on reflective motivation. Conversely, some providers said that the amount of course content was overwhelming, a perception which may have reduced motivation and engagement. Providers reported physical opportunity barriers, including time constraints and being pulled away on differing work-related matters. Time constraints have similarly been reported as a barrier to implementing online learning programs with post-graduate medical trainees [44], and with health professionals [36]. Although the layout of current courses was adaptive in that it could be completed at the providers’ own pace, incorporating suggested break points may help coordinate course completion with other work responsibilities. The technological challenges, indicative of a psychological capability within the COM-B model, was also a common barrier to course completion, as often noted in web-based trainings [4]. Providers reported that the course content was simplistic and may not have represented the complexity of clients seen in their practices, despite reporting that SC2.0 and OAAT services would be effective in addressing client concerns. Training cases were developed to reflect diverse mental health concerns across differing ages often seen in community A&MH services, but may not have depicted the complex comorbidities often presenting for services within the public system. The perceived ease of content may have influenced reflective motivation and contribute to interpreting greater psychological capability through relative advantage. Addressing these concerns in the future by including advanced cases or more experiential activities might provide insight into how SC2.0 and OAAT services fits within their scope of practice, as well as increase providers’ efficacy, as experiential practice has been shown to be helpful to trainees [1]. Even so, these training courses were considered acceptable and appropriate by this study’s providers. Consideration of other factors within the COM-B behavior change model, including opportunities provided by the environment and social opportunities, could also provide insights on the translation and sustainment of this training into practice. Recent reviews have reported that opportunities, such as perceived lack of time by providers, may impede behavioral change [33], which was similarly cited as a barrier in the current study. In addition, it may be beneficial to incorporate social opportunities into the training, such as modelling, as this can also support behavior change [12].

Limitations

The effect of courses on knowledge and attitudes were evaluated using a single-cohort pre-post design, and future research is needed to evaluate the specific impacts of online training using designs that evaluates all measures pre- and post, and controls for time (e.g., waitlist control), expectancy effects (e.g., attention control) and modality of delivery (e.g., in-person vs. online). Similarly, knowledge of OAAT therapy was only reported on at post-course completion given that the questionnaire designed to measure knowledge of OAAT therapy at pre-course was not deemed comparable. Future research is needed to evaluate the effectiveness of this training course on change in knowledge from pre- to post-course completion. It is also important to note that strong evidence is not yet available to establish the validity of knowledge acquisition questionnaires despite questionnaires being developed through consultation with subject-matter experts. Likewise, the measures used to understand providers’ attitudes were self-report which are well-known to be prone to response bias. Although no measure of response tendency was included, triangulation between the surveys and interviews with providers helps reduce the potential of response bias on results. It is also important to note that convenience sampling was used which may introduce selection bias (i.e., providers who took part in optional research and completed interviews may have been more motivated to make change to their practice). More motivated providers may also be more inclined to implement SC2.0 and OAAT services into their practice. Moreover, providers completed surveys before the training and relatively soon after completing the courses and virtual training. Unfortunately, the current study did not include longer-term assessment of outcomes, thus the robustness of training in the long run is yet to be determined. Similarly, since data on the uptake of SC2.0 and OAAT services prior to course completion was not collected, some providers may have been using principles of SC2.0 and OAAT services prior to course completion, as was noted in interviews. This may have inflated their baseline knowledge; thus, the increase in knowledge acquisition of SC2.0 in this study may be an underestimate of knowledge change among providers who have already made this practice change. Similarly, pre- and post-test SC2.0 knowledge acquisition questionnaires were delivered in a specific order and the potential for carry-over effects cannot be ruled out. Also, the providers’ perceptions of applicability of skill development to their clinical practice settings varied, along with the time between course completion and implementation, which may have impacted providers’ perceptions of the training courses. Lastly, although the goal of the current implementation involved a provincial practice change, the current study measured knowledge change and did not measure behavior change in practice. This is important when interpreting the current findings given that outcomes of interest (i.e., knowledge, self-efficacy) are necessary, but not sufficient, to change behavior. Although many other factors will ultimately determine the success of this implementation, it is a positive step that the training courses introducing providers to these changes were generally positively received. Given that multi-component and active trainings tend to produce more favorable results [21], future research could evaluate the effect of these training courses in conjunction with face-to-face in-person or virtual workshops, similar to the live training providers tasked with implementing OAAT received.

Conclusion

Online asynchronous training courses may help facilitate the acquisition of knowledge of SC2.0 and OAAT services, and result in favorable attitudes and outcome expectancies towards these practices. Although this study focuses on the implementation of SC2.0 and OAAT services within NB’s A&MH Services, the components of the courses that made it effective, as well as the aspects that increased course acceptability and feasibility, have been cited in the literature pertaining to implementation science, and can be extended to similar implementation projects. Moreover, the rich information collected through the addition of qualitative interviews highlighted the importance of obtaining the perspective of those meant to implement the innovation, as this provided important context for the quantitative findings. Lessons learned through qualitative feedback may also benefit future implementation projects, as these opinions are likely linked to providers’ motivation and support for implementing professional practice changes, which are crucial components when determining the success of an implementation.