Patient Safety

To explore the feasibility of an online approach and to evaluate the replicability of panel findings, we convened and asked 4 online panels to define the appropriate use of the term "Continuous Quality Improvement"¹. The QI field is rapidly developing [21 (link)]. Healthcare organizations are increasingly investing in QI approaches, and funders and journals support a growing level of QI research. Major communication challenges have arisen, however, due to lack of consensus around QI terminology use [22 (link)]. For example, two studies may both report the use of "CQI" but define or operationalize it so differently that they might as well report entirely different interventions [23 ]. Achieving improved communication thus requires consensus around key terms and must engage the perspectives of both QI practitioners and more research-oriented stakeholders. In this study, we used online expert panel methods to attempt to engage both stakeholder types.
LR and SSS used their professional networks to invite Institute for Healthcare Improvement faculty, members of the editorial boards from leading QI research journals, evaluators of Robert Wood Johnson Foundation (RWJF) quality programs, and RAND patient safety and QI experts to participate in this study. Participants were asked to nominate other QI professionals and health services researchers. Out of 259 professionals contacted, 119 agreed to participate.
As part of the agreement to participate, we asked participants to self-identify themselves as primarily practitioners, primarily researchers, or both equally. We used stratified random sampling to assign participants to one of two small (n1 = 19, n2 = 21) or two large (n3 = 40, n4 = 39) panels and balance panels with regard to the number of researchers and practitioners. Participants were not informed about the size of their panels or the total number of panels. While participants knew that the study would consist of three phases, consistent with the RAND/UCLA Appropriateness Method manual [3 ], we did not explicitly instruct panelists to develop consensus. The study was determined to be exempt from the IRB review by the RAND's Human Subjects' Protection Committee.
ExpertLens is one system for conducting online expert panels. It was created by an interdisciplinary team of researchers at the RAND Corporation [24 (link)]. It uses a modified-Delphi elicitation structure and replaces traditional face-to-face meetings with asynchronous, unmoderated online discussion boards. The online process used in this study consisted of three phases; each phase was limited to one week. In Phase I, panelists rated 11 features of CQI initiatives on four dimensions, including the importance of a feature for a definition of CQI. The initial features came from earlier consensus work that used a traditional expert panel process [23 ], but study participants could also add other important features they felt were missing. In Phase II, panelists saw their own responses as well as the medians and quartiles of their panel responses to Phase I questions. They also participated in asynchronous, anonymous, and unmoderated online discussions with the same group of colleagues in each panel. Phase II was the feedback phase that allowed panelists to review the panel response by looking at measures of central tendency and dispersion and discuss their ideas anonymously, without being influenced by the status of other panelists [12 ]. In Phase III, panelists re-answered Phase I questions. In the optional post-completion survey, participants rated additional features mentioned in Phase I and answered questions about their experiences participating in the online expert panel.
In line with consensus methods guidelines, the definitions of importance of a particular CQI feature, as well as of the level of consensus, were determined in advance [4 (link)]. We considered a feature to be important for a CQI initiative if a panelist rated it as > 3 on a 5-point importance scale. We also used an a priori definition of consensus. If more than two-thirds (> 66.6%) of panelists agreed on the importance of a particular feature, we argued that consensus was achieved [25 (link)]. We used mean absolute deviation from the median (MAD-M) as a measure of disagreement within panels and treated a reduction in its values between phases as a sign of increased consensus [3 ,26 (link)]. MAD-M is the preferred measure of disagreement in expert panels that has been widely used since 1980s when the RAND/UCLA Appropriateness Method was originally created. It is a good measure of disagreement because it is not affected by extreme observations and measures deviation from the median, a measure of central tendency typically used in consensus development and in this study [26 (link)]. Finally, we used four-way kappa to assess agreement between panels, treating the data as ordinal and using a weight matrix comprising the squared deviations between scores [27 ].

Given that the intent of the Delphi survey was to examine the patient safety systems in the context of a nationally accepted management framework (the Malcolm Baldrige National Quality Award Criteria for Performance Excellence in Healthcare), all study experts were selected using stringent criteria, including knowledge of and/or training in the Malcolm Baldrige Criteria for Performance Excellence in Healthcare, and knowledge and experience in patient safety. The number of experts with such qualifications was fairly limited (n ~ 100) and the sample of Delphi panel participants was small (n = 23).
The sample size for the study was based initially on an empirically selected small sample size (n = 15) and the expected response rate necessary to achieve this sample size. It was critical to consider what response rate was usually obtained in surveys in the particular study area (healthcare quality and patient safety). A survey on the quality of healthcare and the problem of medical errors administered to a large random sample of Colorado physicians, national physicians and Colorado households, revealed response rates of 66% for the Colorado physician sample, 36% for the national physician sample, and 82% for the Colorado household sample [23 (link)]. The psychometric validation process for the Safety Attitude Questionnaire conducted in 160 healthcare sites in the U.S., England and New Zealand obtained a response rate of 67% [24 ]. Sumsion (1998), as discussed by Hasson, Keeney and McKenna (2000), argued that in order to maintain the rigor of the Delphi technique, a response rate of 70% must be maintained [22 (link)]. Based on the healthcare study response rates as found in the literature, it was concluded that for this study a response rate of 70% could be expected. Thus, to obtain at least 15 respondents, the study should begin with 22–23 Delphi panellists, where a sample size of 15 to 23 respondents was considered to be small. Responses were obtained from all 23 experts that had made a commitment to serve on the Delphi panel.

At baseline, participants will be asked to consent to the researcher accessing their mental health clinical file; the researcher accessing details of the participant’s emergency contact or key clinician, consultant psychiatrist, and GP and sharing clinically important information with those individuals to maintain patient safety as necessary during the trial; and the collection of data post intervention and at 6-month follow-up.
Participants in the SafePlan condition will hold their own data on their device until they choose to show it to their mental health professional when they attend the clinic in person. Their data are not automatically shared or monitored, and this is communicated to participants both upon enrollment in the study and through the terms and conditions presented when downloading the app.
A separate participant information and consent form has been developed for use with mental health professionals who will partake in semistructured interviews focusing on the feasibility and acceptability of the intervention and study procedures. All interview data will be anonymized in the final report.

Participants will be asked to provide consent to share clinical data and will have demographic data collected at enrollment. This will include participant age, sex, relationship status, highest level of educational attainment, employment status, type of service (AMHS or CAMHS), and geographic service area (Community Health Organisation 1 & 2). Clinical data collected from the participant’s file will include details of current psychiatric diagnoses, current treatment plan (including medication if relevant), key clinician profession, and information in relation to previous suicidality to supplement participant self-report (suicidal ideation, nonsuicidal self-injurious behavior, and previous suicide attempt). Information regarding previous suicide risk or behavior is necessary to support patient safety and to inform factors that may be considered potential moderators or mediators in a full trial. A description of treatment received in both the TAU and the SafePlan conditions will be captured using the CSRI and clinical file data.