A systematic search of the Apple iTunes store was conducted on September 19, 2013, following the PRISMA guidelines for systematic literature reviews [18 (link)]. An exhaustive list of mental-health related mobile apps was created. The following search terms were employed, “Mindfulness” OR “Depression” OR “Wellbeing” OR “Well-being” OR “Mental Health” OR “Anger” OR “CBT” OR “Stress” OR “Distress” OR “Anxiety”.
App inclusion criteria were: (1) English language; (2) free of charge; (3) availability in the Australian iTunes store; and (4) from iTunes categories, “Health & Fitness”, “Lifestyle”, “Medical”, “Productivity”, “Music”, “Education”, and “Utilities”. The category inclusion criteria were based on careful scrutiny of the titles and types of apps present in those categories.
There were 60 apps that were randomly selected using a randomization website [19 ]. The first ten were used for training and piloting purposes. There were two expert raters: (1) a research officer with a Research Masters in Psychology and two years’ experience in mobile app development, and (2) a PhD candidate with a Masters degree in Applied Psychology and over nine years information technology experience, that trialled each of the first 10 apps for a minimum of 10 minutes and then independently rated their quality using the Mobile App Rating Scale (MARS). The raters convened to compare ratings and address ambiguities in the scale content until consensus was reached. The MARS was revised based on that experience, and the remaining 50 mental health and well being related apps were trialled and independently rated. A minimum sample size of 41 is required to establish whether the true interrater reliability lies within .15 of a sample observation of .80, with 87% assurance (based on 10,000 simulation runs) [20 (link)]. The sample size of 50, therefore, provides substantial confidence in the estimation of the interrater reliability in the current study. Data were analyzed with SPSS version 21 (SPSS Inc, Chicago, IL, USA). The internal consistency of the MARS quality subscales and total quality score was calculated using Cronbach alpha. This indicates the degree (correlations) to which items measuring the same general construct produce similar scores. Interrater reliability of the MARS subscales and total score was determined by the intraclass correlation coefficient (ICC) [21 (link)]. This statistic allows for the appropriate calculation of weighted values of rater agreement and accounts for proximity, rather than equality of ratings. A two-way mixed effects, average measures model with absolute agreement was utilized [22 (link)]. The concurrent validity of the MARS total score was examined in relation to the Apple iTunes App Store average star rating for each app (collected from the Apple iTunes App Store on September 19, 2013).
App inclusion criteria were: (1) English language; (2) free of charge; (3) availability in the Australian iTunes store; and (4) from iTunes categories, “Health & Fitness”, “Lifestyle”, “Medical”, “Productivity”, “Music”, “Education”, and “Utilities”. The category inclusion criteria were based on careful scrutiny of the titles and types of apps present in those categories.
There were 60 apps that were randomly selected using a randomization website [19 ]. The first ten were used for training and piloting purposes. There were two expert raters: (1) a research officer with a Research Masters in Psychology and two years’ experience in mobile app development, and (2) a PhD candidate with a Masters degree in Applied Psychology and over nine years information technology experience, that trialled each of the first 10 apps for a minimum of 10 minutes and then independently rated their quality using the Mobile App Rating Scale (MARS). The raters convened to compare ratings and address ambiguities in the scale content until consensus was reached. The MARS was revised based on that experience, and the remaining 50 mental health and well being related apps were trialled and independently rated. A minimum sample size of 41 is required to establish whether the true interrater reliability lies within .15 of a sample observation of .80, with 87% assurance (based on 10,000 simulation runs) [20 (link)]. The sample size of 50, therefore, provides substantial confidence in the estimation of the interrater reliability in the current study. Data were analyzed with SPSS version 21 (SPSS Inc, Chicago, IL, USA). The internal consistency of the MARS quality subscales and total quality score was calculated using Cronbach alpha. This indicates the degree (correlations) to which items measuring the same general construct produce similar scores. Interrater reliability of the MARS subscales and total score was determined by the intraclass correlation coefficient (ICC) [21 (link)]. This statistic allows for the appropriate calculation of weighted values of rater agreement and accounts for proximity, rather than equality of ratings. A two-way mixed effects, average measures model with absolute agreement was utilized [22 (link)]. The concurrent validity of the MARS total score was examined in relation to the Apple iTunes App Store average star rating for each app (collected from the Apple iTunes App Store on September 19, 2013).