Updated: Aug 29, 2022
The August 2022 Online first edition of the American Journal of Lifestyle Medicine (AJLM) published our new paper about health and well-being coaching research. This issue contains our article (Sebastian Harenberg and Joel Edman are my co-authors) entitled “A Rubric to Assess the Design and Intervention Quality of Randomized Controlled Trials in Health and Wellness Coaching.” While the title is a mouthful, read on. We developed a rubric (a checklist of questions) to examine the quality of a research study from two perspectives: how well the study was designed and how well the coaching intervention was planned and implemented.
In all, the rubric we created contains 28 items. Of these, 15 address study design criteria and 13 examine coaching intervention criteria. The questions related to study design assess items such as participant recruitment, participant allocation to group, exclusion criteria, sample size, control group management, outcome measures, and statistical analyses. The rubric questions are scored 0-1 or 0-2, and the greater the total score the better the assessment of study design quality.
For studies of health and well-being coaching, it is also essential to examine the structure and implementation of the coaching intervention. The rubric questions addressing the quality of coaching intervention ask about items such as coach training, certification, and experience. Questions on the coaching intervention include coaching session frequency, duration, and program length (in months). Finally, the rubric also calls for a description of the coaching process, coach quality checks during the research project, and client adherence.
Using the rubric, a given research paper might potentially score high on study design and low on coaching intervention, or vice versa. The best research papers score high on both dimensions. Check out the full-length article, which is cited below and is available open access from AJLM Sage publishing website.
Once the rubric items were selected, they were reviewed by research design and coaching experts. The items were also crossed-scored on 29 articles by the study authors to establish reliability. In other words, all three authors read 29 health and well-being coaching research papers and scored those articles using the rubric. An intraclass correlation of .85 (ICC = .85; CI95% - .68-.93) indicated very good levels of agreement between the three of us (study authors: Harenberg, Sforzo, Edman). These processes of gathering expert opinion and checking reliability provide a measure of validity for the new rubric. We have encouraged other authors
to use the rubric in their work and further validate it as a useful tool for assessing health and well-being coaching research.
In this paper, we applied the rubric to randomized controlled trials (RCTs) studying patients with Type 2 diabetes (T2D; n = 11 articles) or RCTs studying patients with obesity (n = 18 articles). The total rubric score for the T2D studies turned out to be slightly greater than found for the obesity studies – mainly because the scores for coaching intervention design were higher in the T2D research examined. This serves as a caution for readers of those obesity studies and as a reminder to future researchers of health and well-being coaching when applied in an obese patient population. It is important to carefully design and implement the coaching intervention. Only with a well-designed, described, and implemented coaching protocol can we optimize and best understand the effects of coaching. Those papers with carefully designed and presented coaching methodologies contain the research propelling the field forward and informing us how to apply the best coaching strategies in practice.
We believe the most valuable contribution of the new rubric and this publication lies in application for future research. Rather than thinking of the rubric as an evaluation tool, it is best thought of as a roadmap for designing future health and well-being coaching research. When planning a project, researchers can look at the rubric and check off criteria to maximize rubric score and design the best possible study. We know from our Compendium work (2,3) the quantity of health and well-being coaching research is rapidly expanding. It is important to the development of health and well-being coaching profession that the quality of coaching research is continuously improved going forward.
This article and the related explanation may seem like way too much information for the average health and well-being coach – why would they need to know how to assess research? Maybe they don’t! However, the typical coach benefits from understanding the status of the existing research and having an appreciation that this base of knowledge is being examined for quality. With such knowledge, a practicing coach can confidently apply state of the art methods and techniques with their clients. We expect application of the rubric will improve the future of health and well-being coaching research and thereby improve the standards of coaching.
Harenberg S, Sforzo GA, Edman J. A Rubric to Assess the Design and Intervention Quality of Randomized Controlled Trials in Health and Wellness Coaching. American Journal of Lifestyle Medicine. August 2022. doi:10.1177/15598276221117089
Sforzo GA, Kaye MP, Todorova I, et al. Compendium of the Health and Wellness Coaching Literature. American Journal of Lifestyle Medicine. 2018;12(6):436-447. doi:10.1177/1559827617708562
Sforzo GA, Kaye MP, Harenberg S, et al. Compendium of Health and Wellness Coaching: 2019 Addendum. American Journal of Lifestyle Medicine. 2020;14(2):155-168. doi:10.1177/1559827619850489