IVI economists and staff recently published this open-access article describing specifically how IVI puts stakeholder engagement at the center of value assessment. Read about the engagement process IVI is using to build patient-centricity into its major depressive disorder (MDD) health economic model below or on download a copy from PharmacoEconomics here.

Putting Stakeholder Engagement at the Center of Health Economic Modeling for Health Technology Assessment in the United States
By Richard Xie, Erica deFur Malik, Mark Linthicum, and Jennifer Bright, May 13, 2021

While evidence generated from health economic (HE) models is being used more commonly in health technology assessment (HTA) in the US, it is not consistently adopted by different stakeholder groups or across therapeutic areas. We hypothesize that actively engaging with multiple stakeholder groups throughout the model development process may result in models more widely considered by decision makers. To test this hypothesis, the Innovation and Value Initiative has launched a modeling effort to build an open-source HE model focusing on the disease state ‘major depressive disorder’. A 20-member advisory group has been formed with representatives from patients, employers, clinicians, innovators, payers, and researchers to guide the model development process. While this effort is still in the early stages, the ongoing stakeholder engagement effort has yielded valuable insights that inform the model design. We have also identified several challenges to implementing this new approach. Our early findings suggest that the stakeholder engagement approach to HE model development has the potential to improve HTA in the US.

Under pressure to curb rising health care costs in the US [1], private and public payers, policymakers, clinical societies, and other stakeholders have increasingly turned to formalized health technology assessments (HTA) to assess the relative value of health interventions [2, 3, 4]. Health economic (HE) modeling [5] is used to predict and compare the costs and health outcomes of interventions and is commonly utilized to inform HTA outside the US [6, 7]. Although evidence generated from HE models is being used more commonly in coverage and reimbursement deliberations in the US, it is not consistently adopted by different stakeholder groups or across therapeutic areas [3, 8, 9, 10].

Experts from multiple disciplines have called for a more open and inclusive process in the development of HE models and HTA frameworks, and they emphasize the importance of broader inclusion of patient and employer perspectives [2, 11, 12]. We hypothesize that actively engaging with multiple stakeholder groups throughout the HE model development process—and particularly during initial planning and model design phases—may result in more relevant and useful models.

To implement such an approach, the Innovation and Value Initiative (IVI) has established a process that empanels an advisory group in the disease state of interest. These advisors provide guidance on key modeling considerations, identify data sources, and generate use cases throughout different phases of the modeling process. As participants in the HE modeling process (Table 1), the advisory group guides modeling decisions from the beginning (Fig. 1).

In the US, HE models are often developed by for-profit economic consultants, academic researchers, or health economics and outcomes research experts within life sciences companies [13, 14]. Key model assumptions, specifications, and data inputs are often determined by technical experts, and access to key assumptions or data inputs in the HE models can be limited due to intellectual property concerns [13, 15]. Additionally, HE models seldom consider or incorporate direct patient perspectives and inputs, and they might fail to capture real-world patient experiences valued by decision makers [11, 16]. Perspectives from external stakeholders are often sought ex post and used as contextual inputs or considerations rather than being consistently incorporated into models. The lack of transparency about key modeling decisions and limited stakeholder participation in the model development process can lead to misalignment between model design and decision needs of end users [13]. These limitations cause relevant decision makers to question the credibility and relevance of cost-effectiveness estimates and, ultimately, the ability of models to inform HTA in meaningful ways [17, 18].

In July 2020, IVI launched its third Open-Source Value Project initiative to build an HE model to support HTA for major depressive disorder (MDD). We chose MDD due to its prevalence, significant societal burden, and broad interest among stakeholders who are looking for better treatments and more cost-effective resource allocation [19, 20]. As a first step, IVI convened a 20-member Advisory Group¹ (AG) consisting of patients (n = 5), employers (n = 5), clinicians (n = 5), innovators (n = 3), payers (n = 2), and researchers (n = 2) to weigh in throughout the modeling process.

To date, the AG has provided feedback on the model’s conceptual framework [21] as well as the model design, including identifying (1) decision needs (model objectives); (2) gaps in existing economic models; (3) data sources for model inputs; and (4) the most appropriate analytic framework when multiple approaches exist. Feedback was provided through group meetings, surveys, emails, and individual discussions. In prioritizing feedback from the AG, we solicited additional insights from patients and employers, two traditionally under-represented stakeholder groups in the HE modeling process [11, 12, 22].

While still in the early stages, this ongoing stakeholder engagement has already yielded valuable insights, recommendations, and resources (Table 2). This input is meaningfully impacting model design in three key areas discussed here.

AG input has highlighted important decision contexts and factors considered by end users that help inform the model specification. For example, the Hamilton Depression Rating Scale and Montgomery-Asberg Depression Rating Scale are the clinical instruments most widely used to define health states in existing models, but stakeholder input indicated that the Patient Health Questionnaire-9 is more commonly used in clinical practice and increasingly in clinical trials and observational studies. In discussions about potential model structure, all stakeholder groups emphasized that care setting (primary vs specialty care) is an important variable that will impact prescribed treatments and their effectiveness. Therefore, we are reviewing additional literature to evaluate how care settings will impact patient trajectory in model simulations. Patient groups also emphasized the importance of prioritizing data inputs from samples more representative of the MDD population in the US.

AG members have shared preliminary, and sometimes proprietary, findings and data from ongoing research projects that can be used as model inputs. For example, a researcher shared data on caregiver burden that can be a direct model input [23]. Employer, patient, and researcher representatives highlighted the need to measure different types of productivity impacts (e.g., absenteeism, presenteeism) and provided suggestions on data sources and literature.

AG members who are potential end users are interested in engaging in the modeling process and applying the model in their decision making. Despite their differing decisions needs, many stakeholders emphasized similar considerations in model design. These included reflecting patient heterogeneity in treatment experience, facilitating subgroup analyses, and fully measuring different types of productivity loss (Table 2). Moreover, AG members also proactively proposed use case development. For example, one stakeholder suggested that the model design would allow her organization to assess the burden of MDD without active treatments.

The discussions also identified key areas of consensus across stakeholder groups regarding the purpose, design, and use of existing HE models (Table 2 and “Appendix 2”, see ESM). Suggestions included (1) models should be flexibly designed to accommodate emerging and evolving data; (2) models should incorporate real-world data beyond just clinical trials; (3) entire treatment pathways should be modeled rather than just individual treatment comparisons; and (4) models should incorporate patient perspectives and inputs.

Eight months in, we have identified several challenges to implementing this new approach. First, eliciting and incorporating feedback from diverse stakeholders to inform HE model development requires experience with multiple methodologies (both qualitative and quantitative), as well as creativity and flexibility in their use. For example, roundtable discussions required focused questions and well-identified goals to generate specific feedback on research questions. Small-group discussions were sometimes more appropriate for engaging those with specific expertise. Qualitative AG input sometimes needed to be supplemented with additional literature reviews to identify appropriate methods—to understand the measures and estimates of presenteeism, for example. Finally, to ensure inclusive input, discussion materials must account for varying levels of clinical or technical knowledge. While time consuming, these steps are crucial for soliciting useful input.

This approach requires foresight to address differing priorities. Through email communications, meeting summaries, and discussions, we have worked to ensure that all viewpoints are documented and acknowledged, and that methods for prioritizing input are transparent. We are committed to communicating with the AG about our decision-making process, and plan to prioritize considerations with the broadest stakeholder buy-in, inputs from traditionally under-represented stakeholders, and inputs that are most feasible to implement [24].

Finally, engaging a large, diverse stakeholder group increases the scope and complexity of model development. The research team leading the process must invest considerable time and resources to planning and managing engagement activities, synthesizing conflicting viewpoints, and determining how to incorporate feedback into the model design. This may be particularly challenging for those lacking engagement experience. Refining the methodology through further implementation may increase efficiency, however.

Advisory Group members must also commit their time and energy to an ongoing process, and the research team may need to account for missing stakeholder perspectives. Some key stakeholder groups were not able to participate in our Advisory Group, namely pharmacy benefit managers, benefit consultants, and government agencies (e.g., Centers for Medicare & Medicaid Services). As these are key decision makers and potential users, we intend to use public comment periods and individual outreach to solicit feedback.

As the next phases of the MDD development transpire, we will continue to evaluate the process with the AG. Specifically, we will assess how this engagement has impacted model design, whether it will increase the consideration of HE models by different stakeholders, and how well it facilitates open dialogue and trust across stakeholder groups.

As US policymakers and payers continue to prioritize a transition to value-based care, IVI is advocating a concurrent evolution of methods to better reflect the inputs and needs of broader stakeholders. Our multi-stakeholder engagement approach to developing an open-source HE model has the potential to greatly improve HTA in the US.

1. “Appendix 1” in the electronic supplementary material (ESM) describes briefly how the AG members were identified and recruited. Please note some members represent multiple stakeholder perspectives. A full list of members of the MDD Advisory Group is available at http://valueresearch.org/wp-content/uploads/2022/06/MDD-Advisory-Group2022.6.13-1.pdf.

1. Chen A, Goldman D. Health care spending: historical trends and new directions [Internet]. Annu Rev Econ. 2016:291–319. https://papers.ssrn.com/abstract=2870849. Accessed 15 Mar 2021 (Annual Reviews Inc.).
2. Sorenson C, Lavezzari G, Daniel G, Burkholder R, Boutin M, Pezalla E, et al. Advancing value assessment in the United States: a multistakeholder perspective. Value Health [Internet]. 2017;20:299–307. https://pubmed.ncbi.nlm.nih.gov/28237214/. Accessed 15 Sep 2020 (Elsevier Ltd).
3. Luce B, Cohen RS. Health technology assessment in the United States. Int J Technol Assess Health Care. 2009;25(Suppl 1):33–41. https://doi.org/10.1017/S0266462309090400.
4. Garrison LP, Pauly MV, Willke RJ, Neumann PJ. An overview of value, perspective, and decision context—a health economics approach: an ISPOR special task force report [2]. Value Health [Internet]. 2018;21:124–30. https://pubmed.ncbi.nlm.nih.gov/29477389/. Accessed 10 Dec 2020 (Elsevier Ltd).
5. Wu EQ, Zhou ZY, Xie J, Metallo C, Thokala P. Transparency in health economic modeling: options, issues and potential solutions. Pharmacoeconomics [Internet]. 2019;37:1349–54. https://doi.org/10.1007/s40273-019-00842-0. Accessed 18 Mar 2021 (Springer International Publishing).
6. Brogan AP, Hogue SL, Vekaria RM, Reynolds I, Coukell A. Understanding payer perspectives on value in the use of pharmaceuticals in the United States. J Manag Care Spec Pharm [Internet]. 2019;25:1319–27. www.rtihs.org. Accessed 11 Dec 2020 [Academy of Managed Care Pharmacy (AMCP)].
7. Beletsi A, Koutrafouri V, Karampli E, Pavi E. Comparing use of health technology assessment in pharmaceutical policy among earlier and more recent adopters in the European Union. Value Health Reg Issues [Internet]. 2018;16:81–91. https://pubmed.ncbi.nlm.nih.gov/30316029/. Accessed 15 Mar 2021 (Elsevier Inc.).
8. Applying cost-effectiveness thresholds to the real world implications on access for medicare beneficiaries [Internet]. 2020. https://www.xcenda.com/insights/phrma-issue-brief-applying-cost-effectiveness-on-access-for-medicare-beneficiaries. Accessed 15 Mar 2021.
9. ICER’s Growing Influence on Payer Decision Making: the impact of ICER assessments on market dynamics and patient access | Xcenda [Internet]. 2021. https://www.xcenda.com/insights/htaq-winter-2020-icer-payer-decision-making. Accessed 15 Mar 2021.
10. Dubois RW. Cost-effectiveness thresholds in the USA: Are they coming? Are they already here? [Internet]. J Comp Eff Res. 2016:9–11. www.futuremedicine.com. Accessed 15 Mar 2021.
11. Diaby V, Ali AA, Montero AJ. Value assessment frameworks in the United States: a call for patient engagement [Internet]. PharmacoEconomics Open. 2019:1–3. https://doi.org/10.1007/s41669-018-0094-z. Accessed 15 Mar 2021 (Springer).
12. Lakdawalla D, Neumann PJ, Wilensky GR, Balch A, Doshi JA, Garrison LP, et al. Health technology assessments in the U.S.—a vision for the future [Internet]. 2021. https://healthpolicy.usc.edu/wp-content/uploads/2021/02/USC_Schaeffer_FutureHTA_WhitePaper-FNL-Spreads.pdf. Accessed 15 Mar 2021.
13. Neumann PJ. Why Don’t Americans use cost-effectiveness analysis? [Internet]. Am J Manag Care. 2004:308–12. https://europepmc.org/article/med/15152700. Accessed 18 Mar 2021.
14. Bell CM, Urbach DR, Ray JG, Bayoumi A, Rosen AB, Greenberg D, et al. Bias in published cost effectiveness studies: systematic review [Internet]. Br Med J. 2006:699–701. https://doi.org/10.1136/bmj.38737.607558.80.
15. Chapman RH, Kumar V. Can we develop sustainable and sharable cost-effectiveness models for value assessment in the U.S. Health Care System? J Manag Care Spec Pharm. 2019;25(5):521–4. https://doi.org/10.18553/jmcp.2019.25.5.521.
16. Perfetto EM, Oehrlein EM, Boutin M, Reid S, Gascho E. Value to whom? The patient voice in the value discussion. Value Health [Internet]. 2017;20. https://pubmed.ncbi.nlm.nih.gov/28237211/. Accessed 15 Sep 2020.
17. Jansen JP, Incerti D, Linthicum MT. Developing open-source models for the US Health System: Practical experiences and challenges to date with the open-source value project. PharmacoEconomics. 2019;37:1313–20. https://doi.org/10.1007/s40273-019-00827-z.
18. Kent S, Becker F, Feenstra T, Tran-Duy A, Schlackow I, Tew M, et al. The challenge of transparency and validation in health economic decision modelling: a view from mount hood. Pharmacoeconomics [Internet]. 2019;37:1305–12. https://doi.org/https://doi.org/10.1007/s40273-019-00825-1. Accessed 15 Mar 2021 (Springer International Publishing).
19. Greenberg PE, Fournier AA, Sisitsky T, Pike CT, Kessler RC. The economic burden of adults with major depressive disorder in the United States (2005 and 2010). J Clin Psychiatry [Internet]. 2015;76:155–62. http://www.psychiatrist.com/JCP/article/Pages/economic-burden-adults-major-depressive-disorder-united.aspx. Accessed 11 Feb 2021 (Physicians Postgraduate Press Inc.).
20. Birnbaum HG, Kessler RC, Kelley D, Ben-Hamadi R, Joish VN, Greenberg PE. Employer burden of mild, moderate, and severe major depressive disorder: mental health services utilization and costs, and work performance. Depress Anxiety [Internet]. 2010;27:78–89. https://onlinelibrary.wiley.com/doi/full/10.1002/da.20580. Accessed 10 Dec 2020 (John Wiley & Sons, Ltd).
21. Squires H, Chilcott J, Akehurst R, Burr J, Kelly MP. A framework for developing the structure of public health economic models. Value Health. 2016;19:588–601.
22. van Voorn GAK, Vemer P, Hamerlijnck D, Ramos IC, Teunissen GJ, Al M, et al. The Missing Stakeholder Group: Why Patients Should be Involved in Health Economic Modelling. Appl Health Econ Health Policy. 2016;14:129–33. https://doi.org/10.1007/s40258-015-0200-7.
23. Lerner D, Lavelle TA, Adler D, Chow W, Chang H, Godar SC, et al. A Population-Based Survey of the Workplace Costs for Caregivers of Persons With Treatment-Resistant Depression Compared With Other Health Conditions. J Occup Environ Med [Internet]. NLM (Medline); 2020;62:746–56. https://journals.lww.com/10.1097/JOM.0000000000001957. Accessed 10 Dec 2020.
24. Harvard S, Werker GR, Silva DS. Social, ethical, and other value judgments in health economics modelling. SocSci Med. 2020;253:112975.

Affiliations: Innovation and Value Initiative, Alexandria, Virginia, USA (Richard Z. Xie, Erica deFur Malik, Mark T. Linthicum, and Jennifer Bright)

Corresponding Author: Correspondence to Richard Z. Xie

Funding: No funding was received for the preparation of this manuscript.

Conflict of Interest: Richard Z. Xie, Erica deFur Malik, Mark T. Linthicum, and Jennifer L. Bright hold the positions of HEOR Manager, Director of Membership, Director of Scientific Communications, and Executive Director, respectively, at the Innovation and Value Initiative (IVI), a 501(c)(3) non-profit research organization whose members include pharmaceutical companies, patient advocacy organizations, and health systems. Richard Z. Xie was previously employed by Analysis Group, an economics consultancy with clients in the life science industry. Erica deFur Malik is the Managing Director of Malik Group, LLC, and consultant for Momentum Health Strategies. Mark. T. Linthicum was previously employed by Precision Health Economics, a health economics consultancy with clients in the life science and insurance industries. Jennifer L. Bright is also President, Momentum Health Strategies.

Ethics Approval: Not applicable.

Consent to Participate: Not applicable.

Availability of Data and Material: Information on the development of the IVI-MDD model is available at http://valueresearch.org/ivi-mdd-value-model/. Additional information is available upon request from the authors.

Code Availability: Not applicable.

Author Contributions: All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by RZX, EFM, and MTL. The first draft of the manuscript was written by RZX and EFM and all authors provided comment and revision on subsequent versions of the manuscript. All authors read and approved the final manuscript.

Below is the link to the electronic supplementary material.

• Supplementary file1 (DOCX 15 kb)

Open Access: This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc/4.0/.

Xie, R.Z., Malik, E.d., Linthicum, M.T. et al. Putting Stakeholder Engagement at the Center of Health Economic Modeling for Health Technology Assessment in the United States. PharmacoEconomics (2021). https://doi.org/10.1007/s40273-021-01036-3.