INSIGHTS
Deep Dives
Sean Day
Principal
Sari Kaganoff
Chief Commercial Officer
Rob Ryan
Consulting
July 18, 2022

Stacking up against the clinical robustness index: A primer

With special thanks to co-author Simon C. Mathews, MD, and researchers Veeraj Shah and Shannon Powelson.

Amidst the buzz from our recently published peer-reviewed article on clinical robustness in digital health, one question keeps coming our way. Everyone wants to know how they (or their partners/vendors) stack up. This post will explain how to calculate the clinical robustness score for any startup and understand how it compares to the broader ecosystem.

(Re)-introducing: The clinical robustness index
In order to understand how a startup stacks up, it’s important to think about what their clinical robustness score is, but also helps to see how that compares to their peers in the ecosystem. To make that comparison easier, we’re giving an under-the-hood look at how to calculate the clinical robustness score for any company and compare it to the broader ecosystem.

The clinical robustness index shows the distribution of 224 clinically-focused digital health startups (those categorized as prevention, diagnosis, or treatment in the Rock Health Venture Funding Database) according to their “clinical robustness score.” This score is calculated as a company’s total number of FDA filings plus the total number of clinical trials. We used these two metrics because they are objective, publicly available, indicators of clinical rigor. The index then organizes the companies according to percentiles to make it easier to assess where each score sits relative to other clinically-focused companies in the ecosystem.

We found that companies with a clinical robustness score of 3 or more are in the 75th percentile, and those with a score of 7 or more are in the 90th percentile. Clearly, those digital health companies that have a significant number of FDA filings and/or clinical trials are differentiated from the rest of the ecosystem. It’s important to note, however, that the absolute number does not necessarily correspond to quality, but rather signals the degree of effort.

How should this index be used?
The clinical robustness index provides insight into the distribution of robust clinical evidence across the digital health ecosystem.

The index is a useful starting point for evaluating the evidence base of specific companies or sets of companies, but it’s important to note that there are other elements to consider regarding overall clinical impact. When we conduct deeper analyses of clinical evidence, we typically consider several additional factors:

  • Outcomes/endpoints of clinical trials: The clinical robustness score counts all registered trials equally; getting under the hood of the outcomes achieved in each trial is key to evaluating the strength of a company’s evidence base.
  • Peer-reviewed publications: Peer-reviewed publications undergo a higher level of scrutiny than traditional publications. We excluded them from this analysis because we observed that clinical trials and FDA filings are more likely to be directly tied to product outcomes, whereas publications do not necessarily correspond to a product’s impact (e.g., a description of a process such as cognitive behavioral therapy). In addition, not all publications are equal in standards. This wide variation would have introduced additional subjectivity. A deeper analysis of a company’s evidence base should incorporate academic publications and weigh those results appropriately.
  • Non-clinical outcomes: Clinical outcomes are an important, but incomplete picture of impact for many startups. We also considered economic (e.g., cost savings, revenue generation) and engagement (e.g., number of users, engagement rates) claims to develop a more complete picture of potential impact. Economic and engagement outcomes should be evaluated alongside clinical outcomes in a deeper analysis of a company’s evidence base.
  • Comparison to peers: Our analysis includes a broad set of clinical companies (those focused on prevention, diagnosis, or treatment of disease). The most relevant set of companies to compare clinical robustness scores against are those within the same category (e.g., type of solution, therapeutic area). For example, chronic disease management solutions may be more likely to conduct clinical trials than pursue FDA approval, so the clinical robustness scores of these companies should be evaluated against each other in addition to comparisons against the broader clinical robustness index.

Clinical robustness is an essential differentiator between clinically-focused digital health startups. We have been pleased to see the calls for clinical robustness increase following the publication of our article and look forward to continuing to support this conversation.

Rock Health Consulting works with enterprise companies on digital health strategy and innovation. For more information, reach out to advisory@rockhealth.com.