Assessing the clinical robustness of digital health startups
With special thanks to co-author Simon C. Mathews, MD, and researchers Veeraj Shah and Shannon Powelson.
By many measures, digital health has boomed across the past decade—so what clinical impact is there to show for it? It’s often difficult to quantify the outcomes of the digital health industry at large because most companies are still privately held and there are no universal standards for outcomes. This challenge leaves innovators, enterprises, and investors alike trying to understand whether impact matches the hype.
To shed light on this question, we analyzed the clinical trials, regulatory filings, and claims (clinical, economic, and engagement) made by clinically-focused startups1 to see if there are clear patterns.
The short answer: robust clinical evidence is not evenly distributed across digital health. Of the companies we looked at, we found that a sizable minority of those companies (20%) represented an outsized share of the trials and regulatory filings. Additionally, claims are not well correlated with evidence—startups that make more clinical claims are not always those with more trials or regulatory filings. Across the board, average numbers of clinical, economic, and engagement claims are low.
We recently published these results in JMIR—you can check out the full paper here. Read on for the key takeaways.
How we assessed claims and clinical robustness2
Our analysis included the 224 companies offering prevention, diagnosis, or treatment solutions—those that we would expect to have clinical evidence—in Rock Health’s Venture Funding Database3. We examined two key variables for each company:
- Clinical robustness score: we wanted to use the most objective measures available, so we looked at the total number of FDA filings (510(k), De Novo, and premarket approvals) plus the total number of clinical trials on ClinicalTrials.gov
- Claims: we aimed to get an understanding of the claims a company is making about its product, so we used the number of unique quantitative statements about product outcomes made on a company’s website; categorized as clinical (e.g., reduction in HbA1c), financial (e.g., cost savings per patient), or engagement claims (e.g., number of active users)
The goal of this approach was to develop a high-level picture of the distribution of and relationships between clinical evidence, claims, and total venture funding across digital health.
Clinical robustness and claims across the digital health landscape
We would have expected (or hoped) to see a positive relationship between the number of clinical trials and regulatory filings a company has and a company’s total venture funding and the number of clinical claims it makes publicly. Yet we found no clear relationship between clinical robustness and total venture funding or care continuum (prevention, diagnosis, or treatment) phase. There was also little evidence of a relationship between the number of claims a company makes and its clinical robustness score.
Let’s unpack this. Here are six key takeaways from our analysis:
1. Many clinically-focused digital health companies did not have any clinical trials or regulatory filings
Nearly half of the companies in our sample (44%) had a clinical robustness score of 0. This highlights a potentially significant gap in evidence, which wasn’t easily explained by company age or venture funding (i.e., you might try to explain this by assuming that it takes a company a long time and a lot of money to be able to build up an evidence base, but many younger companies and less well-funded companies had high clinical robustness scores).
2. A sizeable minority of companies had an outsized share of the clinical trials and regulatory filings
Forty-four companies, representing 20% of the total, had a clinical robustness score (clinical trials plus regulatory filings) of 5 or more. This cohort shows that digital health startups have an opportunity to meaningfully differentiate by building up their evidence base.
3. Clinical robustness varies significantly across therapeutic areas
Digital health solutions targeting several therapeutic areas such as cardiovascular disease and diabetes have relatively high levels of clinical robustness, while many other areas have not yet produced significant numbers of trials or regulatory submissions. Some of these more clinically robust therapeutic areas have been longtime hotbeds for digital health investment—startups are likely building up their evidence bases in order to differentiate in increasingly crowded markets.
4. The average number of claims made by digital health startups is low
Clinically-focused digital health companies make an average of 1.3 public claims. On average, this looks like 0.5 clinical claims, 0.4 economic claims, and 0.4 engagement claims per startup. Such low numbers could reflect either a desire to keep this data private or a lack of evidence of impact. Additionally, we identified 32 companies that had one or more clinical claims and a clinical robustness score of 0. This suggests a disconnect between marketing and the indicators of clinical robustness we’ve laid out among a minority of digital health startups.
5. Clinical claims are not correlated with clinical robustness
We did not find a positive relationship between the number of clinical claims a company makes and the number of clinical trials and regulatory filings it has. There are two main implications from this finding: in some cases, companies may be making claims that are not grounded in robust clinical evidence; in other cases, companies with robust clinical evidence may be under-selling their impact publicly.
6. Companies selling to employers had higher clinical robustness and claims compared to other customer types
Startups targeting employers had the highest average clinical robustness scores (3.1, which is 15% more than providers, the next highest segment with an average score of 2.7). Employer-focused startups also make the most claims, with an average of 1.3 clinical, 0.9 economic, and 1.0 engagement claims per company. These findings are further evidence of a competitive and increasingly savvy employer market—unsurprising, given the rising cost burden of health care on employers and employees.
Implications for digital health leaders
Our findings underscore the need for more robust evidence of outcomes in digital health. If digital health is to become synonymous with health, then its standards for evaluation and evidence need to be commensurate with conventional healthcare solutions. Investors and buyers of digital health solutions need to carefully evaluate the evidence base upon which startups make their claims. Innovators should seize the opportunity to stand out by rigorously demonstrating the impact of their solutions. They should also think carefully about the mix of clinical, economic, and engagement claims that will resonate with their target audience.
Our hope is that these findings are a signpost for leaders across the digital health ecosystem. The startups that have substantial claims of outcomes backed up by robust clinical evidence should be a model for how to meaningfully impact healthcare.
Footnotes
- We defined clinically-focused startups as those with a value proposition of prevention, treatment, or diagnosis of disease.
- There are, of course, tradeoffs with our analysis approach. For example, some companies may publish their evidence in an academic journal without registering a clinical trial or filing with the FDA. We decided to only include clinical trials and FDA filings because we observed they are more likely to be directly tied to product outcomes. Journal articles frequently report on results relating to a solution’s underlying mechanism (e.g., virtual cognitive behavioral therapy, rather than a company’s specific virtual CBT product). Therefore, while our results are not completely precise we believe they are directionally accurate. Check out the Limitations section of our published paper for additional details on the nuances of the approach.
- Includes digital health companies that raised funding rounds of >$2M through December 2020.