Editor’s note: This article was produced in collaboration with Direct Relief, Macro-Eyes, and the California Primary Care Association.
Establishing ground truth is the first step in preparing for and responding to a disaster. Health systems throughout the world find themselves unable to verify or accurately estimate ground truth in real-time. California is no different. Direct Relief and the California Primary Care Association principally support community health centers in California and have a shared interest in determining the degree to which these health centers are prepared for the increasing threats posed by wildfires and power outages.
While California’s health centers have adapted and become more resilient to “wildfire season,” over the past few years, California’s crises have increased to include record-breaking heat waves, rolling blackouts, historic wildfires and poor air quality. The California Primary Care Association reached out to health centers in all fire-prone areas and has learned that many had to close operations due to evacuation orders, poor air quality and power outages.
In October 2019, California’s largest electric utility shut down its power grid in 34 of California’s 58 counties, leaving an estimated 2 million people without power and some customers cut off for up to a week. To CPCA’s knowledge, there were approximately 180 health center organizations with over 500 sites in these counties where power was shut off or was scheduled to be shut off. Losing access to power has become a recurring challenge for health centers as California’s utility companies plan to continue intentional power shutdowns during periods of extreme fire danger.
California’s power outages have forced the cancellation of thousands of patient visits at community health centers due to closure. Community health centers that lost power, but were still able to open, found themselves crippled by the loss of power, forced to slash services, close units like dental clinics, and attempt to operate without the computer systems that are the backbone of modern healthcare. Even if a health center stays in operation without power, without electronic health records, the doctors can’t access lab results, records of current prescriptions, schedules for screening tests like mammograms, blood pressure and cholesterol level records, or reports from specialists.
Based on the information gathered to date, health centers with generators could keep their doors open to serve patients as usual. However, it is not yet clear how many health centers experienced challenges with storing medications that require refrigeration, pharmacy refills, or accommodating patients in need of medical services during the Public Safety Power Shutoff (PSPS). Loss of access to health care, including medications, is one of the most least-understood risks from natural or human-caused disasters.
California’s widespread power shutdowns have revealed a hidden weakness in our health care safety net, and PSPS events are likely to become more commonplace as California’s utility companies seek to prevent destructive wildfires during periods of extreme risk.
Deployment
In January 2021, Direct Relief and the California Primary Care Association (CPCA) initiated an engagement with Macro-Eyes to apply machine learning to understand individual health facility and health safety network capacity and readiness for managing Covid-19 vaccinations for the communities they serve – specifically focused on federally qualified health centers (FQHC) and look-alike health centers in California.
The first phase of this project was a rapid two-month deployment of the Macro-Eyes health readiness product, Striata, which uses artificial intelligence (AI) to machine learn the current state of infrastructure at each FQHC while generating data on the catchment population. Striata learns about health infrastructure from publicly available data and data derived through a set of proprietary learning tools that power the platform. Providing insight quickly without accessing proprietary health data enables rapid response and maintains health data safety and sovereignty.
The three leading indicators for site readiness included in the initial deployment were as follows:
- Back-up power capacity. Identify presence of back-up power generation capabilities: solar, battery installation, generator.
- Refrigeration capacity. Identify capacity and reliability of refrigeration.
- Catchment population. Learn socio-economic indicators of site-specific catchment population.
These data-driven insights allow decision-makers to allocate resources and make precision investments in essential infrastructure rapidly. Every dollar and scarce resource available for health – especially during natural disasters and pandemics – can be used effectively.
In the context of health systems broadly and COVID readiness specifically, Striata becomes a resource mapping and investment planning tool where decision-makers can see in real-time which facilities have a higher proportion of the population at risk of not being able to access a vaccine and which sites currently have low resiliency to power outages where investment may improve their ranking.
Results
In January 2021, the status of refrigeration capacity and backup power availability across the California health safety net was known in 8% of facilities. Six weeks into deployment, Striata provided visibility into 100% of federally qualified health centers (FQHC) and lookalike sites in California – each rapidly updating in real-time.
Striata yielded insight into 100% of community health centers included in the master facility list compared to 3-8% at the beginning of the project. This was achieved over two months. Predictive accuracies for refrigeration and backup power were 71-85%% and 68-83.6%, respectively, while catchment population insights yielded results using defined vulnerability criteria and machine-learned predictive vulnerability. The technology continues to learn from new data and improve predictive accuracy the longer it is run.
Striata found that out of 2059 sites, 1258 (61%) were without any form of back-up power (meaning no generator and/or no battery), while 212 (10%) were without refrigeration (170 of those are administrative and 42 are health service delivery sites.) 721 sites (35%) had both refrigeration and some form of back-up power.
These results, overlayed with counties at greatest risk for wildfires, clearly highlight a need to invest in infrastructure in those counties most at risk of fires and associated power outages to ensure uninterrupted services at CHCs serving the most vulnerable populations.
Methods and Uncertainty
Macro-Eyes machine learning models were validated retrospectively using standard train-valid-test data splits. This rigorous machine learning and model development framework ensured that the models were evaluated on data they never encountered during the training phase. Additional prospective validation has been done at a limited set of sites and is actively expanding. This is done through user validation and access to other labeled data newly entered in the public domain.
With every 100 pieces of labeled data – which refers to a data point known to be accurate and can also be referred to as ground truth data – the machine learning system can leverage at least twice that amount of unlabeled data. A 1:2 to 1:3 leverage is expected. At the level of current accuracy – 80% – if the machine learning system predicts ten refrigerators at a site, the actual number of refrigerators will be between 8 and 12. Interpreted another way, 80% of predictions of refrigerators will be off by only 1 out of ten.
Impact
Visibility is resilience. This level of insight has near-term and long-term strategic implications. The health safety net is ever-changing, as are those who seek care. Shifts in population and environment require health systems to adjust the care delivery infrastructure and strategy. System leaders’ ability to respond rapidly to disasters, pandemics, or other influences relies on knowing the reality on the ground. Visibility from Striata can support targeted investment in the health safety net and identify sites ready to deliver essential services dependent on reliable refrigeration (such as a mass vaccination campaign) and locations able to continue offering and providing care services during power outages or times of restricted access. This is especially significant in regions where natural disasters are an annual occurrence.
In California alone, 2020 saw a total of 9,917 wildfires, according to the California Department of Forestry and Fire Protection, or CALFIRE. During the same timeframe, the Gulf Coast saw a total of 12 named storms or hurricanes.
In 2021 the state of Texas experienced a power grid failure resulting in 4.5 million homes and businesses left without power over several days. Together, at the height of the Covid pandemic, these events could have impacted 5087 CHCs and 15,798,766 uninsured people.
With these statistics in mind, Striata quickly becomes a resource mapping and investment planning tool where you can see in real-time which facilities have a higher proportion of the population at risk of not being able to access health services and which sites currently have a low resiliency to power outages where investment may improve their ranking. The results of this process are linked to a user interface where decision-makers at the state and local level can extract data to inform decision-making. The user interface allows decision-makers to quickly see where there may be a need to invest in infrastructure improvement or allocate resources at a given point in time. Users can zoom in to a region and scroll over sites to understand whether that site has a particular infrastructure in place. A user might focus on an area with a lower percentage of eligible population vaccinated and then zoom in to see what the site-level infrastructure is for that region. Users can then click to share feedback or correct information, allowing validation and learning over time.
Vista Community Clinic validated infrastructure readiness via an email exchange between DeeAnne McCallin (CPCA) and Dr. Sue Ann Park (Vista Community Clinic). It was conveyed that Dr. Park understood the importance and value of the data: “This is good information to know, will help to keep COVID vaccine safe if there is a power outage.”
Challenges
The project encountered several challenges. Some of these were addressed, others required more time, and some were circumstantial:
- Although simple and quick to validate data for rural FQHCs, it was a longer process for health centers with multiple sites to validate data. Health center organizations range from having 1 site to more than 20 sites. For those health centers with more sites, it was difficult to navigate the map and having to individually validate the site data. In response to this, the project developed a search and filter functionality to facilitate ease of navigation and to reduce the time required to zoom in to facilities of interest.
- One regional area consortium point of contact replied that this information should be easy to get through reporting to Vaccines for Children (VFC) and Vaccines for Adults (VFA) programs. It was suggested that data validation continue via other avenues, such as the California Department of Public Health (CDPH) and myCAvax. In response to this, the project found that these sources of information were not readily available or were not stored in a digitized central location. We would recommend further exploration of public dataset availability as part of ongoing validation and learning. Indeed, the application of machine learning can allow decision makers to bypass time consuming traditional data collection which can involve up to 90 days turn around for freedom of information requests.
- Timing is a key consideration. Late winter 2020 into early spring 2021 was a busy time for CHCs and not ideal for soliciting input due to COVID-19 vaccination efforts. In response to this we included a search functionality so that validation could be carried out more quickly.
- Another key timing response from an FQHC, after having been asked about their cold storage capability was that CDC changed cold storage requirements for Pfizer so that ultra low cold storage is no longer required. Striata reflects this by including all refrigeration capacity. Further, the findings as will be discussed next have applications beyond the immediate context of COVID and thus refrigeration capacity remains relevant.
Opportunities for Scale
After the rapid deployment or build phase of the work, there is an opportunity to scale to include additional infrastructure elements, answer broader questions around resiliency, add features that emerged during stakeholder engagement, and optimize ease of use for all stakeholders accessing the interface. While the use cases extend well beyond Covid, it is worth noting that the pandemic is far from over, and the idea of equitable access becomes more important the further we get into the vaccination campaign, especially when this is combined with a natural disaster.
Striata can pinpoint populations most at risk of wildfires and power outages alongside the existing user interface. This provides insight into whether the health center is fire-ready, what capacities need to be built, and whether they can improve resilience. It can cost a health center hundreds of thousands of dollars to be down for weeks at a time, making the return on investment of resiliency significant.
Striata can empower regional associations to have a more influential role in allocating resources and decision-making power, positioning visibility as an advocacy tool. The ability to see into the current state of that infrastructure can generate enormous efficiencies for the CPCA and similar associations to provide a single source of ground truth for dialogue.
Finally, as daily vaccination rates continue to drop, the cost of vaccinating 70% of the population is no longer just about getting vaccines to sites; it’s also about incentivizing people and ensuring that the infrastructure is in place to meet the demand and avoid wastage where possible.
