Upcoming Event!
View All Events
How Microsoft Sentinel is Disrupting the SIEM and XDR Markets, Virtual
July 13th, 2022 at 12:00 p.m. ET
Register
Tallan Logo

Creating Data Trust: One State's COVID-19 Response

Overview

A Department of Public Health sought assistance in their COVID-19 response. To paraphrase, the mission of this project was to improve the ability of this agency to quickly and effectively respond to the COVID-19 pandemic by enabling data-driven decision making, backed by a cloud-based Data Analytics platform. There were four critical needs identified, with an underlying focus on long-term requirements and goals. Tallan developed a scalable foundation that effectively enhanced data integration, report discoverability, and data quality. With much of the data entering from external labs and providers, our solution focused on the analytics and consumption end of the overall data flow, and supported trust in the data. 

Industry

Government Administration

Company Size

1,000+

Technology

DPi30, Data & AI

Critical Need #1: Data Integration.
  • Data integration was the first identified ‘critical need’ for this project. After an envisioning exercise, it became clear that data quality took precedent over integration, because time was being wasted evaluating why data was missing, and where it was missing from.
  • Since the DOH had no control over collection at COVID testing sites, and thus no control over quality, Tallan focused on providing a clear view of the state of the data in order to ensure trust. To create data trust, we built a process monitoring solution with a dashboard in Power BI that reports on data flow and pipeline processes, their status, typical metrics (run time, volume, etc.) and trends.
  • Built into the re-engineered data pipelines is a robust logging solution that enables the collection of metrics across various stages to pinpoint any potential point of failure in data flow. This client is now processing 100,000+ new lab results per day while performing probabilistic person matching on 50 million+ previous instances of data.
Critical Need #2: Report & Data Discoverability.
  • The 2nd ‘critical need’ was Report & Data discoverability. The DOH teams responsible for reporting to both government staff and constituent audiences required a consistently reliable, central location to support awareness and sharing capabilities for DOH analysts.
  • Data lineage needed to be structured, discoverable, and controlled.  While data quality was out of DOH’s control, data trust was something Tallan could provide through the governance strategy and the process monitoring and data quality dashboards developed to solve the data integration concern. 
Critical Need #3: Data Quality.
  • Third in line of ‘critical needs’ was data quality. Since quality control proved difficult, if not impossible, we developed a dashboard to report on trends of increasing or decreasing data quality. Now, quality can be proactively addressed and preventative actions can be taken if and where possible.
  • To further support trust in data quality, we built and deployed a framework that automates the addition of new data points and quality metrics to the dashboard to continue enhancing the utility of the tool. Tallan worked to cut processing time of data pipeline flows by as much as 80%!
  • These quality metrics can now be shared with the sources (COVID test sites), so they can measure performance. 
Critical Need #4: Operations Impact.
  • The final ‘critical need’ was operations impact. Ultimately, Tallan realized that the issue was not the data quality; it was the data trust.
  • Without understanding data lineage, ownership, and business definition, coupled with the lack of guardrails to control the sharing of COVID-19 analysis data, it was near impossible to understand the origin, intent, and nature of the datasets in play.
  • Instituting best practices for data governance was an important step in addressing data quality. This was a big step forward in supporting the operations impact of the engagement. The tools Tallan used for analytics and data consumption are all cloud-based which will allow for continued innovation and scale.
Start Dreaming with Tallan Today
Contact Us
You must be logged in to post a comment.
Menu