Research Data that Can Be Trusted

Introduction

In this book we argue for the need for a new approach to data provenance and explain how recent advancements in data processing workflow automation present an opportunity to address this need. We introduce descriptive dataflow operators - a novel approach based on integrating descriptive workflow languages with a data modeling Domain-Specific Language (DSL). We review the current workflow automation technologies and propose a DSL that supports complex data transformations, enhances reproducibility, and enables precise data lineage tracking. Within the framework we introduce the concept of descriptive dataflow operators for more flexible and expressive data transformations.

Healthcare Data Challenges

Modern healthcare increasingly relies on complex data pipelines to process diverse diagnostic information, clinical records, and research data. This growing complexity, combined with emerging AI/ML applications and stricter regulatory oversight, demands sophisticated approaches to data preparation, documentation, and validation. Healthcare organizations face mounting pressure to ensure granular traceability and reproducibility of their data transformations while maintaining regulatory compliance. These challenges are particularly acute in research settings, where data provenance and quality validation become critical for scientific reproducibility and regulatory adherence.

Technical Challenges and Framework

Given the increasing complexity of healthcare data, data ingestion and transformation workflows present significant technical challenges, particularly in ensuring the reproducibility and seamless integration of diverse datasets for ML and AI model development. We introduce the Dorieh Data Platform as an exemplar implementation of a DSL, providing a comprehensive framework for reproducible research. The platform's infrastructure supports robust data lineage documentation, validation, and error logging, making it a powerful tool for healthcare data analysis by ensuring transparent, auditable data processes and regulatory conformance.

Application and Outcomes

We show how to apply this framework to analyze healthcare claims data quality, revealing insights into inconsistencies and deficiencies. Our approach demonstrates the potential for improved data management and accountability in scientific research, underscoring the necessity for precise, reproducible data transformation methodologies to produce reliable research outcomes.