RadHarmony: Making Radiological Data Work for Agentic AI

Training models on radiological images is still slowed by a basic infrastructure problem. Public datasets differ in file formats, directory structures, label schemas, and annotation types. Integrating them into a single training pipeline usually requires custom, one-off engineering for each source.

RadHarmony is an open-source library designed to remove that friction. It standardizes metadata from 24 public datasets into a unified format and delivers samples through a single interface that supports classification labels, segmentation masks, bounding boxes, and radiology reports. The system is built to work with chest radiographs first, with early support for CT and MRI.

The most distinctive contribution is an AI-agent skill that guides the full process of adding a new dataset: inspecting the raw data, generating the necessary integration code, and running verification tests. Most of the supported datasets were integrated through this human-AI collaborative workflow.

The library’s utility is demonstrated by pretraining a vision transformer across three major chest X-ray datasets with no dataset-specific code. The result is a practical foundation for multi-source training rather than isolated, single-dataset experiments.

This Ledger Entry expands how the reader thinks by treating data harmonization not as tedious preprocessing, but as a first-class systems problem that agentic workflows can systematically address, and by positioning clean multi-dataset infrastructure as a prerequisite for more capable medical foundation models.