Overview
A pipeline is described in a YAML file, or built in Java with the builder of each module, and started with SpoolRuntime. The same pipeline can run as a single process on a laptop or spread across several machines.
Features
- Every step a record takes is published as an event, with a correlation identifier shared by all the events about that record.
- Records that break a validation rule are not written to the data lake. They go to quarantine with the list of violations.
- Storage and messaging are ports with interchangeable backends, such as the file system or S3 for storage and Kafka or memory for the event bus.
- Validation rules are plain Java classes, discovered at runtime.
- The same pipeline runs on a laptop, as a single service or spread across several machines. Only the infrastructure in the descriptor changes.
- Every module exports traces, metrics and logs with OpenTelemetry, and the devenv stack shows them in a Grafana dashboard for each module.
Getting started
If you are new to Spool, start with these guides.
Modules
Each module is published as its own Maven artifact. Use the ones your pipeline needs.
| Module | What it does |
|---|---|
| crawler | Reads external sources and saves each record in the inbox. |
| ingester | Validates records and writes them to the data lake. |
| validator | Runs the validation rules you write. |
| janitor | Retries stuck records, quarantines failed ones and deletes expired ones. |
| mounter | Aggregates data lake partitions into the data mart. |
| watchdog | Collects heartbeats and reports the health of each module. |