Skip to content

Spool

Spool is a Java framework for building data pipelines that take data from external sources to a data mart. Each part of the pipeline is a separate module, and modules communicate through events, so every record can be traced from the moment it is read until it reaches the data mart.
sourcecrawlerinboxingesterdata lakemounterdata martresponseenvelopesenvelopesrecordspartitionsresultsEnvelopeStoredjanitorupdates statesEnvelopePersistedevent on the busdata
The crawler stores each record in the inbox and announces it on the event bus. The ingester reacts, writes the record to the data lake and announces it again, and the janitor keeps the inbox up to date. The mounter turns data lake partitions into a data mart.

Overview

A pipeline is described in a YAML file, or built in Java with the builder of each module, and started with SpoolRuntime. The same pipeline can run as a single process on a laptop or spread across several machines.

Features

  • Every step a record takes is published as an event, with a correlation identifier shared by all the events about that record.
  • Records that break a validation rule are not written to the data lake. They go to quarantine with the list of violations.
  • Storage and messaging are ports with interchangeable backends, such as the file system or S3 for storage and Kafka or memory for the event bus.
  • Validation rules are plain Java classes, discovered at runtime.
  • The same pipeline runs on a laptop, as a single service or spread across several machines. Only the infrastructure in the descriptor changes.
  • Every module exports traces, metrics and logs with OpenTelemetry, and the devenv stack shows them in a Grafana dashboard for each module.

Getting started

If you are new to Spool, start with these guides.

Modules

Each module is published as its own Maven artifact. Use the ones your pipeline needs.

ModuleWhat it does
crawlerReads external sources and saves each record in the inbox.
ingesterValidates records and writes them to the data lake.
validatorRuns the validation rules you write.
janitorRetries stuck records, quarantines failed ones and deletes expired ones.
mounterAggregates data lake partitions into the data mart.
watchdogCollects heartbeats and reports the health of each module.