Case study · open source
jobradar: an AI data pipeline you can audit
A pipeline that runs on a laptop, uses local models, and can prove where every piece of data came from and how to delete it. Built as the demo for my talk "Who Owns Your AI Data Pipeline?" (SFSCON 2026).
Proof of concept, not a production job tool, and not affiliated with Arbeitnow or the Bundesagentur für Arbeit.
The problem
AI steps are creeping into data pipelines: classification, extraction, matching. Each one raises the same questions that governance teams ask about any pipeline, and a few new ones. Which model produced this value? Which prompt version? Did the data leave our environment? Can we delete everything derived from one person's data, and prove we did?
Most demos skip these questions. I wanted a small, honest system where the answers are part of the design.
How it works
Job ads come in from public APIs, get scrubbed and enriched with local models, and are matched against a CV that stays on the machine. A lineage event is emitted at every step.
Retention deletes ads older than 7 days from every table and the vector store. Erasure follows the lineage log from your CV hash to everything derived from it.
Design decisions
Each governance idea has a concrete place in the code, so it can be inspected rather than trusted.
| Idea | How it is implemented |
|---|---|
| Open formats, no lock-in | Plain Delta tables (delta-rs) and Lance vectors. jobradar exit-test reads every table with DuckDB's own Delta reader. |
| Lineage past the model | OpenLineage events for ingest, enrich, CV load, match, retention and erasure, with table versions and custom facets for model, prompt version and CV hash. |
| Swappable model | Any OpenAI-compatible endpoint, chosen by environment variables. Every output records which model produced it. |
| Sovereignty | Defaults run locally with Ollama. Lineage flags runs_locally for every model call, so a hosted endpoint is visible in the record. |
| Data contracts | ODCS v3 contracts, checked in CI. |
| GDPR in the pipeline | Contact data scrubbed at ingest, 7-day retention with vacuum, and jobradar erase, which finds everything derived from your CV by following lineage and vacuums Delta files so time travel cannot restore it. |
| Explainable extraction | field_sources records whether each value came from the source API, a rule, embeddings or the LLM. |
| Terms travel with the data | Each source's licence and attribution requirement is recorded on its lineage input and on every silver row. |
What I would call out
- Honest by construction. The tailored CV and cover letter feature keeps a bullet only if it appears in your CV word for word. The model chooses and orders; it does not invent. Claims the CV does not support are flagged, and skills the ad wants but you lack are listed, not claimed.
- Privacy by default. The CV becomes a profile without name or contact details, and lineage stores only its hash.
- Local first. The Apply page refuses to generate if the model endpoint is not on the same machine.
field_sources exists. Redaction is pattern-based and can miss unusual formats. There is no authentication; the web page is meant for localhost.Stack
- Python 3.11+
- delta-rs
- DuckDB
- LanceDB
- OpenLineage + Marquez
- Docling
- Ollama
- Qwen3 8B
- BGE-M3
- Streamlit
- ODCS v3
- Apache 2.0