A streaming lakehouse for card transactions where every hop checks a data contract and stops the run when the data breaks it.
Edit a silver.transactions record or trigger a preset breakage, then validate it against the same row assertions the Spark engine runs over 1.85M rows.
A live map of the pipeline, hop by hop. Moving dots are rows in flight, coloured by whether they cleared the contract.
Read top to bottom. Inputs and contracts on the left, the ingest to marts spine in the centre, Iceberg tables and the dashboard on the right, and the Kafka streaming arm at the bottom.
An error breach halts the DAG, so the warehouse keeps serving yesterday's correct data. A warn breach is logged and shown.
Bad transactions (amounts outside $0 to $30k, a card with no hash) would move to silver.transactions_quarantine with the rule that caught them. The run fails only past 10% quarantined, a budget this clean, synthetic feed has never come close to.
Every check compiles to a Spark aggregate, merged into one df.agg. Silver's 14 assertions over 1.85M rows cost about 3.6 seconds.