Job
Knobs
empty = Spark default. Curated set below; freeform rows at bottom.
Run = in-process Spark. Run via Livy = REST batch through the gateway on :8998 · the CDP/Hue path.
Wall time per run (ms)
Data moved · shuffle write MB
Run history · click two rows to compare
| run | job | wall ms | Δwall | shuf rd MB | shuf wr MB | spill MB | tasks | key knobs |
|---|
Stream · payments topic (live while StreamPayments runs)
| batch | rows in | in/s | proc/s | batch ms |
|---|
Bronze · land everything
Raw arrivals, duplicates kept. The relay is at-least-once; bronze is where that truth lands.
→
Silver · merge into truth
MERGE INTO dedupes on a stable key. Same effectively-once move as the streaming sink, done on files with ACID.
→
Gold · serve answers
Balances upsert in place across rebuilds; revenue overwrites. Both write shapes, both in the log.
Commit history ·
| v | operation | +files | −files | rows written | rows @ v | when |
|---|
rows @ v = what a reader sees at VERSION AS OF v: the log replayed to that point. Time travel is exactly this replay.
Commit · the actual log entry
click a commit row. What renders here is parsed live from _delta_log on disk · the JSON that IS the table.
Why a log makes files a table
A Delta table is a directory of ordinary Parquet files plus an ordered list of JSON commits (_delta_log/).
Each commit atomically declares files added and removed; a reader lists the log, replays it, and gets a consistent snapshot ·
no half-written state is ever visible, because a commit either exists or it doesn't.
Version-as-of reads stop the replay early; MERGE, OPTIMIZE and VACUUM are just commits with different intents.
The panels above render the real log of this lake, parsed from disk on every request.