Real-world run

mozilla/bigquery-etl · not the frozen eval

No gold labels — no accuracy claim. This report records observables only (parse rate, verdict distribution, UNKNOWN reasons). It is not a substitute for eval/RESULTS.md.

Project	mozilla/bigquery-etl
Commit	`717b6a14846a3a294049a6cab42168eaf4bb5b2c`
Subject	`clients_daily_v6` · column `client_id` · dialect `bigquery`
Queries scored	118 (mention `clients_daily` and `client_id`; tests/ and mozfun excluded)

Parse rate

84.75% (100/118) non-parse-failure. Failures (18): Jinja templating, BigQuery scripting, sqlglot coverage gaps.

What this run measures

This is missing-schema behavior. With schemas present (a real DataHub-connected warehouse), narrowable references bind instead of abstaining; the 11% CLEARED below are qualified references to known non-subject tables.

Corrected verdict distribution

After fixing a false-CLEARED defect (clear only when the qualifier resolves to a known non-subject table), most incomplete-schema references land in UNKNOWN — the honest outcome when Premortem lacks the schemas.

verdict	count	share
hard	1	0.8%
soft	1	0.8%
unknown	102	86.4%
cleared	13	11.0%
unaffected	1	0.8%

UNKNOWN breakdown

reason bucket	count
unresolvable_qualifier	82
unparseable	18
star	2

View-expansion limitation

6 of the 13 CLEARED resolve to clients_daily while the subject is clients_daily_v6. Correct at the table level, but if that name is a view over the subject, a rename propagates through it. Premortem reasons about tables, not view expansion.

Command

python tools/real_world_mozilla_run.py \
  --repo /path/to/bigquery-etl \
  --out-dir docs \
  --sha 717b6a14846a3a294049a6cab42168eaf4bb5b2c

Source

Full markdown on GitHub (examples, correction note, defect flags)
Raw JSON