Using the Dataset¶

This page is for users of the dataset, for example to evaluate or train your own model. It explains what the dataset contains, what you run, and how to plug in your own model.

The notebook benchmark/llm_benchmark_en.ipynb provides a simple introduction to using the dataset:

Getting the dataset¶

The dataset is part of the repository under benchmark/. No separate download is required:

git clone https://github.com/dgaida/PyADM1ODE.git
cd PyADM1ODE

The PyADM1ODE environment is needed to run code (the biogas conda environment is recommended). Pure graph scoring (matcher.py) and the viewer work without PyADM1ODE.

What it contains¶

benchmark/
  schema/    plant_datapoint.schema.json   JSON schema (Draft 2020-12) of a data point
  dataset/   index.json                    manifest of all data points
             BGA1/ BGA2/ ...               one plant each, input variants + gold.py
  eval/      solve.py runner.py matcher.py batch.py make_index.py validate.py …
  viewer/    index.html                    interactive data point viewer

One folder per plant. It holds all input variants plus a shared gold solution gold.py (correct PyADM1ODE code).

File	Role
`BGAx_<variant>.json`	Task: input (text, image or PDF) for the model and the reference plant (typed graph) for comparison
`gold.py`	Gold solution: known-correct implementation, validates the harness and serves as reference code

Variants and regime¶

Each plant exists in several variants. The _full suffix marks completeness:

fully_specified (_full): all data is in the input – no oracle needed.
underspecified (no _full): values are missing – the model has to ask the
oracle. Guessing does not work; the acceptance bands are too tight for that.

Axis	Values	Distribution (24)
Plant	BGA1, BGA2, BGA3	8 each
Completeness	fully_specified / underspecified	12 / 12
Modality	text / image / hybrid / pdf	18 / 3 / 3 / 0
Language	de / en	18 / 6

The exact makeup of a single data point is described in A Data Point in Detail. The authoritative format lives in the JSON schema at benchmark/schema/plant_datapoint.schema.json.

What you run¶

Goal	Command
Matcher self-test (without PyADM1ODE)	`python benchmark/eval/selftest.py`
Validate data points against the schema	`python benchmark/eval/validate.py`
Baseline – all data points with `gold.py`	`python benchmark/eval/batch.py`
One data point: run code + score	`python benchmark/eval/runner.py <datapoint.json> <code.py>`
Score the graph only (no code run)	`python benchmark/eval/matcher.py <datapoint.json> <candidate.json>`
LLM evaluation	`python benchmark/eval/solve.py [filters]`
Refresh viewer/index	`python benchmark/eval/make_index.py`

Examples:

# 1) Quick functional check of the matcher (no PyADM1ODE needed)
python benchmark/eval/selftest.py

# 2) Baseline: score all 24 data points against their gold.py
python benchmark/eval/batch.py

# 3) Run and score a single data point with the gold solution
python benchmark/eval/runner.py \
    benchmark/dataset/BGA1/BGA1_text_de.json benchmark/dataset/BGA1/gold.py

batch.py writes benchmark/results.csv. solve.py stores a CSV plus the generated code per data point under benchmark/results/.

Plugging in or training your own model¶

The input is in the data point under input (fields modality, language, content, optionally image_path or document_path). The target is runnable PyADM1ODE code that builds the plant. The reference for comparison is under reference (components + connections as a typed graph). gold.py shows a correct implementation per plant.

The quickest path is interactively in the notebook. For automated runs there are two scripted ways (A and B).

A) Generate code offline, then score¶

Have your model produce Python code per data point and save it as <datapoint-id>.py in a folder. Then:

python benchmark/eval/batch.py --candidates path/to/model_outputs

batch.py runs each candidate in isolation and scores it.

B) Directly via solve.py (API)¶

solve.py is a full run including oracle rounds. By default it uses the Groq API. For your own model, only the client section in solve.py needs adapting; the rest of the workflow (prompt, oracle, scoring) stays the same.

pip install -e ".[benchmark]"   # groq + pypdf; or your own client library
export GROQ_API_KEY=...    # or your own API key
python benchmark/eval/solve.py --regime fully_specified   # simplest entry point

Follow-up questions on underspecified data points¶

If a value is missing, the model may ask first — as a JSON block with open_questions (format see prompt.py). solve.py forwards the questions to the oracle and sends the answers in a second turn.

The questions themselves are not graded. Only the plant that finally comes out is scored: not asking and still hitting plausible values costs nothing — not asking and getting it wrong costs points on Measures.

Programmatic access¶

The manifest dataset/index.json lists all data points with id, path, language, modality, regime:

import json
from pathlib import Path

root = Path("benchmark/dataset")
index = json.loads((root / "index.json").read_text(encoding="utf-8"))

for entry in index["datapoints"]:
    dp = json.loads((root / entry["path"]).read_text(encoding="utf-8"))
    task = dp["input"]["content"]        # description for the model
    reference = dp["reference"]          # target plant (components + connections)
    # ... call your own model, generate code, score ...

Scoring¶

Three scores are computed, deliberately disjoint — every mistake counts exactly once:

Completeness – are all required components and connections present?
Measures – simulated parameters (V_liq, V_gas, T_ad, P_el_nom, …) within the acceptance band.
No inventions – does the plant contain only components and connections the reference knows?

How the final score is produced is explained in Scoring & Workflow. You can explore the dataset visually in the Viewer.