Can you trust your analysis? Verification, validation and reproducibility

Find five planted bugs in a battery study, match a benchmark, validate it and rerun it from a clean clone

Research studio R5 of Energy Systems, Step by Step: check a one-day battery study solved in your browser, find its five planted bugs (a unit error, a sign flip, a free lunch, a missing rule, a leaked future) with named checks, match a benchmark worked by hand, validate the day-before forecast against measured data, check an input and how much the result depends on it, rerun it from a clean clone, and rate your own verification log against the course rubric.
  • Researcher · L4
  • Research studio
  • 35 min guided · about 2 h practice
  • After R4 and M1 and K8

The task

  1. Produce: a one-page verification log for a model or a data analysis: each check with its result, one benchmark against a known case, one validation against measured data or an independent reference, and a rerun from a clean clone.
  2. From: this page's battery study (the notebook has all five versions), or your own project.
  3. Done when: Trustworthy analysis reaches Proficient: reruns from a clean clone (or a complete audit trail); verification log; one validation against real data or an independent reference.

About 35 minutes guided here, then about 2 hours on your own: the notebook, Git and your log.

The study, its five buggy versions and the checks that catch them

How much would a 50 MW, 200 MWh battery have saved Prairie Town on Wednesday 20 August 2025? One day of dispatch as a linear program; 50 MW, 200 MWh (4 hours) battery, 85% round trip; data: Wednesday 20 August 2025.

  • Balance: Plants + battery discharge − battery charge + unserved = demand, every hour (MW). (24 rows)
  • Stored energy: Stored now = stored an hour ago + (0.85 × charging power − discharging power) × 1 hour, in MWh. All the round-trip loss is counted on charging. (24 rows)
  • Gas ramp: The combined-cycle plant changes by at most 40 MW an hour, from 115 MW. (24 rows)
  • End of day: The battery ends the day with the 100 MWh it started with. (1 row)
  • Sizes: Each plant within its size and the weather; battery 0–50 MW, 0–200 MWh. (limits on the variables)

Prairie Town's plants (lesson 0):

Plant Size (MW) Running cost ($/MWh) How it runs
Nuclear plant 100 10 runs flat
Wind farm 150 0 follows the wind
Solar farm 200 0 follows the sun
Gas combined-cycle plant 150 40 adjustable, slow to change, at most 40 MW an hour
Gas peaker 100 100 adjustable, fast

In the code:

  • c[t], d[t]: the battery's charging and discharging power in hour t, MW
  • e[t]: stored energy at the end of hour t, MWh (the state of charge)
  • E, E0: the battery's energy size, 200 MWh, and what it holds at the start of the day, 100 MWh
  • cc[t], cc_prev[t]: the gas combined-cycle plant's output in hour t and in the hour before, MW (115 MW before the first hour)

  1. Version 1: a unit error. Saving reported: $12,150. Most energy stored (the battery holds 200 MWh): 279.5 MWh.

    Caught by: Units and sizes. The battery holds more than its size: the energy limit was typed in kWh into a model that counts MWh.

    E = 200_000 # battery energy, from the spec sheet → E = 200 # battery energy, MWh (the sheet says 200,000 kWh)

    Fixed: $8,205.88, and stored energy never passes 200 MWh.

  2. Version 2: a sign flip. Saving reported: $19,058.82. Hours charging and discharging at once: 22 of 24. Net energy to the town, as the balance rows count it: 176.47 MWh. Stored energy at the end of the day: 100 MWh, as at the start.

    Caught by: Balance and signs. Charging was counted as supply, so the battery fed the town while filling up: energy from nowhere.

    supply[t] += c[t] - d[t] # battery → supply[t] += d[t] - c[t] # battery: discharge feeds the town

    Fixed: $8,205.88. The battery now charges when the gas plant has room and discharges in the hours the peaker would run.

  3. Version 3: a free lunch. Saving reported: $18,205.88. Stored energy, start and end of the day: 100 and 0 MWh.

    Caught by: Closure over the day. The 100 MWh the battery started with cost nothing in the model, and it sold them in the evening.

    # (no rule for the end of the day) → m.add(e[23] == E0) # end the day with what it started with

    Fixed: $8,205.88. The gap, $10,000, is the starting 100 MWh at the peaker's $100/MWh.

  4. Version 4: a missing rule. Saving reported (the fixed study: $8,205.88): $8,205.88. Gas plant's biggest change in an hour, in this solution (limit 40): 55 MW.

    Caught by: Paper against code. The formulation has 24 ramp rows; the code has 0. Today the headline happens not to move.

    # (the ramp rows were never written) → for t in T: m.add(-40 <= cc[t] - cc_prev[t] <= 40) # gas ramp

    On 3 September 2025 (lesson 0's day) the same code says the day without a battery costs $102,600 instead of $104,000, and that the battery saves $0 instead of $1,286.49.

  5. Version 5: a leaked future. Saving reported: $8,205.88. Schedule planned on: 20 August itself.

    Caught by: Information at decision time. The schedule is set at midnight, as 20 August begins: all 24 hours of 19 August are measured by then, and none of 20 August.

    plan = solve(day='2025-08-20') # the day itself → plan = solve(day='2025-08-19', tie='keep') # the last full day measured at 00:00 on 20 August
    result = run(day='2025-08-20', battery=plan)

    Planned on 19 August's measured day (a persistence forecast) and then followed: $3,600, 44% of the $8,205.88 that perfect foresight gives.

    Many schedules tie at $218,050 on 19 August. The rule, set in advance: keep the most energy in store. One that empties the battery first ties too, yet saves only $300 on 20 August.

Benchmark (a made-up day: 220 MW in every hour except 330 MW from 12:00 to 17:00, no wind or solar). Without the battery, the gas plant runs 120 MW in the flat hours; in the five dear hours it runs flat out and the peaker adds 80 MW, at $100/MWh.

  1. Discharge. The battery empties its 200 MWh into the five dear hours: 200 MWh × $100/MWh = $20,000 of peaker output avoided.
  2. Recharge. Over the day it puts 200 MWh into store (100 before noon, 100 after, so it ends as it began): 200 ÷ 0.85 = 235.29 MWh of charging, from the gas plant's 30 MW of spare room, at $40/MWh: $9,411.76.
  3. Saving. $20,000 − $9,411.76 = $10,588.24.
Version Saving on the benchmark Matches the hand answer?
Fixed study $10,588.24 yes
Version 1 (unit) $13,235.29 no
Version 2 (sign) $21,044.12 no
Version 3 (free lunch) $15,294.12 no
Version 4 (missing rule) $10,588.24 yes

Validation: The plan's prediction of 20 August is its forecast: 19 August's measured net load (demand minus wind and solar). Compare it with 20 August as measured. Tolerance, set first: the battery's 50 MW, in every hour. The forecast misses by 22.1 MW on average and 55 MW at 12:00: beyond the tolerance in 1 of 24 hours. So the claim is narrowed: $3,600 with this forecast; $8,205.88 only as an upper bound.

Input check: One input can be checked against measured data: the round trip. The study assumed 85% (NLR's 2025 ATB); the U.S. battery fleet averaged 82% in 2019 (EIA). Does the saving hinge on it? $7,939.02 instead of $8,205.88: 3% less, so the conclusion does not hinge on this input. That is an input check and a sensitivity run, not a validation of the saving.

Rerun from a clean clone (simulated here): it failed on a data file read from one laptop's desktop; a download script with a checksum, relative paths, a pinned environment and a tag fix it. The notebook's practice does the rerun for real.

With JavaScript on, you run each version with a real solver in your browser, pick the check that catches its bug, fix it and run it again, then copy your verification log.

Data: demand, wind and solar follow the real hourly shapes of 19 and 20 August and 3 September 2025 on the Southwest Power Pool grid (EIA-930), scaled to Prairie Town as in lesson 0. Illustrative: Prairie Town, its plants, costs and battery; the benchmark day is made up. Solver: HiGHS (open source), loaded on your first run (about 3.5 MB, once).

Practice on your own · notebook and Git, about 2 hours

The same study in scipy.optimize.linprog(method="highs"), with all five versions behind a switch. You write each check as a function, add the benchmark as a test, put the folder under version control and rerun it from a fresh clone.

  1. Download the notebook; the page's three days are inside it.
  2. For the robustness check on other days (practice 3), also download town_year.json and put it next to the notebook.
  3. Run it in Google Colab (File → Upload notebook) or locally with Python 3.9 or later, NumPy and SciPy 1.9 or later.
  4. For the Git part, work locally: the notebook lists the commands, and K8 (Git and reproducible notebooks) shows them step by step.

Notebook (.ipynb) Year data: town_year.json Study data (JSON)

Verify, benchmark, validate, rerun

  1. Formulation on paper, then code and data.
  2. Verification checks: units, sizes, balance and signs, closure, paper against code, information at decision time. If one fails, fix the code and run every check again.
  3. Benchmark: a case with a hand answer. If it does not match, fix the code.
  4. Validation: a prediction against measured data or an independent reference, with a tolerance set first. If the model is not close enough for the claim, revisit the assumptions or narrow the claim.
  5. Rerun from a clean clone, then report with limits.

Two logs for the same study

Weak log · Developing

Checks: I checked the model and the results look reasonable [1]. Benchmark: none. Validation: the model was validated [2]. Input check: none. Result: the battery saves $8,205.88 on 20 August [3], so about $3.0 million a year [4]. Rerun: the code is in a folder on my laptop [5].

  1. Falls short: No check is named, so nobody can repeat it or see what it would have caught.
  2. Falls short: Validated against what? A validation names its data, its tolerance and the outcome.
  3. Falls short: This is the perfect-foresight saving, and the log does not say the plan knew the day in advance.
  4. Falls short: One hot day times 365 is not a year: on 3 September the same battery saves $1,286.49.
  5. Falls short: Nobody else can rerun a folder on one laptop.

Strong log · Proficient

Checks: stored energy within 0–200 MWh in every hour [1]; every hour balances, and stored energy falls whenever the battery feeds the town [2]; the day ends with its starting 100 MWh [3]; 24 ramp rows in the code, as on paper [4]. Benchmark: a made-up two-level day, hand answer $10,588.24, model $10,588.24 [5]. Validation: the day-before forecast against 20 August as measured, off by up to 55 MW, beyond the 50 MW tolerance set in advance, so the claim is narrowed [6]. Input check: the 85% round trip against the fleet's measured 82%: the saving moves 3% [7]. Result: $3,600 with the plan made the day before, $8,205.88 with perfect foresight, an upper bound [8]. Rerun: a clean clone at tag v1.0, with the pinned environment, matches the first run's checksum [9].

  1. Works: A size check with its range: it would have caught version 1.
  2. Works: A balance check that also tests the sign convention: it would have caught version 2.
  3. Works: A closure check: it would have caught version 3.
  4. Works: Paper against code: the only check here that catches version 4 on this day.
  5. Works: A known case, worked by hand, matched to the cent.
  6. Works: A prediction against measured data, with the tolerance set first, and what the outcome changes.
  7. Works: An input checked against measured data, then a sensitivity run; the log does not call it validation.
  8. Works: Both numbers, each labelled with what the plan knew.
  9. Works: A tag, a pinned environment and a checksum let anyone get the same numbers.
Weak log Strong log
Checks None named Four, each with its result
Benchmark None Hand answer matched
Validation Claimed A forecast against measured data, tolerance first
Information Perfect foresight, unstated Both numbers, labelled
Rerun One laptop A tagged clean clone, same checksum

Check your own log

A verification log in nine lines

  1. Units: Every input in the model's units, with the unit written beside it.
  2. Sizes: Every result within its physical size and limits.
  3. Balance and signs: Each period balances; a store's energy falls when it feeds the system.
  4. Closure: The end state matches the start, or the difference is valued.
  5. Paper against code: Each rule's rows counted against the formulation.
  6. Information: Every input was known when the decision was made.
  7. Benchmark: One case with a hand answer, matched.
  8. Validation: One prediction compared with measured data or an independent reference, with its source and a tolerance set in advance.
  9. Rerun: A clean clone at a tagged commit, a pinned environment, the same checksum.
Competence Developing Proficient (the bar) Strong
Trustworthy analysis
Look for: Your log names each check with its result, one benchmark with its hand answer, one validation with its data and tolerance, and the tag a clean clone reran.		 Results cannot be rerun. Reruns from a clean clone (or a complete audit trail); verification log; one validation against real data or an independent reference. Independent reproduction by someone else; validation on data not used to build the analysis.

Myth or fact? “If the solver says optimal, the study is right.”

  • Myth
  • Fact
Answer

Myth. All five versions on this page solved to optimal. Optimal means the best answer to the equations as written, bugs included.

Myth or fact? “A study that matches its benchmark has no bugs.”

  • Myth
  • Fact
Answer

Myth. Version 4, with no ramp rows at all, matched the benchmark to the cent: $10,588.24. A benchmark tests only what it exercises.

Myth or fact? “Verification and validation are two words for the same check.”

  • Myth
  • Fact
Answer

Myth. Verification compares the code with its own equations; validation compares the equations with the real system, for the purpose of the claim.

Myth or fact? “With the same data, code and pinned environment, a clean clone should reproduce the reported numbers.”

  • Myth
  • Fact
Answer

Fact, for numbers that are unique. When several plans cost exactly the same, another solver version may return another plan at the same cost. A number that depends on the plan, such as the day-before saving here, needs a tie rule set in advance.

Go deeper · ResearcherFrom Proficient to Strong: someone else reruns it, on data you did not use

The rubric’s Strong level asks for two more things.

  • Independent reproduction. A colleague reruns your study from your repository, without your help, and reports any difference.
  • Validation on data the analysis did not use. Here: the day-before forecast checked on many more days than 20 August; or, for a real battery, the model’s stored energy predicted from its metered charging and discharging and compared with its metered state of charge, within a stated tolerance.
  • Not validation: running the same model on other days of 2025 (practice 3). That is a robustness check: it shows how far one day’s saving carries over (3 September gives a very different one), not that the model is right.

In the capstone research project you show both in the verification log.

Verification and validation for computer models have their own literature; for energy-system optimization studies, DeCarolis and co-authors set out good practice, including how to test a model and report what it leaves out (DeCarolis et al. 2017).

Go deeper · ModelerPerfect foresight is a modelling choice, not a bug, as long as it is named

Many planning studies schedule storage knowing the whole day, week or year in advance, because it is simple and gives an upper bound on what storage can earn. That is fine when the report says so. It becomes the leaked future of version 5 when the upper bound is reported as what an operator would have saved. Between the two sit rolling-horizon schedules that re-plan as forecasts improve (M5, in the Modeler tier).

Sources and what’s next

Numbers and tools used on this page

Number or tool What it is for Source
2025 hourly shapes Demand, wind and solar of 19 and 20 August and 3 September 2025, scaled to Prairie Town. EIA-930, Southwest Power Pool · accessed 2 Nov 2026
85% The study's round-trip efficiency. NLR, 2025 Electricity ATB, Utility-Scale Battery Storage · checked 2 Nov 2026
82% Average monthly round-trip efficiency of U.S. utility-scale batteries, 2019 data: the input check. EIA, Today in Energy, 12 Feb 2021 · accessed 3 Nov 2026
HiGHS The open-source solver in the page and, through SciPy, in the notebook. HiGHS documentation

Next

R6 · Uncertainty done properly: what to vary, how, and how to say it

Related lab: a SciPy/HiGHS notebook with the study's five versions, the checks as functions, the benchmark as a test and a clean-clone rerun, about 2 hours.

Sources and further reading

Spotted an error, or stuck? Write to hlli@ksu.edu. Corrections are made in place and dated here.

Text licensed CC BY 4.0, code MIT: teachers and students may reuse and adapt it, with credit to Honglin Li.