Data you can defend: sources, cleaning, licences and provenance

Log every cleaning choice and show what it changes

Research studio R4 of Energy Systems, Step by Step: clean a real year of hourly grid data (SPP 2025, EIA-930), log what each cleaning choice changes, check data licences and write a data note.
  • Researcher · L4
  • Research studio
  • 30 min guided · about 2 h practice
  • After R3 and B2

The task

  1. Produce: a cleaning log (rule, count, reason and effect for each step), a licence check of every source, and a data note someone else can rerun from.
  2. From: SPP's hourly file for 2025 (demand, wind, solar; EIA-930, public domain), the starter notebook and, in practice, one spatial layer.
  3. Done when: the rubric's bar for R4's part: a complete audit trail, with licences checked and credit stated. The verification log and a validation come in R5.

About 30 minutes guided here, then about 2 hours on your own.

Four numbers, raw and with the course's choices

Choices Demand, whole year Lowest hour of demand Peak hour, clock time Wind, whole year
Raw: UTC labels, no rules 299.48 TWh 1.5 GW 21:00–22:00 UTC, 29 July 110.46 TWh
The course's: time base: Keep the UTC rows; label results in Central time with daylight saving; spike rule: Flag demand under half the median; frozen-reading rule: 3 hours; fill method: Use the same hour on nearby valid days 298.56 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.73 TWh
Change time base: Keep the UTC rows and report in UTC 298.56 TWh 24.4 GW 21:00–22:00 UTC, 29 July 110.73 TWh
Change time base: Shift to Central Standard Time (UTC−6) all year 298.56 TWh 24.4 GW 15:00–16:00 CST, 29 July 110.73 TWh
Change time base: Shift to the local clock and force every day to 24 rows 298.56 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.72 TWh
Change spike rule: Keep every value 298.53 TWh 1.5 GW 16:00–17:00 CDT, 29 July 110.73 TWh
Change spike rule: Flag demand more than 3 standard deviations below the mean 298.56 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.73 TWh
Change frozen-reading rule: Off 299.51 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.46 TWh
Change frozen-reading rule: 2 hours 298.56 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.72 TWh
Change frozen-reading rule: 4 hours 298.56 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.73 TWh
Change frozen-reading rule: 6 hours 298.57 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.73 TWh
Change frozen-reading rule: 12 hours 298.73 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.68 TWh
Change frozen-reading rule: 24 hours 298.61 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.42 TWh
Change fill method: Leave the flagged hours out 275.09 TWh 24.4 GW 16:00–17:00 CDT, 29 July 101.23 TWh
Change fill method: Draw a straight line across each gap 297.72 TWh 24.4 GW 16:00–17:00 CDT, 29 July 110.93 TWh

With the course's rules the counts match its data build: demand 699 hours flagged (699 in the build), wind 671 (671), solar 1,377 (1,377).

Licences: what each source asks of you

  • EIA-930 hourly grid data (this studio's file): Public domain. A U.S. government work; EIA asks for an acknowledgment with the date.
  • U.S. Wind Turbine Database (turbine locations): Public domain. Public domain; the USGS asks for an acknowledgment line.
  • Ember monthly electricity data: Attribution (CC BY). CC BY 4.0: credit Ember.
  • NLR Annual Technology Baseline, 2024 release: Attribution (CC BY). The 2024 release is CC BY 4.0; check each year's release.
  • OpenStreetMap roads and power lines: Share-alike or non-commercial. ODbL: credit OpenStreetMap and its contributors. A database built from it and shared publicly stays under ODbL; a map made from it carries the credit, and may oblige you to offer the changed data.
  • IEA World Energy Outlook 2025 free dataset: Share-alike or non-commercial. CC BY-NC-SA 4.0 IGO: credit, no commercial use, share alike.
  • ENTSO-E Transparency Platform: Check each item. Open data policy since 2019, but check the terms of each item.
  • A co-op's meter data, shared by a co-author: Check each item. Governed by the data agreement, not an open licence: read it before you share anything (R9).

A check the rules do not pass: with the course's choices, 65 night hours (starting 00:00 to 03:00 local time) still show solar above 5% of its peak, 49 of them written by the same-hour fill. The log reports it as a known limit; a night rule is the next step (B2).

A spatial layer

Your study adds wind-turbine locations from the U.S. Wind Turbine Database for Prairie Town's county (illustrative). The file stores latitude and longitude in degrees (EPSG:4326).

  • Which coordinate system for 1 km setback buffers (the zone within 1 km of each turbine)? Right: a local projected system in metres, such as the county's UTM zone, keeps distances close to true.
  • And for the buildable area in square kilometres? Right: an equal-area system for the contiguous United States keeps areas true.

With JavaScript on, you make each choice yourself, watch the four numbers move, sort the licences and write your own data note; your drafts stay in this browser.

Data: Southwest Power Pool (balancing authority SWPP), 2025, from EIA's Form EIA-930 six-month files (public domain), downloaded on 2 November 2026.
Before this page: an EIA-adjusted teaching extract: each fuel's adjusted generation columns summed and negative values set to 0; nothing else repaired.
On this page: every number is computed from that extract with the rules you choose.
Illustrative: the county and its turbine layer. Licence terms were checked on the dates in the sources.

Two answers to the same task

Weak answer · Developing

Data from EIA [1]. We removed outliers [2] and filled missing values [3], and converted times to local time [4]. Annual demand was about 299 TWh [5].

  1. Falls short: Which form, area, files, download date and licence? A reader cannot find the same file.
  2. Falls short: No rule and no count; and removing is not flagging: the hours vanish from every total.
  3. Falls short: EIA's adjusted file has no blanks: the gaps are the hours you flagged. Which method, and how many hours?
  4. Falls short: Which zone, and what happened to the 23- and 25-hour days?
  5. Falls short: Raw or cleaned? Here both round to 299 TWh (299.48 and 298.56), so only a log can say which.

Strong answer · Proficient (audit trail)

Source: EIA Form EIA-930 six-month BALANCE files, balancing authority SWPP, 2025, accessed 2 November 2026, SHA-256 fac4bb99… and 1146158b… [1]. Extraction: EIA's adjusted columns; each fuel's columns summed, negative values set to 0 [2]. Licence: public domain, cited with the date [3]. Time: rows kept in UTC, hour ending; results labelled in Central time with daylight saving [4]. Spikes: demand under half the median (16,541 MW): 1 hour, 21 June, 1,505 MW [5], flagged as implausible; to check against SPP's records. Frozen: runs of 3 or more identical non-zero readings: demand 698 h, wind 671 h, solar 1,377 h [6]; suspected stuck feeds, as in the course's data build. Fill: the same UTC hour on valid days within 7, 21 or 45 days, to keep the daily shape. Effect: annual demand 298.56 TWh (raw 299.48); lowest hour 24.4 GW (raw 1.5) [7]; leaving the flagged hours out would give 275.09 TWh [8]. Known limits: 65 night hours still show solar above 5% of its peak, 49 of them filled; a night rule comes next [9]. Rerun: the starter notebook r04-data.ipynb, with these four settings, prints the four numbers [10]. Validation: not yet; studio R5 adds the verification log and a check against an independent source [11].

  1. Works: Date and checksums let anyone confirm they hold the same file.
  2. Works: Even an extract changes values: the clipping belongs in the audit trail.
  3. Works: The licence is checked and its request (an acknowledgment with a date) is met.
  4. Works: No hour is lost or doubled, and every clock time on the page is named.
  5. Works: Rule, threshold, count and the value itself.
  6. Works: Counts per column, so the reader sees what each rule flagged.
  7. Works: Raw against cleaned: the reader sees what cleaning changed.
  8. Works: The rejected alternative and its effect: the choice is defended, not just made.
  9. Works: A check the rules fail, reported rather than hidden.
  10. Works: Anyone can rebuild the numbers from the note and one file.
  11. Works: Proficient for R4's part, the audit trail: the rubric's verification log and validation come in R5.
Weak answer Strong answer
Provenance “EIA” Form, area, year, access date, checksums, extraction, licence
Each step A verb Rule, threshold, count, and a flag called provisional
Effect One rounded number Raw against cleaned, and the rejected alternative
Limits and rerun None A failed check reported; a notebook that rebuilds the numbers
Defend, not just clean
For each step: the rule, the count, the reason, and what the result would be with the alternative.
Flag, then fill
Never delete silently: flag by a stated rule, then fill or leave out, and log which. A flag is a suspicion until other evidence confirms it.
Check the result
Run a check the rules did not make, such as solar at night, and log what still fails.
Time
Keep the rows as published (UTC, hour ending) and change only the labels.
Licences
Public domain still asks for a citation. Share-alike terms can differ for a database you derive and for a map made from it: check which apply to your output.
Go deeperWhy this page does not fill demand from the forecast

The course's own data build fills flagged demand hours from SPP's day-ahead forecast, rescaled to the demand around each gap, which gives a lowest hour of 23.7 GW.

This raw file carries no forecast column, so here every column uses the same-hour rule, which gives 24.4 GW.

Two defensible fills, two answers: that difference belongs in the note.

Go deeperWhat this studio leaves out
  • Metadata standards and data papers: the FAIR principles (findable, accessible, interoperable, reusable) ask that machines, not only people, can find and read your data.
  • Personal data: meter readings, survey answers and addresses need consent and protection before any cleaning (R9).
  • Revisions: agencies revise past data; the access date and checksums in your note are what make a revision visible.

Check your own work against the rubric

Competence Developing Proficient (the bar) Strong
Trustworthy analysis
Look for: Every cleaning step is logged with its rule, count, reason and effect, so another person can rerun it from the note alone. R4 covers this audit trail; R5 adds the verification log and validation.		 Results cannot be rerun. Reruns from a clean clone (or a complete audit trail); verification log; one validation against real data or an independent reference. Independent reproduction by someone else; validation on data not used to build the analysis.
Integrity
Look for: The licence of every source is checked and stated, with the attribution lines it asks for.		 Licences or credit unchecked. Licences checked, credit and conflicts stated, stakeholders treated fairly. Data and code released with documentation where allowed.
Go deeper · ResearcherFrom a data note to a verification log

The data note is the first page of a study's audit trail.

The next studio (R5) adds the rest: tests that fail on a planted bug, a benchmark against a known case and a rerun from a clean copy.

The Modeler checkpoint (CK-M) asks for both on an unseen dataset.

Sources and what’s next

Tools and data named on this page

Tool or source What it is for Link
Raw SPP 2025 file Demand, wind and solar in MW, hour by hour: the EIA-adjusted extract, before this studio's cleaning (the same file as B2). JSON · from EIA-930, downloaded 2 November 2026
R4's starter notebook This page's rules in pandas: your four numbers, the sensitivity table, the night check, the data note, and the spatial practice for Ford County, Kansas. Needs only the raw file. .ipynb
B2's notebook Loading, checking and plotting the same file, with the night rule; it also needs town_year.json. .ipynb
QGIS Free desktop GIS for the spatial practice: reproject, buffer, measure areas. qgis.org
EIA-930 six-month files The source files, with their checksums in the JSON's metadata. eia.gov · as of 2 November 2026
U.S. Wind Turbine Database Turbine locations for the spatial step; public domain, acknowledgment requested. usgs.gov · v9.1, checked 3 November 2026

Next

R5 · Can you trust your analysis? Verification, validation and reproducibility

Sources and further reading

Spotted an error, or stuck? Write to hlli@ksu.edu. Corrections are made in place and dated here.

Text licensed CC BY 4.0, code MIT: teachers and students may reuse and adapt it, with credit to Honglin Li.