Researcher checkpoint: defend your question and method, then revise
About four hours: review a stranger’s study, get a written review of your own, answer it, revise, and rerun your main result from a clean clone
How this checkpoint works · about 240 minutes, no hints
- A · A stranger's study (about 45 min). Review the co-op proposal below with the review form and submit it as task 1; tasks 2 to 5 open after it.
- B · Your mock review (about 60 min). Get a written review of your research capstone's question and method: from a mentor, a peer, or yourself after a day away.
- C · Respond and revise (about 70 min). Answer each concern point by point; revise your proposal and log each change.
- D · Rerun (about 50 min). Order and check the co-op study's rerun (tasks 9 to 11), then rerun your own main result from a clean clone.
- Submit (about 15 min). Enter your results in the twelve tasks below, in any order. The worked review, response and revision open once all twelve are answered.
When to take it: once your research capstone has a proposal and a main result. A review from someone else can take days: ask for it first, and do part A while you wait.
How it is marked, before you start
- Proficient: at least 5 of the 7 auto-marked tasks right on the first try, plus at least 3 of the 4 points of each written task (1, 6, 7, 8 and 12), which you tick yourself.
- Strong: at least 6 of 7 on the first try, and every written point.
- Opening solutions early: a task whose answer you open before answering counts as not right. Opening the worked solution before all twelve are answered makes the attempt practice, with no level.
- After the worked solution opens: an answer given after that counts as practice, not as right, and a written task then earns no points.
- Task 1 first: the seven marked tasks open once your review is submitted, and it then stays as written. A review submitted after a marked task was answered earns no points.
- The level is not the table's: it comes from this rule. The criterion table in the rubric section rates the work itself; its Strong asks for a review and a rerun by someone else.
Refreshers: R1 (framing), R3 (method), R5 (reruns), R8 (findings with limits), R9 (integrity), R10 (the question card), M14 (difference-in-differences), K8 (clones).
Part A · the proposal to review
Did a night-pumping tariff move irrigation out of the co-op's peak?
A graduate student's proposal, with a first result, for the board of Prairie Valley Co-op.
- Question
- Did Prairie Valley Co-op's 2024 night-pumping tariff lower irrigation peak demand?
- Type
- Evaluative.
- Audience
- The co-op's board, which votes in 2026 on keeping the tariff.
- The tariff
- Cheaper energy from 22:00 to 06:00 and a higher price from 14:00 to 20:00 on summer weekdays. Each farm chose whether to join.
- Data
- Hourly meter readings of the co-op's 112 irrigation pumps, summers 2020 to 2024.
- Metric
- Peak demand, kW per pump.
- Method
- Difference-in-differences: the 41 pumps that joined in 2024 against the 71 that did not, before and after.
- Literature
- Five studies of time-of-use tariffs for households.
- Checks
- Run once, on data the author cleaned by hand in a spreadsheet.
- First result
- Peak demand fell 6.2 kW per pump.
The co-op, its farms and every number here are fictional: the results are assumed example outputs, not computed from data.
The forms
Four plain templates: one for a review, one for a response, a change log and a rerun report. Use them for the co-op proposal and for your own study.
The review form (parts A and B)
1. The study in two sentences: question, method, data. 2. The main claim it will make, and the evidence it plans. 3. Major concerns (at most three): ones that could change the conclusion. For each: the rubric competence, and what evidence or change would answer it. 4. Minor concerns. 5. One alternative method from another family, and what it would show that this one cannot. 6. Can the main result be rerun from the repository? What is missing? 7. Recommendation: proceed; proceed once the major concerns are answered; or rethink the question.
The response (part C)
For each concern, in the reviewer's order: Concern 1 (competence): accepted / partly accepted / declined. Reason: one or two sentences. Change: what changed, and where (a section of the proposal, a file in the repository). Evidence: a test, a figure or a number, where you have one.
The change log (part C)
One line per change: Before → after · the concern it answers · where. Then: your rubric self-rating after the revision, competence by competence, and any level that moved, with its evidence.
The rerun report (part D)
Repository and commit; date; machine and environment. Tolerance for the main result, written down before the rerun. Reported value; rerun value; difference as a share of the reported value (in the result's units if that value is near zero). Qualitative work: the audit trail followed from the raw material, each theme traced to its sources, coding decisions and discrepancies explained. Cause of any difference, and the fix in the repository. Who reran it (you, or someone else).
Words used in the tasks: clean clone, lock file, checksum, 95% interval.
Reviewing a peer's proposal: keep it confidential and do not use its ideas without permission (ALLEA 2023, §2.8). Reviewing your own: wait a day, and read it as someone who has not seen it.
Without JavaScript nothing here is locked, and the worked solution is further down the page. Write task 1, your review, before you open the marked tasks: their questions name concerns your review should find. The level rule is in the rubric section.
Your review of the co-op proposal Part A · Write a review
Review the co-op proposal above with the review form: the study in two sentences, at most three major concerns, the minor ones, and what would answer each. Paste your review.
Self-rated against 4 points, shown after you submit.
The most serious concern Part A · Rank the concerns
Open after task 1, your review
Which concern could most change the study's conclusion?
- a. Only five studies are cited, all of households, so the map of what is known is thin.
- b. Each farm chose whether to join, so the joiners may differ from the rest in ways that also change their pumping.
- c. The proposal does not name the software, or its version, that ran the regression.
- d. The board votes on the tariff in 2026, which leaves the study little time to finish.
Whose peak? Part A · Name the competence
Open after task 1, your review
A reviewer writes: “The proposal never says whose peak: each pump's own highest hour, or the co-op's highest-load hours.” Which rubric competence does this concern?
- a. Framing
- b. Positioning
- c. Integrity
- d. Communication
Answering the selection concern Part A · Answer a critique
Open after task 1, your review
The authors answer the selection concern. Which response answers it, rather than deflecting it?
- a. Self-selection affects most observational studies, so we list it as an accepted limitation in section 6.
- b. We will add more pumps from this summer to the sample, so that the estimate is more precise.
- c. We now compare the 2024 joiners with the 29 farms that joined in 2025, before they joined, and show both groups' four summers before the tariff.
- d. The reviewer has misunderstood the tariff: farms join it for reasons that have nothing to do with pumping.
From Proficient to Strong Part A · Move a competence up a level
Open after task 1, your review
The revised study is Proficient on Method fit: difference-in-differences, defended against a matched comparison and a pump-scheduling simulation. Which change moves it to Strong?
- a. Add a third alternative method, an agent-based model of the farmers, to the methods compared.
- b. State the method's main limit here (the later joiners' demand might still have moved differently) and partly mitigate it: show that both groups' peak-hour demand moved together in the four summers before 2024.
- c. Replace difference-in-differences with a machine-learning model trained on every pump's hourly readings.
- d. Write a longer method section that explains difference-in-differences step by step.
Your mock review Part B · Get a review
Get a written review of your research capstone's question and method (the review form; a mentor, a peer, or you after a day away). Paste it, and say who wrote it.
Self-rated against 4 points, shown after you submit.
Your response Part C · Answer the critique
Answer each major concern point by point: accepted, partly accepted or declined, with a reason, and for each accepted one the change and where it is made. Paste your response.
Self-rated against 4 points, shown after you submit.
Your revision Part C · Revise and log
Revise your proposal. Paste your change log (before → after, and the concern each change answers) and your rubric self-rating after the revision.
Self-rated against 4 points, shown after you submit.
A clean-clone rerun Part D · Rerun from a clean clone
Open after task 1, your review
After the review, the authors rerun the revised co-op study from a clean clone. Each step needs the one before it. Put the steps in order.
- Compare the main result with the reported value, against the tolerance written down beforehand.
- Build the environment from the repository's lock file.
- Run the whole pipeline, from raw data to the main result, with one command.
- Clone the repository into a new, empty folder.
- Log the outcome, and any difference with its cause, in the verification log.
- Fetch the raw data with the repository's script, which runs in that environment, and check the checksums.
Put these in order.
How far apart? Part D · Report a difference
Open after task 1, your review
Reported: a fall of 4.6 kW per pump. The clean-clone rerun: 4.3 kW. How far apart are they, as a share of the reported value?
Within 2% of the value (about 0.13 percentage points) counts as right.
Answer with a number and a unit.
What to report Part D · Report it honestly
Open after task 1, your review
The tolerance written down before the rerun was 1%. What should the authors do?
- a. Report 4.6 kW, the published value: a gap of 0.3 kW is small next to the interval.
- b. Report 4.3 kW instead, since the clean clone is the run to trust, and drop 4.6.
- c. Find the cause, turn any hand step into code with a stated rule, rerun from a clean clone, and log both numbers, the cause and the fix.
- d. Widen the tolerance to 10%, since 1% was too strict for real meter data.
Your rerun Part D · Rerun your own result
Rerun your own main result from a clean clone, or follow your audit trail for qualitative work. Paste your rerun report (the form above).
Self-rated against 4 points, shown after you submit.
The co-op study is fictional; its results are assumed example outputs, not computed from data. Your own study is your research capstone. Nothing you type leaves this browser unless you copy it; the copied summary includes what you typed in the five written tasks.
Worked solution
Opens once all twelve tasks are answered. The checkpoint asks for your own review, response, revision and rerun first, so the worked examples wait for them. Opening them now makes this attempt practice, with no level.
Two reviews of the co-op proposal
Review the co-op proposal: the study, the major and minor concerns, and what would answer each.
A weak review · Developing
Summary: An interesting study of a new tariff. Concerns: The literature review is thin. [1] The method seems fine. [2] Figure 2 is hard to read. [3] Recommendation: accept with minor changes. [4]
- Falls short: A minor concern led with as if it were the main one; no competence named, and nothing that would answer it.
- Falls short: Each farm chose whether to join: the comparison is untested.
- Falls short: Format, not substance: it cannot change the conclusion.
- Falls short: A verdict without reasons the authors can act on.
The worked review · Strong
Summary: asks whether the 2024 tariff lowered the pumps' peak demand, by difference-in-differences on 112 pumps' meter readings, 2020 to 2024. [1] Major 1 · Method fit: farms chose whether to join, so their demand may have moved differently even without the tariff; show the summers before, and try farms that joined later as a comparison. [2] Major 2 · Framing: say whose peak, in kW, over which hours: the board pays for the co-op's highest-load hours. [3] Major 3 · Trustworthy analysis: the data were cleaned by hand in a spreadsheet, so the result cannot be rerun from the repository. [4] Minor · Positioning: the five studies are of households; add studies of farm or business tariffs. [5] Another family: a pump-scheduling simulation would show how much pumping could move at most, which meter data cannot. [6] Recommendation: proceed once the three major concerns are answered. [7]
- Works: The study stated fairly, before any criticism.
- Works: The concern that could change the conclusion comes first, with the evidence that would answer it.
- Works: Tied to a competence, and to what the decision-maker needs.
- Works: A result no one can rerun cannot be checked.
- Works: Kept minor: better sources would not move the estimate.
- Works: An alternative from another method family, and what it adds.
- Works: A verdict tied to the concerns.
| A weak review | The worked review | |
|---|---|---|
| Order | A minor concern first | By what could change the conclusion |
| Competences | None named | One for each concern |
| What would answer it | Not said | Said for each major concern |
| Another family | None | A simulation, and what it adds |
Both reviews are illustrative; the study is fictional.
Two answers to Major 1
Answer the concern that the farms chose whether to join.
A deflecting answer · Developing
Response: We thank the reviewer. Self-selection is a well-known limitation of observational studies, [1] and we have added it to the limitations section. [2]
- Falls short: It names the problem and keeps it.
- Falls short: A sentence moved; the analysis unchanged.
An answering response · Strong
Accepted, with thanks to the reviewer. Change: the comparison group is now the 29 farms that joined in 2025, before they joined (section 3; analysis/did.py). [1] Evidence: in the four summers before 2024, the two groups' peak-hour demand moved within 0.5 kW of each other each year (Figure 2). [2] Result: with this comparison and the metric of Major 2, the fall is 4.6 kW per pump, not 6.2. [3] Limit kept: the later joiners' demand might still have moved differently; stated in section 6. [4]
- Works: The change, and where it is made.
- Works: New evidence for the parallel-trends assumption.
- Works: The change moved the answer, and the response says so.
- Works: What the change cannot fix, said plainly.
Both answers are illustrative.
The revision: what moved it from Proficient to Strong
The revised study's question, method, checks and result.
Revised: Proficient · Proficient
Question: Did the 2024 tariff lower the joining pumps' average demand in the co-op's 20 highest-load summer hours, compared with farms that joined in 2025, before they joined? [1] Method: difference-in-differences, defended against a matched comparison and a pump-scheduling simulation. [2] Checks: reruns from a clean clone within 1%; a verification log; the pumps' summed readings checked against the meters of the feeders that serve only pumps, same hours, after line losses. [3] Result: 4.6 kW per pump (95% interval 2.9 to 6.3).
- Proficient: Typed, with an audience (the board), a metric with units and a counterfactual.
- Proficient: Two alternatives, one from another family (simulation).
- Proficient: A clean-clone rerun, a log and one check against an independent reference (feeder meters): of the measurements, not of the tariff's effect.
Revised: Strong · Strong
Question: the same, and is the fall above 1.5 kW per pump, the least for which the board would keep the tariff? [1] Method: the same. Limit: the later joiners' demand might still have moved differently. Mitigated: in the four summers before 2024 the two groups moved together, within 0.5 kW a year. [2] Rerun: a co-op engineer reran the clean clone and got 4.6 kW. [3] Validation, set up first: summer 2025, not used to build the analysis: the same code on the 29 farms that joined in 2025, against the 42 that never joined. [4] Rule and result: agree if the gap between the two estimates, treated as independent, has a 95% interval that includes zero. 2025: 4.1 kW (1.8 to 6.4); gap 0.5 kW (−2.4 to 3.4): they agree. [5] Result: 4.6 kW per pump (95% interval 2.9 to 6.3), above 1.5 kW across the interval.
- Moves it to Strong: Framing's Strong level: the answer that would change the decision.
- Moves it to Strong: Method fit's Strong level: the method's limit stated and partly mitigated.
- Moves it to Strong: Trustworthy analysis at Strong, first half: reproduced by someone else.
- Moves it to Strong: Second half: data not used to build it, a named comparison group, and a rule written down before looking.
- Moves it to Strong: Consistent with the main estimate, which supports it; on its own it does not prove the method: the never-joiners' demand may have moved differently.
| Revised: Proficient | Revised: Strong | |
|---|---|---|
| Framing | Typed question, audience, metric with units, counterfactual | Also the answer that would change the board's vote (1.5 kW) |
| Method fit | Defended against two alternatives | Also its limit stated and partly mitigated |
| Trustworthy analysis | Clean-clone rerun, log, a meter check against an independent reference | Rerun by someone else; checked on a summer not used to build it, by a rule set first |
Illustrative: the study is fictional, and its results are assumed example outputs, not computed.
The rerun, worked
- Tolerance
- 1% of the main result, written down before the rerun.
- Values
- Reported 4.6 kW; rerun 4.3 kW. Difference: (4.6 − 4.3) ÷ 4.6 = 0.3 ÷ 4.6 = 0.065, or 6.5%: over the tolerance.
- Cause
- Three pumps whose meters were replaced mid-summer had been removed by hand before the first run; the code kept them.
- Fix
- The rule is now in the cleaning script and the data note; the rerun from a clean clone gives 4.6 kW.
- Logged
- Both numbers, the cause and the fix, in the verification log. Then a co-op engineer reran it from the repository: 4.6 kW.
The seven marked tasks
| Task | Answer | Why |
|---|---|---|
| 2. The most serious concern | b. Each farm chose whether to join, so the joiners may differ from the rest in ways that also change their pumping. | Joiners chose the tariff, so their demand might have moved differently even without it (selection bias). Different starting levels alone do no harm: parallel trends needs evidence. The other three concerns are minor. |
| 3. Whose peak? | a. Framing | Framing: a metric with units. The board pays for the co-op's highest-load hours (coincident peak), so the pumps' average kW in the co-op's 20 highest-load summer hours answers its question; each pump's own peak does not. |
| 4. Answering the selection concern | c. We now compare the 2024 joiners with the 29 farms that joined in 2025, before they joined, and show both groups' four summers before the tariff. | Answer with a change and evidence. The 2025 joiners are one possible comparison; their four summers before 2024 test whether the two groups' demand moved alike (parallel trends), which matters more than equal starting levels. |
| 5. From Proficient to Strong | b. State the method's main limit here (the later joiners' demand might still have moved differently) and partly mitigate it: show that both groups' peak-hour demand moved together in the four summers before 2024. | Method fit's Strong level: “limits of the chosen method stated and partly mitigated”. Limit: the later joiners' demand might still have moved differently. Mitigation: the two groups moved together, within 0.5 kW a year, in the four summers before 2024. |
| 9. A clean-clone rerun | 1. Clone the repository into a new, empty folder. 2. Build the environment from the repository's lock file. 3. Fetch the raw data with the repository's script, which runs in that environment, and check the checksums. 4. Run the whole pipeline, from raw data to the main result, with one command. 5. Compare the main result with the reported value, against the tolerance written down beforehand. 6. Log the outcome, and any difference with its cause, in the verification log. | Clone, environment, data, run, compare, log. The download script runs in the environment, so the environment comes first; each step uses only what the repository holds, so a missing file or a hand step shows up. |
| 10. How far apart? | About 6.5%: (4.6 − 4.3) ÷ 4.6 = 0.3 ÷ 4.6 = 0.065 | (4.6 − 4.3) ÷ 4.6 = 0.3 ÷ 4.6 = 0.065, or 6.5% of the reported value: far over the 1% tolerance written down before the rerun. |
| 11. What to report | c. Find the cause, turn any hand step into code with a stated rule, rerun from a clean clone, and log both numbers, the cause and the fix. | Here the cause was three pumps whose meters were replaced mid-summer, removed by hand before the first run. The rule is now in the cleaning script and the data note; the rerun gives 4.6 kW, and the log says so. |
Rubric
Your work, criterion by criterion
| Criterion | Not yet | Developing | Proficient | Strong |
|---|---|---|---|---|
| The review | No review. | Comments on wording or format, or concerns with no competence and nothing that would answer them. | Major concerns (those that could change the conclusion), each tied to a competence and to the evidence that would answer it; the method tested against another family. | Also written by someone else, and ranked by what each concern could change in the conclusion. |
| The response | No response. | Thanks and promises with no change, or concerns declined without reasons. | Every concern answered with a reason; each accepted one names its change and where; a concern about the analysis answered in the analysis. | Also new evidence for the changes (a test, a figure), and the reviewer credited. |
| The revision | No revision. | Wording changed, the analysis not. | A before → after log; the question still passes the card's eight checks; the rubric self-rating redone. | Also each competence that moved up shown with its evidence, or, where none could, why. |
| The rerun | No rerun. | Rerun in the working folder, not a clean clone, or a difference not reported. | A clean clone (or a complete audit trail), a tolerance set first, both values and the difference reported, any cause fixed. | Also rerun by someone else from the repository, within the tolerance. |
The checkpoint's level (not the table's)
- Proficient: at least 5 of the 7 auto-marked tasks right on the first try, and at least 3 of the 4 points of each of the five written tasks.
- Strong: at least 6 of 7 on the first try, and every point of all five written tasks.
- Developing: at least 3 of 7 on the first try.
- Not yet: anything else.
- Solutions opened early: a task whose answer you open before answering counts as not right on the first try. If the worked solution opens before all twelve tasks are answered, the attempt is practice and gets no level.
- Answered late: an answer given after the worked solution opened counts as practice, not as right; a written task submitted after it earns no points.
- Task 1 first: a review submitted after a marked task was answered earns no points.
- Not the table's Strong: this level needs no second person. The criterion table's Strong (a review and a rerun by someone else) is rated apart, by you or a mentor.
The written tasks are self-rated against their points. A mentor can rate them again with the table, from the copied summary, which includes what you typed. The checkpoint certifies nothing: with your research capstone and its self-check, it is evidence you can show.
| Competence | Developing | Proficient (the bar) | Strong |
|---|---|---|---|
| Framing Look for: After the revision your question card passes its eight checks, and your response says what answer would change your audience's decision. |
Question vague or untyped; no audience or counterfactual. | Typed question with audience or decision-maker, metric with units, counterfactual, scale and boundary; one falsification statement. | Also explains why the question matters to its audience and what answer would change a decision. |
| Method fit Look for: Your response defends the method against the reviewer's alternative from another family, and states one limit you mitigated. |
Method chosen by habit. | Method defended against two alternatives, one of them a different family. | Limits of the chosen method stated and partly mitigated. |
| Trustworthy analysis Look for: Your clean-clone rerun matches within the tolerance you set first, or the difference and its cause are in your verification log. |
Results cannot be rerun. | Reruns from a clean clone (or a complete audit trail); verification log; one validation against real data or an independent reference. | Independent reproduction by someone else; validation on data not used to build the analysis. |
Numbers, sources and what’s next
Numbers used in the tasks
| Number | What it means | Source |
|---|---|---|
| 112 pumps | The co-op's irrigation pumps: 41 joined the tariff in 2024, 29 in 2025, 42 never. | Illustrative |
| 6.2 and 4.6 kW | The fall per pump in the first result, and after the revision (95% interval 2.9 to 6.3 kW). | Assumed example output, not computed |
| 4.3 kW and 1% | The clean-clone rerun before the fix (an assumed example output), and the tolerance written down before it. | Illustrative |
| 1.5 kW | The least fall per pump for which the board would keep the tariff. | Illustrative |
Keep going
Sources and further reading
- National Academies of Sciences, Engineering, and Medicine (2019). Reproducibility and Replicability in Science. The National Academies Press. Summary, p. 1:
- Reproducibility: “obtaining consistent computational results using the same input data”, code and conditions of analysis.
- Replicability: consistent results across studies of the same question, each with its own data.
- ALLEA (2023). The European Code of Conduct for Research Integrity, Revised Edition 2023. Berlin (CC BY 4.0): §2.7 Publication, Dissemination, and Authorship (prompt corrections); §2.8 Reviewing and Assessment (conflicts declared, confidentiality, permission to use the ideas reviewed).
- Pfenninger, S. et al. (2017). The importance of open data and software: Is energy research lagging behind? Energy Policy 101, 211–215.
Spotted an error, or stuck? Write to hlli@ksu.edu. Corrections are made in place and dated here.
Text licensed CC BY 4.0, code MIT: teachers and students may reuse and adapt it, with credit to Honglin Li.