Did it work? Learning from data with regression and natural experiments
Estimate a tariff’s effect on simulated homes whose true effect is known, then break difference-in-differences on purpose with selection, confounding, spillovers and anticipation
Simulated panels with known effects: what each estimate says
- Simulated: 400 homes, 24 months; 92 opted into a time-of-use tariff from month 13. The true effect on them is −0.29 kW on the evening peak (known because the data are simulated).
- After the start, TOU homes against flat homes: −0.85 kW. With home size as a control: −0.73 kW. Engagement, never measured, still differs.
- TOU homes after against before: −0.36 kW, because flat homes also changed by −0.06 kW (a cooler second summer).
- Difference-in-differences: −0.30 kW (about 95% interval −0.33 to −0.27 kW), against the truth −0.29 kW. Before month 13 the change in the TOU–flat gap, against month 12, stays within 0.04 kW of zero.
- Threats move it: already cutting −0.44 kW, early cuts −0.26 kW, neighbours copy −0.22 kW, a thermostat offer −0.42 kW.
- If every home were put on the tariff by default, the average effect would be −0.20 kW, not −0.29 kW: those who opt in respond more.
- Simulated regions: 14 of 30 adopt a policy in 2021. DiD: 10.0 MW a year of new solar per adopting region (truth +10.0); if the policy followed a boom, 22.5 MW a year.
- If projects moved across the border, DiD reads 14.0 MW a year. Adopters then gain 140 MW a year: 64 moved from the other regions and 76 is new.
| Estimate of the tariff's effect | kW |
|---|---|
| Truth (homes that opted in) | −0.29 kW |
| Naive after-period gap | −0.85 kW |
| Regression with home size | −0.73 kW |
| Before against after, TOU homes only | −0.36 kW |
| Difference-in-differences | −0.30 kW |
With JavaScript on, you build these estimates step by step and break them on purpose.
- Simulated: every home, tariff, region and policy on this page is made up by a seeded generator in your browser, with a true effect set in advance. Nothing is real data, and nothing is downloaded.
- Why simulate: real studies never see the truth. Here you can, so you can score each method against it.
- Seeds: 78 for the homes, 233 for the regions, each picked among the first few hundred for a calm default draw (a teaching choice). The notebook redraws 200 panels to show the spread.
- Outcome: a home's evening peak, its average daily highest hourly demand from 16:00 to 21:00 in a month, in kW (kilowatts).
- Real evidence: only the DOE report cited under Sources (November 2016).
About 40 guided minutes in ten steps. Everything is computed in your browser; nothing is sent anywhere. Need a refresher on standard errors? K5.
Build it in Python · notebook lab, about 60 minutes, then 90 minutes of practice
The same two panels in NumPy, with assert cells against the page. Then:
- the two-way fixed-effects and event-study regressions, each checked against the page's calculation;
- standard errors clustered by home;
- 200 redrawn panels;
- five practice tasks, with starter code for the harder ones.
- Download the notebook and the lesson settings into one folder (in Colab, upload the settings file with the folder icon).
- Open it in Google Colab (File → Upload notebook) or locally with Python 3.9 or later and NumPy; matplotlib is optional.
- Run all cells (under a minute on a 2025 laptop), then try the practice tasks at the end.
What you just did, as diagrams
- Counterfactual
- An effect is a comparison with what would have happened without the change; it is never observed.
- Selection
- People who choose a programme differ from those who do not, before it starts.
- Difference-in-differences
- Compare the change in the treated group with the change in a comparison group.
- Parallel trends
- The assumption that both groups would have moved alike; the months before can make it plausible, never prove it.
- Threats
- Selection on trends, confounding, spillovers and anticipation each move the estimate away from the truth.
- For whom
- An effect on volunteers is not the effect on everyone.
Go deeper · ModelerThe formulas: DiD, the two-way regression, the event study and standard errors
| Symbol | Meaning |
|---|---|
| \(i\), \(t\) | a home; a month, 1 to 24 (the tariff starts in month 13) |
| \(y_{it}\) | home \(i\)’s evening peak in month \(t\), kW |
| \(D_{it}\) | 1 if home \(i\) is on the tariff in month \(t\), else 0 |
| \(\mathbf{1}[\cdot]\) | 1 when the condition in the brackets holds, else 0 |
| \(\tau\) | the tariff’s effect, kW |
| \(\alpha_i\), \(\lambda_t\) | one level for each home; one level for each month |
| \(\varepsilon_{it}\) | what the model leaves unexplained (noise), kW |
| \(\bar y_{T,\text{post}}\) | the mean over TOU homes (\(T\)) in months 13 to 24 (post); \(C\) flat homes, pre months 1 to 12 |
| \(n_T\), \(n_C\) | the numbers of TOU and flat homes: 92 and 308 |
| \(\Delta_i\), \(s_T^2\), \(s_C^2\) | home \(i\)’s change, after minus before; the variance of the changes in each group |
| \(\beta_k\), \(k\) | the change in the gap \(k\) months from the start (\(k = t - 13\)) |
- DiD of means. \(\hat\tau = (\bar y_{T,\text{post}} - \bar y_{T,\text{pre}}) - (\bar y_{C,\text{post}} - \bar y_{C,\text{pre}})\). On this page: \(-0.36 - (-0.06) = -0.30\) kW (rounded).
- Two-way fixed effects. \(y_{it} = \alpha_i + \lambda_t + \tau D_{it} + \varepsilon_{it}\), one level \(\alpha_i\) per home and one \(\lambda_t\) per month. With one start date and every home seen every month, its \(\hat\tau\) equals the DiD of means; the lab checks this.
- Event study. Replace \(\tau D_{it}\) by \(\sum_{k \ne -1} \beta_k \,\mathbf{1}[\text{TOU}_i,\ t - 13 = k]\). Before the start (\(k < -1\)) the \(\beta_k\) should sit near zero.
- Computed or fitted. With one start date and every home seen every month, each \(\beta_k\) equals that month’s DiD against month 12. The page computes it that way; the lab fits the regression and checks that the two agree.
- Standard error. \(\text{SE} = \sqrt{s_T^2/n_T + s_C^2/n_C}\), from each home’s change \(\Delta_i = \bar y_{i,\text{post}} - \bar y_{i,\text{pre}}\).
- Clustering. A home’s months are not independent, so the home, not the month, is the unit: this is what clustering by home does in the regression.
- Interval. Estimate ± 1.96 SE: about 95% of such intervals hold the truth when the design’s assumptions hold (K5). When they fail, a narrow interval misses: step 7.
- Start dates that differ. When units start at different dates, the two-way regression mixes comparisons, some of them against units already treated. Goodman-Bacon (2021) and Callaway and Sant’Anna (2021) set out the problem and estimators for it (Sources).
Go deeper · ResearcherRandomized trials and natural experiments
- Randomized trial. Chance decides who gets the tariff, so the groups differ only by chance before the start. DOE’s consumer behaviour studies used randomized designs for this reason (Sources).
- Randomized encouragement. When people cannot be forced onto a tariff, chance decides who is invited or defaulted; the comparison is between the invited and the not invited.
- Natural experiment. An outside rule or event creates a comparison: a start date by postcode, a threshold, a border. For DiD, explain why the untreated trends would have matched, and check for anticipation, spillovers and other changes at the same time.
- Pre-register. Write the comparison, the outcome and the threats down before looking at the after-period data.
Check yourself
Myth or fact? “Add enough control variables and a regression gives the causal effect.”
- Myth
- Fact
Answer
Myth. Controls fix only measured differences. On this page home size and air conditioning moved the estimate from −0.85 to −0.70 kW; the truth was −0.29 kW, and engagement was never measured.
Myth or fact? “Flat months before the start prove the DiD estimate is right.”
- Myth
- Fact
Answer
Myth. With neighbours copying or a thermostat offer, the months before stayed flat while DiD read −0.22 and −0.42 kW against a truth of −0.29. Similar earlier trends support a comparison; they never prove it.
Myth or fact? “An effect measured on volunteers tells you the effect of making the tariff everyone's default.”
- Myth
- Fact
Answer
Myth. Here volunteers cut 0.29 kW, the average home would cut 0.20. DOE's studies reported peak cuts of 12% for opt-in TOU customers and 6% for opt-out (2016).
Treated homes' peak fell from 2.00 to 1.70 kW; comparison homes' from 2.10 to 2.00 kW. The DiD estimate?
- −0.30 kW
- −0.20 kW
- −0.10 kW
- −0.30 kW minus the 0.10 kW gap at the start
Answer
−0.20 kW: (1.70 − 2.00) − (2.00 − 2.10) = −0.30 + 0.10.
A city's subsidy moves solar projects from the next county into the city. A DiD with that county as the comparison…
- is unbiased, since the county had no subsidy
- understates the city's gain
- overstates the net gain across both
- cannot be computed
Answer
It overstates it: the county lost projects, so the comparison group falls and the gap grows. The net change across both is smaller (step 9).
Numbers, sources and what’s next
Numbers to remember
| Number | What it means | Source |
|---|---|---|
| −0.30 kW | The DiD estimate on the simulated homes; the truth was −0.29 kW, the naive gap −0.85 kW. | This lesson's simulation · seed 78 |
| 12% and 6% | Average TOU peak-period cuts reported for opt-in and for opt-out (default) customers in DOE's consumer behaviour studies. | DOE (PDF), p. vi · Nov 2016 |
| 6% and 18% | Average TOU peak-period cuts for peak to off-peak price ratios below 2:1 and above 4:1. | DOE (PDF), Table ES-2, p. viii · Nov 2016 |
Keep going
Where this came up in practice
Under the Smart Grid Investment Grant, ten utilities ran eleven consumer behaviour studies of time-based rates, applying randomized and controlled designs to estimate customers' responses more precisely and credibly (DOE, November 2016, pp. iv–v).
Opt-out enrolment was far higher than opt-in (92% against 15%), peak-period cuts were generally lower (6% against 12% for TOU), and retention was about the same (90% against 87%) (p. vi).
A selection example (same report, Table ES-2, p. viii): in one utility's opt-in TOU study, homes with in-home displays cut their peak somewhat more than homes without.
The report says pre-treatment differences between the two groups can explain that gap.
Sources and further reading
- U.S. Department of Energy, Smart Grid Investment Grant Program (2016). Final Report on Customer Acceptance, Retention, and Response to Time-Based Rates from the Consumer Behavior Studies (PDF), November 2016: executive summary, pp. iv–viii (accessed 6 Nov 2026).
- Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277.
- Callaway, B., and Sant'Anna, P. H. C. (2021). Difference-in-Differences with multiple time periods. Journal of Econometrics, 225(2), 200–230.
- Further reading (free online textbooks, by name): Cunningham, S., Causal Inference: The Mixtape (2021); Huntington-Klein, N., The Effect (2021), chapters on difference-in-differences and event studies.
Spotted an error, or stuck? Write to hlli@ksu.edu. Corrections are made in place and dated here.
Text licensed CC BY 4.0, code MIT: teachers and students may reuse and adapt it, with credit to Honglin Li.