Difference-in-differences (DiD) measures the incremental effect of a marketing campaign by comparing the change in outcome for treated units against the change for untreated controls, isolating the causal lift from everything else moving in the market. The output analysts should report is the Average Treatment Effect on the Treated (ATT) alongside a 95% confidence interval, not a bare percentage. DiD suit geo tests, regional rollouts, and store-level launches where sales, signups, or revenue can be tracked across a pre-period and a post-period. Before trusting any of it, run the pre-trend check first.
TL;DR:
- DiD requires a proper panel with multiple pre- and post-period observations across treated and control units, ideally with at least three pre-periods for stronger validity.
- Use modern estimators like Callaway-Sant'Anna or Sun‑Abraham for staggered campaign launches, as traditional TWFE regressions can produce biased or misleading effects.
- Run diagnostics such as pre-trend visualization, placebo tests, and spillover checks to verify the parallel trends assumption before trusting the estimated effect.
- Report the Average Treatment Effect on the Treated with its 95% confidence interval, translating it into dollar impact to enable actionable insights.
- Automation platforms like Cassandra help scale DiD analysis across multiple markets, but human judgment remains essential for evaluating the plausibility of assumptions.
Table of Contents
- DiD Intuition and the Basic 2x2 Formula
- When Difference in Differences Marketing Analysis Fits Your Campaign
- Assumptions and Diagnostics That Make or Break a DiD Result
- Modern Estimators for Staggered Rollouts and Uneven Effects
- Running a Campaign DiD Analysis Step by Step
- Where DiD Fits in a Real Measurement Stack
- A Platform Path for Scaling Incrementality Testing
- Sources
- FAQ
DiD Intuition and the Basic 2x2 Formula
Picture two lines on a chart: one tracking sales in a market that got the campaign, one tracking a market that did not. If both lines move in parallel before launch, and then the treated line bends upward after launch, that bend, not the total change, is your signal. DiD work by differencing twice: once across time, once across groups. That double subtraction cancels out anything constant within a market (baseline demand, seasonality patterns shared by both) and anything that shifted for everyone at once (a holiday, a category-wide price hike).
The regression version of that logic is:
outcome_it = α + β·treated_i + γ·post_t + δ·(treated_i × post_t) + ε_it
Here, δ is the coefficient that matters. It is the ATT: the incremental effect attributable to the campaign, holding group-level and time-level differences constant.
treated_iflags whether a unit ever received the campaign.post_tflags whether the observation falls after launch.- The interaction term isolates the effect unique to treated units after treatment.
A quick example: if treated stores went from $50,000 to $58,000 in weekly revenue (+$8,000) while control stores went from $50,000 to $52,000 (+$2,000), the DiD estimate is $6,000. Building an event-study plot, one coefficient per pre and post period, shows whether that $6,000 built gradually or landed all at once.
When Difference in Differences Marketing Analysis Fits Your Campaign
DiD work when you have a panel: repeated observations of the same units (markets, stores, customer cohorts) over time, with an outcome like sales, signups, or revenue attached to each. The method fails quietly when that structure is missing, so match your scenario before you run anything.
- One-time rollouts. A single launch date across all treated units calls for the classic 2x2 comparison.
- Matched-market geos. Comparing similar DMAs or regions where only some received a campaign works well when markets share pre-period trends.
- Staggered rollouts. Campaigns that launch in different markets at different times need a different estimator, covered below.
- Dose or continuous treatment. Spend levels that vary by market (not just on/off) call for continuous DiD specifications.
Minimum data requirements: a proper panel layout, at least one pre-period and one post-period observation (three or more pre-periods is far more convincing), and a genuine set of untreated units. Skip DiD entirely if there is no control group, if a large shock hit everyone around launch (a competitor's price war, a macro event), or if treated and control markets bleed into each other through shared customers or overlapping media.
Assumptions and Diagnostics That Make or Break a DiD Result
Parallel trends is the assumption the whole method rests on: absent the campaign, treated and control units would have moved together. It cannot be proven, only argued for, and the Columbia Population Health Methods program treats it as the central plausibility check in any DiD design, the same framework that underlies Card and Krueger's landmark 1994 minimum-wage study, which found a 2.75 full-time-equivalent employment increase per store using this exact logic.
Run these diagnostics before you trust an ATT:
- Pre-trend visualization. Plot outcomes for both groups across every pre-period; parallel lines support the assumption, diverging ones undercut it.
- Placebo and falsification tests. Assign a fake launch date before the real one and confirm the estimated effect is statistically indistinguishable from zero.
- Lead-effect checks. Look for effects showing up before launch, a sign of anticipation or a confound, not causation.
- SUTVA and spillover checks. Verify treated and control units aren't influencing each other; redefine unit boundaries or add buffer zones if they are.
Passing a pre-trend test is necessary, not sufficient. Broader difference-in-differences methodology documents cases where visually parallel pre-trends still mask a weak test with low statistical power, which is why sensitivity analysis (the Rambachan and Roth framework is the standard reference) matters when your pre-period is short or noisy.
Pro Tip: Report the ATT with its 95% confidence interval and a plain-language translation into revenue, never a p-value alone. A stakeholder can act on "$6,000 more per store per week, plausible range $2,000 to $10,000." Nobody can act on "p = 0.03."
Cluster standard errors at the market level, not the observation level, and add covariates only when they improve precision without absorbing the treatment signal itself.
Modern Estimators for Staggered Rollouts and Uneven Effects
Two-way fixed effects (TWFE) regression, the default DiD workhorse for decades, breaks down once campaigns launch in different markets at different times. The problem is what researchers call the "forbidden comparison": TWFE can end up comparing already-treated units to other treated units as if one were a valid control, sometimes assigning negative weight to genuine treatment effects. World Bank impact evaluation researchers have documented how this bias can flip the sign of an estimated effect entirely.
Two estimators built specifically to fix this now anchor marketing analysis for staggered launches:
- Callaway‑Sant'Anna computes a separate ATT for each cohort (grouped by when they were treated) and time period, then aggregates those cohort-time estimates into a single summary you can present to stakeholders.
- Sun‑Abraham builds an interaction-weighted event-study, useful as a robustness check against Callaway‑Sant'Anna and for visualizing how effects evolve after launch.
- Synthetic DiD fits better when only a handful of markets received treatment and a weighted synthetic control needs to be constructed.
- Continuous DiD apply when treatment intensity varies by market, spend level rather than simple on/off.
The practical rule: run at least one modern estimator alongside an event-study plot before reporting any staggered-rollout result. Naive TWFE alone is no longer defensible once launch dates differ across markets.
Running a Campaign DiD Analysis Step by Step
A clean DiD analysis follows a predictable sequence, and skipping steps is where most marketing measurement projects go wrong.
- Assemble the panel. You need a unit ID, a time period, the outcome variable, a treatment indicator (or first-treated period for staggered designs), relevant covariates, and a clustering variable. Aim for three or more pre-periods when the campaign calendar allows it.
- Pick the model. Simple one-time launch across all markets: 2x2 DiD. Staggered dates: Callaway‑Sant'Anna, cross-checked with Sun‑Abraham. Few treated units: Synthetic DiD. Varying spend levels: continuous DiD. The diff-diff practitioner decision tree maps these scenarios directly to estimator names.
- Run diagnostics in order. Pre-trends, then placebo tests, then covariate balance, then spillover checks, then sensitivity analysis. A failed pre-trend check means narrowing the control group or adding a synthetic control before trusting anything downstream.
- Translate ATT to dollars. Multiply the ATT by the exposed population and the average value per conversion, then propagate the confidence interval through that same multiplication so the dollar range stays honest.
- Report it clearly. Lead with ATT and confidence interval, state the parallel trends assumption plainly, show the event-study plot, and list every diagnostic performed.
Applied incrementality research built on this logic, including PIE's use of RCTs to predict effects for non-experimental campaigns, reports out-of-sample R² near 0.88 for incremental conversions per dollar, compared with roughly 0.19 for last-click attribution, a gap that shows how much signal DiD-style causal designs recover that standard attribution simply misses. Converting a validated ATT into a per-conversion dollar figure is the same math behind most incremental lift calculations marketing teams already run.
Where DiD Fits in a Real Measurement Stack
DiD and geo-lift experiments give you something Marketing Mix Models cannot generate on their own: a causal anchor. An MMM regresses spend against outcomes across time, but without an experimental benchmark, it is fitting correlations. A validated DiD estimate from a regional rollout lets you calibrate MMM coefficients against a number you actually trust.
What automation changes is scale, not judgment. A platform can run pre-trend checks, placebo tests, and spillover diagnostics across dozens of markets continuously, integrating ad account and sales data without a manual pull every quarter. It cannot decide whether your parallel trends assumption is plausible for a specific category shock, that still takes an analyst who understands the market. The platform pairs automation with human-reviewed experiment design, not by replacing the judgment call.
— Gabriele Franco
A Platform Path for Scaling Incrementality Testing
Running one clean DiD analysis on a single market rollout is a manageable analyst project. Running the same rigor across twenty markets, every quarter, with staggered launches and shifting media mix, is where most in-house teams hit a wall. Cassandra was built to close that gap: it connects ad accounts and sales data, runs geo-lift experiments continuously, and reports incrementality with the same ATT-and-confidence-interval discipline this article just walked through, then feeds those results into MMM calibration automatically.
A platform earns its keep when you're managing many markets at once, need always-on monitoring rather than a one-off study, or have to hand stakeholders a dashboard instead of a spreadsheet every week. Bespoke, hand-coded analysis still has a place for a single high-stakes test with unusual data quirks. For everything else, Cassandra's Incrementality Testing product runs the geo-experiment design and the diagnostics for you, while the Performance Incrementality Dashboard turns the output into numbers a CFO can act on. Plans start with Essentials from €3,000 per month; full details sit on the pricing page. If you're running campaigns across enough markets that manual DiD checks eat a full week every month, that's the signal to request a walkthrough of the platform.
Sources
For deeper implementation guidance, consult the diff-diff decision tree, the Callaway‑Sant'Anna implementation guide, and the original PIE research on scaling incrementality beyond RCTs.
- Difference-in-difference estimation (Columbia University)
- Did you see beta beta who? Part 2 (World Bank blog)
- Predictive Incrementality by Experimentation (PIE) for ad measurement (Kellogg)
FAQ
What Is a Difference-in-Differences Analysis?
DiD is a quasi-experimental method that compares the change in an outcome for a treated group against the change for a control group over the same period. It cancels out shared time trends and fixed differences between groups, producing an ATT that isolates the campaign's incremental effect, as Columbia's methods primer lays out.
When Should You Use DiD Instead of Another Method?
Use DiD when you have panel data with at least one pre-period and one post-period, a plausible control group, and no major concurrent shocks hitting only one side of the comparison. It fits geo tests, regional rollouts, and store-level launches particularly well; it fits poorly when spillovers or missing controls make the comparison unreliable.
How Do You Interpret DiD Results for Stakeholders?
Report the ATT as a dollar or unit figure with its 95% confidence interval, then explain what the parallel trends assumption means for that specific market pair. Show the event-study plot and list which diagnostics (pre-trend, placebo, spillover) passed before presenting the number as final.
What Is the Difference-in-Differences-in-Differences (DDD) Model?
DDD adds a third comparison dimension, often a subgroup within treated and control units, to control for a confound that a standard two-group DiD can't isolate on its own. It's used sparingly, mainly when a single control group isn't credible and a second layer of differencing can strip out an additional source of bias.
Can a Measurement Platform Run DiD Analysis Automatically?
Platforms can automate data integration, run geo-lift experiments continuously, and surface diagnostics like pre-trend checks at scale, which is where tools like Cassandra's Incrementality Testing product fit. Judgment calls, like whether the parallel trends assumption holds for a given market pair, still require an analyst reviewing the output.