An online garden shop wants to disclose shipping charges earlier in checkout. The product team expects fewer shoppers to abandon payment when they can see the cost in advance. It also worries that an early estimate could turn shoppers away before they see the pickup option. Design the test around that decision. Record who is eligible, what each version changes, and how much improvement would justify maintaining the new screen.
Freeze the experiment plan before allocation
Randomize by account so a signed-in shopper sees the same checkout on a return visit. Define eligibility as the first checkout visit during enrollment with a delivery address in the pilot area. Define the primary outcome as at least one successful payment within seven days of assignment. Count eligible assigned accounts in the denominator, including those who never pay. Deduplicate payment retries at the account level. Specify a window and a threshold for each guardrail, such as checkout errors, refunds, or contacts about shipping charges.
Use historical traffic that meets the same eligibility definition to estimate the baseline. Suppose 8% of accounts pay and an increase to 9% would justify the maintenance cost. The difference is one percentage point, or 12.5% relative to baseline. Record which interval and decision rule the team will use. A design powered to distinguish 8% from 9% in a test against zero lift does not establish that lift exceeds one point. If the business decision requires evidence above that threshold, specify that boundary and plan for an alternative above it.
Convert the detectable effect into a calendar
For independent accounts and a binary outcome, a normal approximation can give a planning estimate. With equal allocation, a two-sided alpha of 0.05, 80% power, and assumed rates of 8% and 9%, the target is roughly 12,200 accounts per group. You can reproduce the estimate with Statsmodels by passing the absolute value from proportion_effectsize into NormalIndPower.solve_power and rounding up. Save the inputs, software version, and resulting per-group target. When assignment happens by household or store, account outcomes within each cluster may be correlated. Calculate power for the actual assignment structure before using that target.
Use qualified daily traffic to estimate enrollment. If traffic and buying behavior change on weekends, cover full weeks where feasible. After enrollment closes, allow the last assigned accounts their seven-day conversion window. When expected traffic cannot reach the target, document which effects the test can detect before the first outcome is read. Keep monitoring instrumentation and safety guardrails during the run. If the team wants to stop early based on the primary result, define the sequential method and stopping rule in the plan.
Validate the pipeline before interpreting lift
Run an A/A exercise with the same experience in both arms. Compare assignments, exposures, payment events, and join coverage. Use a test account that returns on a second device to confirm that assignment is stable. A quiet A/A run is one check on the pipeline. Validate the production experiment separately because different content or traffic can expose new logging and eligibility problems.
For a planned equal split, investigate a persistent 53:47 allocation using a sample ratio mismatch check. Compare unique accounts in assignment logs with those in the exposure and outcome tables. Break the counts down by platform, deployment version, and day. If the assignment table is balanced but exposure events are missing on older phones, investigate event delivery and eligibility before interpreting results. Dropping excess rows to force an even split obscures the failure.
Report accounts, payments, rates, absolute lift, and a confidence interval. Suppose lift is 0.3 percentage points and the interval runs from a decline of 0.2 to a gain of 0.8 points. State the uncertainty and compare the range with the planned business threshold. If an interval includes both a material gain and a material decline, say that the experiment leaves the decision unresolved. A p-value alone cannot communicate how much harm or benefit remains plausible.
Investigate effects that fade or reverse
Plot results by assignment cohort with equally mature observation windows. If early cohorts show a larger gain, check concurrent promotions, outages, and differences in traffic composition before attributing the change to novelty. Compare the desktop and mobile counts and rates if the overall estimate points in a different direction from both device estimates. Inspect the mix of traffic and any filtering applied after assignment. Include the original overall estimate in the report and document which breakdowns were planned and which were added during investigation.
Identify the metrics and groups that can trigger a decision, then specify how to control for multiple comparisons. If returning customers appear to benefit more, estimate the difference between the effects in the two groups and report its uncertainty. One subgroup passing a significance threshold and another missing it is insufficient evidence that their effects differ. Save exploratory findings as testable hypotheses with a proposed follow-up.
Build a pre-experiment RFM snapshot
Use an agreed historical window, such as 90 days. Recency is days since the latest qualifying purchase. Frequency counts distinct qualifying orders, and monetary value sums net paid amounts in a consistent currency. Document how you handle cancellations, refunds, late-arriving transactions, and accounts without purchase history. Give new shoppers a distinct label. Keep the snapshot date and cut points so an analyst can reproduce the assignment.
For example, one shopper bought 12 days ago, completed four orders, and paid a net 320 units in the window. Save those features alongside the segment and rule version, then join to the experiment by a stable pseudonymous key. Report each group's size, coverage, lift, and interval. If the group is too small to guide a rollout, use the finding to plan a targeted follow-up. Freezing segment membership before exposure keeps outcomes affected by the test from changing the definition of a subgroup.
What to hand over and how to troubleshoot
Leave a power worksheet with the assumptions and per-arm target, an analysis table with raw counts and interval method, an anomaly log, and a versioned segment definition. Use synthetic records to exercise the joins and check that one account receives one assignment and at most one baseline segment. Run the real analysis in the team's approved data environment.
Can the test end as soon as it reaches its target? Wait for the latest account's outcome window and check that late events have arrived. An updated dashboard date alone does not guarantee complete data.
What if a statistically detectable lift is too small to justify the change? Compare the interval with the business threshold and the observed guardrails. Record why the expected return does or does not cover rollout costs, and preserve the evidence for the next decision.
What if a segment changes its purchasing behavior during the run? Keep its original label for the experiment analysis. Use a later snapshot for subsequent operations, with its own effective date and decision record.
Use the linked Statistical Significance Analysis Setup Guide when preparing the analysis handover. Give it a written hypothesis, an explicitly chosen test, and a synthetic input table. Ask it to outline the setup and review steps. Verify the calculation independently with the recorded counts and assumptions.