project-006 · Analytics and statistics · e-commerce
From session data to an experiment backlog: which conversion drivers deserve a test?
An online shop's growth team wants to know which session behaviours and contexts go with purchases, so that it can choose two or three changes to A/B test next quarter. The deliverable is an evidence-ranked experiment backlog; it must not present associations as effects.
At a glance
Data
Online Shoppers Purchasing Intention Dataset (UCI 468) (public) · 12,330 sessions x 18 columns; online_shoppers_intention.csv 1 MB
Details on dataBaseline
Crude comparisons: conversion rate by each candidate driver (for example returning versus new visitors, any administrative page versus none) with simple intervals.
Details on baselineValidation
Split sessions 50/50 at random with a fixed seed. Exploration half: generate and refine hypotheses freely.
Details on validationMetric
Adjusted risk differences and odds ratios with 95% intervals, BH q-values, replication rate from exploration to confirmation.
Details on metric
Success criterion
- Ten or more hypotheses written down before looking at the confirmation half of the data.
- For each: adjusted risk difference (or odds ratio) with 95% interval estimated on the confirmation half, and Benjamini-Hochberg control at 10% FDR across the list.
- Each backlog item states the smallest effect worth shipping, the sample size for 80% power at the site's conversion rate, and the confounding or definitional risk that the test would remove.
- A documented decision on
PageValues, whose definition ties it to completed transactions.
Context
The UCI dataset has 12,330 sessions over one year, each from a different user. Page-category counts and durations come from URLs; BounceRates, ExitRates and PageValues are Google Analytics page-level metrics attached to the pages a session visited. PageValues is the average value of pages visited before a transaction, so it is partly a consequence of purchasing rather than a cause. Purchase intent drives both browsing depth and buying, which is why "more product views, more purchases" says little about what to change.
Data
- Dataset
- Online Shoppers Purchasing Intention Dataset (UCI 468) (public)
- Licence
- CC BY 4.0 (as stated on the UCI page)
- Size
- 12,330 sessions x 18 columns; online_shoppers_intention.csv 1 MB
- Verification
- identity verified
Access
Direct download (1 MB CSV) from the UCI page or ucimlrepo.fetch_ucirepo(id=468); no account. Fallback: none needed.
Grain
One row per session; the provider states each session belongs to a different user over a one-year period.
| Column | Meaning |
|---|---|
| Administrative / Informational / ProductRelated | Number of pages of each type visited in the session |
| *_Duration | Total time on each page type in the session |
| BounceRates / ExitRates | Google Analytics page-level rates averaged over the session's pages |
| PageValues | Average value of the visited pages before a transaction (definitionally linked to purchases) |
| SpecialDay | Closeness of the visit date to a special day such as Valentine's Day (0-1) |
| Month / Weekend | Calendar context of the session |
| OperatingSystems / Browser / Region / TrafficType | Integer-coded categories |
| VisitorType | Returning, new or other visitor |
| Revenue | True if the session ended in a purchase (outcome) |
EDA and data-quality rules
Count sessions per Month and check for absent or thin months before any seasonal statement.
Durations: zero duration with positive page counts and extreme durations are counted; decide on log transforms and caps before modelling.
Integer-coded categoricals (TrafficType, Region, Browser, OperatingSystems) are treated as categories, with rare levels grouped by a stated rule.
BounceRates and ExitRates are page-level averages; do not describe them as whether this session bounced.
Report the overall conversion rate and its per-month variation, since every power calculation depends on it.
Baseline
Crude comparisons: conversion rate by each candidate driver (for example returning versus new visitors, any administrative page versus none) with simple intervals. These are the numbers most dashboards show.
Features
Pre-registered hypothesis variables: page-depth bins per page type, log durations, any administrative or informational visit, SpecialDay > 0, Weekend, VisitorType, grouped TrafficType. Adjusters: Month and TrafficType group (context that affects both browsing and buying). PageValues is excluded from driver analysis and shown in a sensitivity run only.
Candidate methods
Mantel-Haenszel risk differences stratified by Month
Simple, transparent adjustment for seasonal mix; shows Simpson-type reversals directly.
Multivariable logistic regression with pre-specified adjusters and robust errors
Adjusts for several contexts at once; average marginal effects translate to risk differences for the backlog.
Shallow gradient boosting with held-out permutation importance (exploration half only)
Screens for interactions and non-linear patterns to propose hypotheses; never used for the confirmatory estimates.
Validation design
Split sessions 50/50 at random with a fixed seed. Exploration half: generate and refine hypotheses freely. Confirmation half: estimate only the frozen list, apply FDR control, and report how many exploration findings replicate. This keeps the reported intervals honest despite open-ended exploration.
Leakage checks
PageValues is computed from transactions, so it inflates any "driver" model. Page counts and durations accumulate through the session; a real-time intervention would only see the session so far, so backlog items that rely on full-session totals must say so.
Bias and representativeness
Reverse causation and intent confounding: shoppers who already intend to buy view more pages. Month and visitor mix create Simpson-type reversals. The data cover one site and one year.
Robustness checks
With and without PageValues; alternative binning; excluding months with few sessions; logistic versus stratified estimates.
Technical metrics
Adjusted risk differences and odds ratios with 95% intervals, BH q-values, replication rate from exploration to confirmation.
Business interpretation
For each backlog item: smallest effect worth shipping, required sample size and test duration at current traffic (hypothetical), and the incremental conversions per 10,000 sessions if the lower interval bound were causal (an optimistic ceiling, labelled as such).
Error slices
New versus returning visitors.
Months near special days versus other months.
Traffic-type groups (acquisition channels).
Sessions with zero product pages.
Implementation steps
Download, checksum, load; run the EDA rules and write the PageValues decision.
Split exploration/confirmation halves; explore with crude rates and the boosting screen; freeze at least ten hypotheses.
Estimate stratified and logistic adjusted effects on the confirmation half; apply BH.
Compute power and sample size for each candidate test with statsmodels.
Write the backlog table: hypothesis, evidence, risk of confounding, test design, sample size.
Reproduction
Learner-built. Seeded split saved to disk with the row ids in each half; one command produces the backlog table
and charts; dependencies pinned with uv.lock.
Limitations
Observational session data cannot identify which change would increase purchases.
Aggregated analytics metrics hide the page sequence within a session.
One site, one year; the Month variable mixes seasonality with that year's campaigns.
Production and monitoring
The backlog feeds the experimentation roadmap. After each test, record the observed effect next to the observational estimate; over time this calibrates how much the team should trust observational screens.
Interview questions
Sessions that visit an administrative page convert twice as often. Should we push users to those pages?
Why would including PageValues make your 'driver' model look excellent and be useless for decisions?
You tested 40 associations and 6 are significant at 5%. How many would you expect by chance, and what do you do?
Postmortem and alternatives
A real-time purchase-propensity model, as in the dataset paper, answers a different question (whom to target mid-session) and needs features restricted to what is known at each moment. For drivers, the only definitive evidence is the A/B tests the backlog proposes.
Rubric3 criteria
| Criterion | Meets | Exceeds |
|---|---|---|
| Honest inference | Exploration/confirmation split with a frozen hypothesis list and FDR control. | Reports the replication rate and discusses what it implies for future screens. |
| Confounding and definitions | Excludes PageValues as a driver with a clear reason and adjusts for Month. | Shows a Simpson-type reversal or explains why none appears, and names intent confounding per item. |
| Backlog quality | Each item has effect size, sample size and the risk the test removes. | Prioritises items by expected value per week of traffic and flags tests that are infeasible. |
References5
- Online Shoppers Purchasing Intention Dataset
Dataset identity and feature definitions incl. PageValues.
- Real-time prediction of online shoppers' purchasing intention using multilayer perceptron and LSTM recurrent neural networks
Dataset paper; real-time purchase prediction framing.
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
FDR control for a list of hypotheses.
- To Explain or to Predict?
Why a good predictor is not an explanation of drivers.
- Statistics stats (statsmodels): power and sample size, multiple tests, proportion confidence intervals
Power and sample size for the proposed tests.