project-002 · Analytics and statistics · logistics

Late deliveries and review scores in a marketplace: how much damage, and on which lanes?

A marketplace's logistics team must choose between two investments: faster shipping on specific seller-to-customer lanes, or more conservative delivery-date promises. Both depend on one estimate: how much does delivering after the promised date lower the customer's review score, and where are late, badly reviewed orders concentrated?

Dataset: Brazilian E-Commerce Public Dataset by Olist (Kaggle olistbr/brazilian-ecommerce)
Licence: CC BY-NC-SA 4.0 (as stated on Kaggle)
Reference run: none
Lessons: day 2 day 5 day 6 day 7 day 8

Data
Brazilian E-Commerce Public Dataset by Olist (Kaggle olistbr/brazilian-ecommerce) (public) · Nine CSV files, about 126 MB in total; about 100k orders
Details on data
Baseline
Unadjusted comparison: mean review score and share of 1-2 star reviews for late versus on-time delivered orders, with a simple (unclustered) interval.
Details on baseline
Validation
Causal estimates cannot be validated against held-out labels, so validation is design-based: Write the analysis plan on 2017 orders, then run it unchanged on 2018…
Details on validation
Metric
Adjusted differences with 95% clustered-bootstrap intervals, standardized mean differences after weighting, overlap summary, agreement across specifications and years,…
Details on metric

Success criterion

An adjusted estimate of the difference in mean review score and in the share of 1-2 star reviews between late and on-time delivered orders, with 95% intervals from a bootstrap clustered by seller.
A dose-response curve by days late (on time, 1-3, 4-7, 8+ days) with intervals.
The sign and size of the main estimate agree (overlapping intervals) across three specifications and between the 2017 and 2018 halves of the data; disagreements are explained, not hidden.
A lane table ranking where lateness-associated low reviews concentrate, with the assumption that makes it causal stated next to it.

Context

The Olist dataset contains about 100k orders placed in 2016-2018 on Brazilian marketplaces, anonymised by the provider. Sellers ship through logistics partners; the customer receives a satisfaction survey after delivery or once the estimated delivery date passes, so some reviews are written before the parcel arrives. An order can contain several items from different sellers. Lateness and low scores share causes (seller reliability, product category, distance), so a raw comparison mixes delivery effects with seller quality. The promise date itself is set by the platform, which means "late" depends on how much padding a lane receives.

Data

Dataset: Brazilian E-Commerce Public Dataset by Olist (Kaggle olistbr/brazilian-ecommerce) (public)
Licence: CC BY-NC-SA 4.0 (as stated on Kaggle)
Size: Nine CSV files, about 126 MB in total; about 100k orders
Verification: identity verified

Access

Kaggle: requires a signed-in Kaggle account for browser download, or a Kaggle API token for kaggle datasets download -d olistbr/brazilian-ecommerce. Keep raw files out of version control; the license is non-commercial share-alike, so do not redistribute derived tables in a public portfolio beyond aggregates. The geolocation file (about 61 MB) is optional for this project. Fallback if Kaggle access is blocked: build the pipeline against a small seeded synthetic table with the same columns, label all outputs synthetic, and claim nothing about Olist.

Grain

orders: one row per order_id (customer_id is per order; customer_unique_id identifies a person). order_items: one row per item within an order (order_item_id is the position). order_reviews: one row per review. Analysis grain: one delivered order with one review (rule below for duplicates); item-level attributes are aggregated to the order before joining reviews.

Column	Meaning
order_id	Order key shared by orders, order_items, order_reviews, order_payments
order_status	Lifecycle status; analysis keeps delivered orders
order_purchase_timestamp	Purchase time; defines the cohort month
order_delivered_customer_date	Actual delivery to the customer
order_estimated_delivery_date	Promised date shown to the customer; defines lateness
seller_id (order_items)	Seller fulfilling an item; clustering unit for the bootstrap
price / freight_value (order_items)	Item price and freight; summed per order
product_category_name (products)	Portuguese category; English via the translation file
customer_state / seller_state	States that define the shipping lane
review_score / review_creation_date	1-5 stars and when the survey answer was created

EDA and data-quality rules

Grain assertions: order_id unique in orders; (order_id, order_item_id) unique in order_items; aggregate items to the order before joining reviews, then assert row counts did not grow.
Reviews: count orders with more than one review and review_ids attached to more than one order; keep the earliest review per order and report how many were dropped.
Timestamps: delivered dates missing for delivered status, delivery before purchase, or estimated date before purchase are counted and excluded with reasons.
Review timing: share of reviews created before the delivery date; analyse them separately because the customer was reacting to a missing parcel.
Coverage: plot orders per month and trim thin months at the start and end of the period before any trend statement.
Multi-seller orders: count them; lateness is attributed at order level and these orders are excluded in one robustness run.

Baseline

Unadjusted comparison: mean review score and share of 1-2 star reviews for late versus on-time delivered orders, with a simple (unclustered) interval. It is the number most dashboards show and the one the adjusted analysis must explain.

Features

Treatment: late = delivery date after the estimated date (date level); days_late buckets for dose-response.
Pre-treatment adjusters: product category group, lane (seller state x customer state, grouped to same-state / neighbouring / distant), price and freight bands, number of items and of sellers, purchase month, and the seller's historical late rate computed only from orders delivered before this order's purchase date.
Not adjusted for: actual lead time (it contains the treatment), review text presence (affected by the experience) or anything recorded after delivery.

Candidate methods

Exact stratification on lane group x category group x quarter, averaging within-stratum differences
Transparent, shows which strata lack overlap, and needs no functional-form assumption.
Linear probability / OLS regression of score and of 1-2 star indicator on late plus adjusters, seller-clustered errors
Uses all strata, gives one adjusted estimate per outcome, and makes the adjustment set explicit.
Inverse-probability weighting with a propensity model for lateness
Separates the lateness model from the outcome model and exposes overlap problems through extreme weights and balance diagnostics.

Validation design

Causal estimates cannot be validated against held-out labels, so validation is design-based:

Write the analysis plan on 2017 orders, then run it unchanged on 2018 orders.
Negative-control outcome: lateness cannot change the price paid at purchase; an adjusted "effect" of late on price signals residual confounding by seller or lane.
Balance diagnostics after weighting (standardized mean differences below 0.1) and overlap plots.
Local comparison near the promise date: orders delivered 1-2 days early versus 1-2 days late differ little in logistics but sharply in the late label; a discontinuity there supports a promise-violation effect.

Leakage checks

Post-treatment adjustment is the main trap: actual lead time, review timing and comment presence all occur after or because of the delivery outcome. The seller late-rate feature must be computed only from earlier orders.

Bias and representativeness

Selection into reviewing (only respondents are observed; late orders may be reviewed more often), unmeasured seller quality, and promise padding that differs by lane. Claims are restricted to reviewed, delivered orders.

Robustness checks

Late defined as at least 1 day versus at least 3 days; excluding multi-seller orders; excluding reviews written before delivery; seller-clustered versus order-level bootstrap.

Technical metrics

Adjusted differences with 95% clustered-bootstrap intervals, standardized mean differences after weighting, overlap summary, agreement across specifications and years, negative-control estimate.

Business interpretation

Estimated monthly count of 1-2 star reviews associated with lateness per lane group (stated as causal only under no unmeasured confounding), and the share of late orders by lane to target logistics fixes.

Error slices

Lane group: distant interstate lanes versus same-state lanes.
Product category groups, especially bulky items with high freight.
Days-late bucket (dose-response) and early deliveries (does arriving early raise scores?).
Seller size: sellers with few orders, where clustered intervals are widest.

Implementation steps

Download with the Kaggle CLI, record file sizes and SHA-256 checksums, and load with explicit dtypes and parsed timestamps.
Build the order-level analysis table in SQL or pandas with grain assertions after every join; document dropped rows in a reconciliation table.
Define treatment, outcomes and adjusters; run the EDA rules and write down exclusions before estimating.
Compute the baseline, then the three candidate estimates on 2017 data; freeze the plan.
Run the frozen plan on 2018, the negative control, the near-promise comparison and robustness variants.
Write a memo with the dose-response chart, lane table and the explicit causal assumption.

Reproduction

Learner-built (no reference implementation). Required: one command that rebuilds every table and figure from the raw CSVs, a pinned environment (uv.lock), bootstrap seed and replicate count in a config file, and a manifest of raw-file checksums. Raw data stay local because of the Kaggle login and the non-commercial license.

Limitations

Observational data: seller quality and customer expectations are not fully measured, so adjusted differences can still be confounded.
Only customers who reviewed are observed, and the provider anonymised the data; representativeness of the full order base is unknown.
The promise-date algorithm is unknown, so 'late' mixes delivery speed with promise padding.
One marketplace in one country in 2016-2018; transport to other contexts needs new evidence.

Production and monitoring

A weekly delivery-performance report: late rate and 1-2 star share by lane with control limits, built from a tested SQL model with grain checks. A randomized test of promise padding on selected lanes is the route to a causal answer; the report tracks it.

Interview questions

Late orders average 1.5 stars lower. Can you tell the COO that faster delivery will lift ratings by 1.5 stars?
Why does joining reviews directly to order_items produce a biased average score?
Design an experiment to decide between faster shipping and padded promise dates on one lane.

Postmortem and alternatives

The design that would settle the question is a randomized promise-padding test, because the platform controls the promise. Without it, the near-promise comparison is the strongest observational evidence; the regression adjustment mostly documents which confounders were considered.

Rubric4 criteria

Criterion	Meets	Exceeds
Grain and data quality	Order-level table with assertions after joins and a reconciliation of dropped rows.	Separates pre-delivery reviews and multi-seller orders with justified rules and shows their impact.
Causal reasoning	States the adjustment set with a DAG and excludes post-treatment variables.	Adds the negative control and near-promise comparison and interprets disagreements.
Uncertainty	Seller-clustered bootstrap intervals for the main estimates.	Shows how much clustering widens intervals and why order-level intervals mislead.
Decision memo	Lane table and dose-response chart with an insight-led title and the key assumption.	Recommends a specific experiment with a power estimate and the decision it would change.

References5

Brazilian E-Commerce Public Dataset by Olist
Dataset identity, license, files and columns.
Kimball Dimensional Modeling Techniques: Grain
Declaring the grain before joining fact tables.
Causal Inference: What If
Confounding, adjustment and IP weighting for observational comparisons.
Endogenous Selection Bias: The Problem of Conditioning on a Collider Variable
Why conditioning on post-treatment variables such as review timing biases estimates.
Causal Inference: The Mixtape
DAG-based choice of adjustment set (ch. 3); the near-promise comparison borrows the logic of the book's regression-discontinuity chapter but is a design-based check here, not a full RD analysis.

At a glance