Skip to content
ML Academy

project-033 · Deep learning · aviation

Routing airline complaints: TF-IDF + logistic regression vs a small neural text classifier

An airline's social-care team wants negative customer tweets routed automatically to the right queue (for example delays, cancellations, baggage, customer service). Does a small neural text classifier beat a TF-IDF plus logistic-regression baseline by enough to matter, and what share of complaints can be routed automatically at a target precision with the rest sent to a person?

Licence
CC BY-NC-SA 4.0
Reference run
none

At a glance

  • Data

    Twitter US Airline Sentiment (public) · 14,640 tweets × 15 columns; Tweets.csv 3,421,431 bytes

    Details on data

  • Baseline

    Majority reason (macro-F1 floor) and TF-IDF over word 1-2-grams plus character 2-5-grams with multinomial logistic regression (class weights balanced, regularisation…

    Details on baseline

  • Validation

    After deduplication, sort negative tweets by tweetcreated; the first 70% train, the next 15% validation, the last 15% test (read once). Neural models run with 3 seeds.

    Details on validation

  • Metric

    Macro-F1 (primary; small queues matter), per-class precision and recall, log loss and a reliability curve; bootstrap intervals for differences.

    Details on metric

Success criterion

On a time-ordered test slice (the latest 15% of tweets), read once:

  • macro-F1 over complaint reasons for every model, with bootstrap 95% intervals;
  • a neural model is recommended only if its macro-F1 gain interval excludes zero and recall on the smallest queues is not worse;
  • coverage at 90% precision for auto-routing, with abstention to a human queue below the validation-chosen threshold.

Context

Days 32, 35, 38 and the Day 56 neural project. A linear bag-of-words model is a strong text baseline: fastText, a linear model with n-gram features, matched the accuracy of deep classifiers of its time at a fraction of the cost (Joulin et al., 2017). The data are real public tweets from February 2015, labelled by crowd workers with a confidence score, so label noise is part of the problem. The licence is non-commercial (CC BY-NC-SA 4.0): fine for learning and a portfolio, not for a commercial product. CPU only.

Data

Dataset
Twitter US Airline Sentiment (public)
Licence
CC BY-NC-SA 4.0
Size
14,640 tweets × 15 columns; Tweets.csv 3,421,431 bytes
Verification
identity verified

Access

Kaggle dataset page (free Kaggle account required; kaggle datasets download with an API token works). Files: Tweets.csv (3.4 MB) and database.sqlite. Byte-identical community mirrors exist, but take the licence and provenance from the Kaggle page. Fallback: none that keeps the airline domain; keep the project specified if you cannot get access.

Grain

One row = one tweet mentioning one of six US airlines, with crowd labels for sentiment and, for negative tweets, a reason.

ColumnMeaning
tweet_id, text, tweet_createdIdentifier, raw tweet text, creation timestamp (check the time zone and range in EDA).
airline_sentiment, airline_sentiment_confidenceCrowd label positive/neutral/negative and the crowd's confidence (not a feature).
negativereason, negativereason_confidenceComplaint reason for negative tweets (the routing target) and its crowd confidence; includes an ambiguous 'can't tell'-style category to inspect.
airlineAirline the tweet is about; used for the airline-holdout robustness test.
name, tweet_location, user_timezone, tweet_coordUser handle and location fields: personal data, dropped before modelling and never displayed in reports.
airline_sentiment_gold, negativereason_gold, retweet_countSparse gold-label fields and retweet counts; not features.

EDA and data-quality rules

  • Distribution of negative reasons among negative tweets; decide and document how the ambiguous reason category is treated (excluded from training, or kept as an 'abstain' class).

  • Distribution of label confidence; define a low-confidence slice (for example below 0.7) before modelling.

  • Duplicate and near-duplicate texts (retweets, copied complaints) after masking handles and URLs; deduplicate before splitting.

  • Range of tweet_created and its time zone; the time split depends on it.

  • Drop personal fields; mask @handles and URLs with placeholder tokens, and decide whether airline handles stay (they predict the airline, not the reason).

Baseline

Majority reason (macro-F1 floor) and TF-IDF over word 1-2-grams plus character 2-5-grams with multinomial logistic regression (class weights balanced, regularisation strength tuned on validation). This is the model a careful team would ship first.

Features

Lowercased text with handles and URLs masked, HTML entities decoded, negations kept, no stemming. TF-IDF vectoriser fitted on the training split only. Neural models use the same cleaned text; the from-scratch model builds its vocabulary from training tweets only.

Candidate methods

  • fastText-style classifier trained from scratch (averaged word and bigram embeddings → linear softmax), or a one-hidden-layer MLP on TF-IDF

    The smallest neural model that learns its own representation; trains in seconds on CPU. Shows whether learned embeddings beat sparse features at this data size.

  • Frozen sentence embeddings (all-MiniLM-L6-v2) + logistic regression

    Brings knowledge from large-scale pretraining without fine-tuning; needs a 90.9 MB model download (mark as not run if blocked).

  • Fine-tuning a small Transformer encoder

    Optional and out of the CPU default; mention as the next step only if the frozen-embedding result suggests headroom.

Validation design

After deduplication, sort negative tweets by tweet_created; the first 70% train, the next 15% validation, the last 15% test (read once). Neural models run with 3 seeds. Regularisation, epochs and the abstention threshold come from validation. Second protocol for robustness: train on five airlines and test on the sixth, rotating, to see whether the model learned complaint language or airline-specific base rates.

Leakage checks

Duplicates and retweets across splits; vectoriser and vocabulary fitted on training data only; confidence and gold columns excluded from features; airline handles masked in the airline-holdout protocol.

Bias and representativeness

There are no demographic labels, so group fairness cannot be measured; say so rather than implying it was checked. Inspect errors on informal spelling, slang and very short tweets, which can concentrate in some user groups. Privacy: tweets are personal data; drop identity and location fields, never quote individual tweets with handles in the report, and respect the non-commercial licence.

Robustness checks

Airline holdout; typo injection; label noise (train on high-confidence labels only and compare); seed spread for neural models.

Technical metrics

Macro-F1 (primary; small queues matter), per-class precision and recall, log loss and a reliability curve; bootstrap intervals for differences.

Business interpretation

Auto-routing coverage at 90% precision; misrouted complaints per 1,000; a hypothetical cost per misroute (extra handling time) and per human-routed tweet, labelled as assumptions.

Error slices

  • Per complaint reason, especially the smallest queues.

  • Label confidence below 0.7 against at or above 0.7.

  • By airline, and in the airline-holdout protocol.

  • Very short tweets (fewer than 8 tokens) and tweets that mention several problems.

Implementation steps

  • Studio 0-20 min: load, drop personal fields, mask, deduplicate, EDA rules, time split.

  • Studio 20-40 min: majority and TF-IDF + logistic baselines; per-class report.

  • Studio 40-70 min: fastText-style model or TF-IDF MLP: check input and output shapes, the loss at init near ln(number of classes), overfit one batch; train with 3 seeds.

  • Studio 70-90 min: compare on validation; choose the abstention threshold; decision-log entry.

  • Homework: frozen-embedding candidate; airline-holdout and label-noise experiments; test read once; slices; one-page report with the routing recommendation.

Reproduction

Kaggle download script that checks the file size (3,421,431 bytes) and records a SHA-256; pinned scikit-learn (and PyTorch CPU or sentence-transformers in your own environment if used); saved split ids and seeds; one command that regenerates all metrics and the per-class table. No academy reference implementation.

Limitations

  • Crowd labels with varying confidence; part of the remaining error is label noise, not model error.

  • Tweets about six US airlines scraped in February 2015 (check the exact date range in EDA); complaint topics and platform language change, so retraining would be needed before any real use.

  • Non-commercial licence: results cannot be used in a commercial product.

  • Routing by single label ignores tweets that raise several problems.

Production and monitoring

Abstain below the threshold and send to a human queue; monitor queue mix, abstention rate and agent re-routing (the live label signal) weekly; watch for new topics (for example a disruption event) as a sudden rise in abstentions. Keep personal data out of logs; retrain on re-routed examples with a fixed evaluation set so improvements are comparable.

Interview questions

  • Your TF-IDF model and a small neural model differ by one macro-F1 point. How do you decide, and what else would you look at?

  • Why might a model trained on these tweets perform worse on a new airline, and how did you test that?

  • How would you handle low-confidence crowd labels in training and in evaluation?

Postmortem and alternatives

Record whether the neural model's gain (if any) came from rare queues or from frequent ones, and whether it survived the airline holdout. If the baseline wins, the finding is that sparse n-grams already capture complaint vocabulary at this data size. Alternatives: more labelled data for the smallest queues, a hierarchical scheme (complaint or not, then reason), or fine-tuning a small Transformer when a GPU is available.

Rubric4 criteria

CriterionMeetsExceeds
Data handling and privacyPersonal fields dropped, handles and URLs masked, duplicates removed before the time split.Shows the metric inflation caused by cross-split duplicates and documents the licence constraints.
Fair comparisonSame split and macro-F1 for all models; neural seeds reported; test read once.Bootstrap intervals plus the airline-holdout protocol for the two best models.
Error analysisPer-class results and the label-confidence slice.Label-noise experiment and a qualitative review of multi-problem tweets, without quoting identifiable users.
DecisionRouting recommendation with coverage at 90% precision and stated cost assumptions.Monitoring plan using agent re-routing as the live label and a retraining protocol with a fixed evaluation set.

References6