Sentiment analysis that survived contact with real reviews

The problem

Clients wanted to know whether sentiment about them was improving. The raw material was review text from several platforms, and the first version did the obvious thing: run each review through a sentiment classifier, average the scores, plot the line.

The line moved. Nobody could say whether it meant anything.

That is the failure worth dwelling on. A number that is precise, updates daily and has no established relationship to reality is more dangerous than no number, because people make decisions with it.

Getting the inputs comparable

Before any modelling, the text had to be made comparable across sources, and it was not. Platforms differ in review length, in whether a star rating accompanies the text, in how they handle edits and deletions, and — most importantly — in who writes reviews at all. Sentiment on one platform is not on the same scale as sentiment on another, and averaging them produces a number describing the mix of platforms as much as the mix of opinions.

So sentiment was computed and reported per platform, then combined only with an explicit, documented weighting. A single blended figure with no stated weighting is a number nobody can interpret.

Calibration before trust

The step that turned this from a plausible feature into a usable one was boring: we hand-labelled a sample and checked the classifier against it.

Two changes followed directly from reading those errors:

Aspect-level rather than whole-review sentiment. A review praising the product and criticising delivery is not "neutral", and collapsing it to one score destroys the only actionable thing in it. Splitting by aspect meant the insight became delivery is the problem, which a client can act on.

Confidence became a first-class output. Low-confidence classifications are excluded from headline figures and surfaced separately for review, rather than averaged in where they quietly move a trend line nobody can explain.

Pipeline shape

Ingest, normalise, classify, aggregate — each stage writing its output rather than passing it along in memory. That costs storage and buys the ability to reclassify history when the model changes, which happened more than once. A pipeline that only holds final scores cannot answer "what would last quarter look like under the current model", and that is the first question anyone asks after an upgrade.

Outcome

The numbers became defensible. Per-platform figures with a stated weighting, aspect-level breakdowns, and low-confidence output kept out of the headline rather than buried in it.

The thing I'd tell anyone building this: the modelling was the small part. The work was making the inputs comparable and establishing what the output actually meant — and until that existed, a moving line was just a moving line.