My recommender's design was wrong, and the measurements said so

When I started H-Tech, my movie recommender, I wrote the design down before writing any model code. It was the textbook answer. Collaborative filtering (CF) can’t help a brand-new user, because it has no ratings to compare. Content-based filtering (CBF) can, because every film has genres and tags. So: content-based for new users, collaborative filtering once they’ve rated enough, and a switch somewhere around five ratings.

Almost none of that design survived contact with the data. I only found out because I measured each choice instead of trusting it. This post is about the four places where a number changed my mind.

How I measured

The project is trained on MovieLens 25M (about 25 million ratings). To test how each method treats a new user, I took real users out of the training data and pretended they were new: reveal their first n ratings in the order they made them, ask each method for a top 10, and check those 10 films against the films that user actually rated 4 stars or more later on.

Two scores come out of that. Precision@10 is how many of the 10 the user went on to like. nDCG@10 is the same idea but rewards putting the hits near the top. I ran it at several profile sizes (1, 3, 5, 10, 20 and 50 revealed ratings), because “new user” and “established user” behave very differently.

1. A good rating predictor can be a bad recommender

My collaborative filter was SVD, the classic matrix factorization model. It predicts what rating you’d give a film, and it’s scored by RMSE, the average size of its error. It won that comparison at every profile size:

Lower RMSE is better, so CF predicts ratings better throughout. But once a user has a real profile, content-based filtering picks better films.

The reason was in SVD’s own parts. Most of its advantage over a plain average came from its bias terms: one number for how generous a user is, one for how well liked a film is. Those terms supplied 18.9 of the 23.5 percentage points by which SVD beat the global mean. A film’s bias is the same for everyone, so a model that leans on it gives everyone roughly the same list. That’s great for predicting a rating and poor for choosing what to show someone.

So I split the two jobs. Predicting a rating and choosing ten films are different questions, and they don’t need the same model. CF still answers “what would you rate this?”, and it was taken out of the ranking path.

2. A filter beat model tuning

Left to rank the whole catalogue, SVD liked obscure films. One sample user’s top 10 included titles with only 17, 20 and 21 ratings. With that little evidence, a film’s learned factors are mostly noise, and nothing stops its predicted rating coming out extreme.

The fix wasn’t a better model. It was refusing to rank films without enough support:

That’s a 4.2× improvement with the model untouched. It’s the standard two-stage shape: a cheap step picks sensible candidates, then the expensive step ranks them. I set the cutoff from a quantile rather than a fixed number, so it means the same thing at any catalogue size.

3. “Most popular” beat both personal methods for new users

This is the one I least expected. I added plain popularity, a Bayesian weighted rating with no personalization at all, as a baseline:

Popularity won until 15 ratings, not 5. A thin profile carries little signal, so a “personal” model mostly fits noise, while “films almost everyone likes” stays a strong guess. And as a user rates more, the popular films move into what they’ve already seen, so popularity runs out just as their profile becomes worth using. At 15 the two curves were within 0.0002 of each other, so I wrote down that the threshold was marginal there, not a clean line.

That routing (popularity, then content-based at 15 ratings) shipped. Then I tested a method I hadn’t tried: item-based CF. It scores a film by how often people who liked your films also liked it (4 stars or more), built from all 24.6 million ratings in about 35 seconds. It roughly doubled both earlier tiers, starting from a single liked film:

That retired the 15-rating switch for ranking. Today the score is mostly item-kNN with a small share from MovieLens’s tag genome, blended with popularity for very new users: 30% personal at one liked film, rising to fully personal at 12. The genome share is there for its labels rather than accuracy. Each of its axes has a human name (“dark”, “funny”, “twist ending”), which is what lets people steer their feed and see why a film was picked.

The cold-start lesson I took away isn’t “use popularity”. It’s that the right answer depends on the data, so the switch point has to be measured, not assumed.

4. The evaluation caught a bug I’d already shipped

Users wanted a refresh to show different films, so the home feed adds randomness to each visit. The first time I scored the feed with that randomness switched on, its precision had nearly halved: from 0.338 to 0.187 at 50 ratings. At one rating it was 0.201, barely above plain popularity’s 0.189.

The cause was a units mistake. I added noise of 0.06 to normalized scores, but the best 120 candidates all sat within about 0.001 of each other. The noise swamped the ranking, so the pick was effectively random.

The fix was to sample by rank instead of by score: each candidate is drawn with a probability that depends only on its position, so it doesn’t matter how bunched the scores are. Then the trade-off becomes a dial I can measure:

About 4 new films in every 10 for a 3–6% precision cost. The real lesson: I’d been evaluating a cleaner version of the feed than the one users got. Now the evaluation scores exactly what’s served.

As a last check, I scored the shipped configuration on a fresh sample of users that none of the tuning had seen. The served feed, randomness included, reached 0.319 precision@10 at one liked film. The deterministic feed it replaced reached 0.294.

The memory problem, honestly

One more decision, and this one only partly worked. Search uses sentence embeddings (all-MiniLM-L6-v2). Running it through PyTorch kept the API off free-tier hosts, and the main cost wasn’t the model’s weights but PyTorch’s own runtime overhead. So the Docker build converts the model to ONNX once, in a stage that’s thrown away, and the server runs it with ONNX Runtime and the tokenizers library. PyTorch never ships.

That made the image much smaller, but it wasn’t enough to fit a 512 MB instance. On Google Cloud Run the API was killed for using too much memory at 1 GiB and again at 2 GiB while starting up, and it runs at 4 GiB. Removing PyTorch was the right call, but it wasn’t the whole answer.

What I’d keep doing

• Write the decision down before the code, so there’s something specific for a measurement to disagree with. Each of these changes is a short decision record in the repo, and each says which earlier one it revises.
• Add the dumb baseline. Popularity was the most useful line in every table.
• Measure what ships, randomness and fallbacks included, on users the tuning never saw.
• Keep the caveat. These numbers come from one dataset, simulated new users and one family of metrics. A different catalogue could move every threshold here. The method should carry over. The constants won’t.

You can try H-Tech live, or read the case study for the architecture and stack.