Recommendation systems power Netflix, Spotify, Amazon, and YouTube — they’re one of the highest-value applications of machine learning in industry. This guide covers the main approaches to building recommendation systems in Python, from simple collaborative filtering to matrix factorization and neural approaches.
Types of Recommendation Systems
There are three main paradigms. Collaborative filtering recommends items based on the preferences of similar users (“users who liked X also liked Y”). Content-based filtering recommends items similar to what a user has liked before (“you liked Action movies, here are more Action movies”). Hybrid systems combine both approaches — most production systems are hybrids because each approach has different failure modes.
Collaborative Filtering – User-Based
import pandas as pd
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
# Create user-item rating matrix
ratings_matrix = df.pivot_table(index='user_id', columns='item_id', values='rating').fillna(0)
# Compute user-user similarity
user_similarity = cosine_similarity(ratings_matrix)
user_sim_df = pd.DataFrame(user_similarity,
index=ratings_matrix.index,
columns=ratings_matrix.index)
def recommend_for_user(user_id, n_recommendations=10):
similar_users = user_sim_df[user_id].sort_values(ascending=False)[1:11]
items_rated_by_user = set(ratings_matrix.loc[user_id][ratings_matrix.loc[user_id] > 0].index)
scores = {}
for sim_user, sim_score in similar_users.items():
sim_user_ratings = ratings_matrix.loc[sim_user]
for item, rating in sim_user_ratings[sim_user_ratings > 0].items():
if item not in items_rated_by_user:
scores[item] = scores.get(item, 0) + sim_score * rating
return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:n_recommendations]
Item-Based Collaborative Filtering
# Item-item similarity (often more stable than user-user)
item_similarity = cosine_similarity(ratings_matrix.T)
item_sim_df = pd.DataFrame(item_similarity,
index=ratings_matrix.columns,
columns=ratings_matrix.columns)
def similar_items(item_id, n=10):
return item_sim_df[item_id].sort_values(ascending=False)[1:n+1]
Matrix Factorization with Surprise
pip install scikit-surprise
from surprise import SVD, Dataset, Reader, accuracy
from surprise.model_selection import cross_validate, train_test_split
reader = Reader(rating_scale=(1, 5))
data = Dataset.load_from_df(df[['user_id', 'item_id', 'rating']], reader)
# SVD (Singular Value Decomposition) — Netflix Prize winner
trainset, testset = train_test_split(data, test_size=0.2, random_state=42)
svd = SVD(n_factors=100, n_epochs=20, lr_all=0.005, reg_all=0.02)
svd.fit(trainset)
predictions = svd.test(testset)
print(f"RMSE: {accuracy.rmse(predictions):.4f}")
print(f"MAE: {accuracy.mae(predictions):.4f}")
# Predict for a specific user-item pair
pred = svd.predict(uid='user_42', iid='item_99')
print(f"Predicted rating: {pred.est:.2f}")
Content-Based Filtering
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import linear_kernel
# Movies with description, genre, cast as features
movies['features'] = (movies['genre'] + ' ' + movies['description'] + ' '
+ movies['director'] + ' ' + movies['cast'])
tfidf = TfidfVectorizer(stop_words='english', max_features=10000)
tfidf_matrix = tfidf.fit_transform(movies['features'])
# Compute cosine similarity between all movies
cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)
def content_recommendations(title, n=10):
idx = movies[movies['title'] == title].index[0]
sims = list(enumerate(cosine_sim[idx]))
sims = sorted(sims, key=lambda x: x[1], reverse=True)[1:n+1]
return movies.iloc[[i[0] for i in sims]][['title', 'genre']]
Neural Collaborative Filtering
import torch
import torch.nn as nn
class NeuralCF(nn.Module):
def __init__(self, n_users, n_items, embedding_dim=64, hidden_dims=[128, 64, 32]):
super().__init__()
self.user_embedding = nn.Embedding(n_users, embedding_dim)
self.item_embedding = nn.Embedding(n_items, embedding_dim)
layers = []
in_dim = embedding_dim * 2
for out_dim in hidden_dims:
layers += [nn.Linear(in_dim, out_dim), nn.ReLU(), nn.Dropout(0.2)]
in_dim = out_dim
layers.append(nn.Linear(in_dim, 1))
self.mlp = nn.Sequential(*layers)
def forward(self, user_ids, item_ids):
u = self.user_embedding(user_ids)
v = self.item_embedding(item_ids)
x = torch.cat([u, v], dim=-1)
return self.mlp(x).squeeze()
model = NeuralCF(n_users=10000, n_items=5000)
Evaluation Metrics
from sklearn.metrics import ndcg_score
import numpy as np
# RMSE (rating prediction tasks)
from sklearn.metrics import mean_squared_error
rmse = np.sqrt(mean_squared_error(y_true, y_pred))
# Precision@K and Recall@K (ranking tasks)
def precision_at_k(recommended, relevant, k=10):
recommended_k = recommended[:k]
return len(set(recommended_k) & set(relevant)) / k
def recall_at_k(recommended, relevant, k=10):
recommended_k = recommended[:k]
return len(set(recommended_k) & set(relevant)) / len(relevant)
# NDCG@K (normalized discounted cumulative gain)
ndcg = ndcg_score([relevance_scores], [predicted_scores], k=10)
Production Considerations
Pre-compute recommendations offline and store in a cache (Redis, DynamoDB) for low-latency serving — running matrix factorization at request time is too slow. Retrain weekly or daily on fresh interaction data. Handle the cold-start problem (new users/items with no history) with content-based recommendations until enough interaction data accumulates. Log all impressions and clicks — this interaction data is what makes recommendation systems improve over time.
Conclusion
Start with item-based collaborative filtering (simple, no GPU needed, strong baseline) or SVD matrix factorization from the Surprise library for rating prediction tasks. For production systems with millions of users and items, neural approaches like Two-Tower models or LightFM handle scale better. The cold-start problem and offline-to-online serving gap are the two hardest engineering problems in production recommendation systems — plan for both before you start building.
