Recommendation Engine
Overview
This skill provides comprehensive implementation of recommendation systems using collaborative filtering, content-based filtering, matrix factorization, and hybrid approaches to predict user preferences and deliver personalized suggestions.
When to Use
- Building personalized product recommendations for e-commerce platforms
- Creating content recommendation systems for streaming services, news platforms, or social media
- Implementing user-user or item-item collaborative filtering based on interaction patterns
- Addressing cold start problems for new users or items with limited interaction history
- Evaluating recommendation quality using precision@k, recall@k, and NDCG metrics
- Scaling recommendation systems to handle millions of users and items efficiently
Recommendation Approaches
- Collaborative Filtering: Using user-item interaction patterns
- Content-Based: Recommending similar items based on features
- Hybrid: Combining multiple approaches
- Matrix Factorization: Decomposing user-item matrix
- Neural Networks: Deep learning for embeddings
- Knowledge-Based: Using domain knowledge and rules
Key Techniques
- User-User Similarity: Finding similar users
- Item-Item Similarity: Finding similar items
- Latent Factors: Hidden patterns in data
- Embeddings: Vector representations of users/items
- Graph-Based: Social networks and item graphs
Python Implementation
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics.pairwise import cosine_similarity, euclidean_distances
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
from scipy.sparse import csr_matrix
import warnings
warnings.filterwarnings('ignore')
print("=== 1. Collaborative Filtering ===")
# Create sample user-item interaction matrix
np.random.seed(42)
n_users = 50
n_items = 30
# Create sparse interaction matrix (ratings: 0-5)
interaction_matrix = np.random.randint(0, 6, size=(n_users, n_items))
# Make it sparse (many zeros)
interaction_matrix[np.random.random((n_users, n_items)) > 0.3] = 0
print(f"User-Item Matrix Shape: {interaction_matrix.shape}")
print(f"Sparsity: {(interaction_matrix == 0).sum() / interaction_matrix.size:.2%}")
# User-based collaborative filtering
print("\n=== User-Based Collaborative Filtering ===")
# Normalize ratings
user_means = np.nanmean(np.where(interaction_matrix != 0, interaction_matrix, np.nan), axis=1, keepdims=True)
user_means[np.isnan(user_means)] = 0
interaction_normalized = interaction_matrix - user_means
# Convert to sparse matrix
interaction_sparse = csr_matrix(interaction_normalized)
# Compute user-user similarity
user_similarity = cosine_similarity(interaction_sparse)
print(f"User Similarity Matrix Shape: {user_similarity.shape}")
print(f"Sample user similarity [0,1]: {user_similarity[0, 1]:.4f}")
# 2. Item-based collaborative filtering
print("\n=== Item-Based Collaborative Filtering ===")
# Compute item-item similarity
item_similarity = cosine_similarity(interaction_sparse.T)
print(f"Item Similarity Matrix Shape: {item_similarity.shape}")
print(f"Sample item similarity [0,1]: {item_similarity[0, 1]:.4f}")
# 3. Matrix Factorization (SVD)
print("\n=== Matrix Factorization (SVD) ===")
# Apply SVD
svd = TruncatedSVD(n_components=5, random_state=42)
user_factors = svd.fit_transform(interaction_sparse)
item_factors = svd.components_.T
print(f"User Factors Shape: {user_factors.shape}")
print(f"Item Factors Shape: {item_factors.shape}")
print(f"Explained Variance Ratio: {svd.explained_variance_ratio_.sum():.4f}")
# Reconstruct ratings
reconstructed_ratings = user_factors @ item_factors.T + user_means
print(f"Reconstructed Ratings Shape: {reconstructed_ratings.shape}")
print(f"Reconstruction Error: {np.mean((interaction_matrix - reconstructed_ratings) ** 2):.4f}")
# 4. Content-Based Filtering
print("\n=== Content-Based Filtering ===")
# Create item features (e.g., product descriptions)
item_descriptions = [
"action adventure movie thriller",
"romantic comedy drama love",
"sci-fi technology future space",
"horror scary thriller dark",
"animation family kids fun",
"adventure action explosions",
"documentary educational learning",
"sports competition championship",
"musical dance entertainment",
"historical drama biography"
]
# Expand to 30 items
item_descriptions = (item_descriptions * 4)[:30]
# Create TF-IDF vectors
tfidf = TfidfVectorizer(lowercase=True)
item_features = tfidf.fit_transform(item_descriptions)
# Compute item-item similarity based on content
content_similarity = cosine_similarity(item_features)
print(f"Item Feature Matrix Shape: {item_features.shape}")
print(f"Content-based Item Similarity [0,1]: {content_similarity[0, 1]:.4f}")
# 5. Hybrid Recommendation System
print("\n=== Hybrid Recommendation System ===")
class HybridRecommender:
def __init__(self, user_similarity, item_similarity, interaction_matrix):
self.user_similarity = user_similarity
self.item_similarity = item_similarity
self.interaction_matrix = interaction_matrix
self.n_users = interaction_matrix.shape[0]
self.n_items = interaction_matrix.shape[1]
def recommend_user_based(self, user_id, n_recommendations=5):
"""User-based collaborative filtering recommendation"""
# Get similar users
similar_users = self.user_similarity[user_id]
similar_indices = np.argsort(similar_users)[-5:-1] # Top 4 similar users
# Get items rated highly by similar users
similar_users_ratings = self.interaction_matrix[similar_indices]
user_items = self.interaction_matrix[user_id]
# Items not rated by user but rated by similar users
recommendations = {}
for item_id in range(self.n_items):
if user_items[item_id] == 0:
avg_rating = np.mean(similar_users_ratings[:, item_id])
if avg_rating > 2:
recommendations[item_id] = avg_rating
top_items = sorted(recommendations.items(), key=lambda x: x[1], reverse=True)[:n_recommendations]
return top_items
def recommend_item_based(self, user_id, n_recommendations=5):
"""Item-based collaborative filtering recommendation"""
# Get items rated by user
user_items = self.interaction_matrix[user_id]
rated_items = np.where(user_items > 0)[0]
if len(rated_items) == 0:
return []
# Find similar items
recommendations = {}
for rated_item in rated_items:
similar_items = self.item_similarity[rated_item]
similar_indices = np.argsort(similar_items)[-10:]
for sim_item in similar_indices:
if user_items[sim_item] == 0:
if sim_item not in recommendations:
recommendations[sim_item] = 0
recommendations[sim_item] += user_items[rated_item] * similar_items[sim_item]
top_items = sorted(recommendations.items(), key=lambda x: x[1], reverse=True)[:n_recommendations]
return top_items
def get_hybrid_recommendations(self, user_id, n_recommendations=5, alpha=0.5):
"""Hybrid approach combining user-based and item-based"""
user_based = dict(self.recommend_user_based(user_id, n_recommendations * 2))
item_based = dict(self.recommend_item_based(user_id, n_recommendations * 2))
hybrid = {}
for item_id in set(list(user_based.keys()) + list(item_based.keys())):
score = (alpha * user_based.get(item_id, 0) +
(1 - alpha) * item_based.get(item_id, 0))
hybrid[item_id] = score
top_items = sorted(hybrid.items(), key=lambda x: x[1], reverse=True)[:n_recommendations]
return top_items
# Create recommender
recommender = HybridRecommender(user_similarity, item_similarity, interaction_matrix)
# Generate recommendations for user 0
print("\nRecommendations for User 0:")
print("User-Based:", recommender.recommend_user_based(0, 5))
print("Item-Based:", recommender.recommend_item_based(0, 5))
print("Hybrid:", recommender.get_hybrid_recommendations(0, 5))
# 6. Evaluation Metrics
print("\n=== Recommendation Metrics ===")
class RecommendationMetrics:
@staticmethod
def precision_at_k(actual, predicted, k=5):
"""Precision at K"""
pred_k = predicted[:k]
hits = len(set(actual) & set(pred_k))
return hits / k if k > 0 else 0
@staticmethod
def recall_at_k(actual, predicted, k=5):
"""Recall at K"""
pred_k = predicted[:k]
hits = len(set(actual) & set(pred_k))
return hits / len(actual) if len(actual) > 0 else 0
@staticmethod
def ndcg_at_k(actual, predicted, k=5):
"""Normalized Discounted Cumulative Gain"""
pred_k = predicted[:k]
dcg = sum([1 / np.log2(i + 2) for i, item in enumerate(pred_k) if item in actual])
idcg = sum([1 / np.log2(i + 2) for i in range(min(len(actual), k))])
return dcg / idcg if idcg > 0 else 0
# Compute metrics
actual_items = [1, 5, 8, 12]
predicted_items = [1, 3, 5, 7, 9, 12, 15]
p5 = RecommendationMetrics.precision_at_k(actual_items, predicted_items, 5)
r5 = RecommendationMetrics.recall_at_k(actual_items, predicted_items, 5)
ndcg5 = RecommendationMetrics.ndcg_at_k(actual_items, predicted_items, 5)
print(f"Precision@5: {p5:.4f}")
print(f"Recall@5: {r5:.4f}")
print(f"NDCG@5: {ndcg5:.4f}")
# 7. Cold Start Problem Handling
print("\n=== Cold Start Problem ===")
class ColdStartHandler:
def __init__(self, interaction_matrix):
self.interaction_matrix = interaction_matrix
self.item_popularity = interaction_matrix.sum(axis=0)
self.item_quality = (interaction_matrix > 0).sum(axis=0) / len(interaction_matrix)
def recommend_for_new_user(self, n_recommendations=5):
"""Recommend popular items for new user"""
scores = self.item_popularity + self.item_quality * 100
top_items = np.argsort(scores)[-n_recommendations:][::-1]
return list(top_items)
def recommend_for_new_item(self, n_recommendations=5):
"""Recommend new item to users who liked similar items"""
# Return users most likely to rate new item
user_activity = (self.interaction_matrix > 0).sum(axis=1)
active_users = np.argsort(user_activity)[-n_recommendations:][::-1]
return list(active_users)
cold_start = ColdStartHandler(interaction_matrix)
print("Popular items for new user:", cold_start.recommend_for_new_user(5))
# 8. Visualization
print("\n=== Visualization ===")
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
# User similarity heatmap
axes[0, 0].imshow(user_similarity[:10, :10], cmap='YlOrRd', aspect='auto')
axes[0, 0].set_title('User Similarity Matrix (First 10 Users)')
axes[0, 0].set_xlabel('User ID')
axes[0, 0].set_ylabel('User ID')
plt.colorbar(axes[0, 0].images[0], ax=axes[0, 0])
# Item similarity heatmap
axes[0, 1].imshow(item_similarity[:10, :10], cmap='YlOrRd', aspect='auto')
axes[0, 1].set_title('Item Similarity Matrix (First 10 Items)')
axes[0, 1].set_xlabel('Item ID')
axes[0, 1].set_ylabel('Item ID')
plt.colorbar(axes[0, 1].images[0], ax=axes[0, 1])
# Interaction matrix
axes[1, 0].imshow(interaction_matrix[:15, :15], cmap='Blues', aspect='auto')
axes[1, 0].set_title('User-Item Interaction Matrix (First 15x15)')
axes[1, 0].set_xlabel('Item ID')
axes[1, 0].set_ylabel('User ID')
plt.colorbar(axes[1, 0].images[0], ax=axes[1, 0])
# Rating distribution
rating_counts = np.bincount(interaction_matrix.flatten(), minlength=6)
axes[1, 1].bar(range(6), rating_counts, color='steelblue', edgecolor='black')
axes[1, 1].set_xlabel('Rating')
axes[1, 1].set_ylabel('Frequency')
axes[1, 1].set_title('Rating Distribution')
axes[1, 1].grid(True, alpha=0.3, axis='y')
plt.tight_layout()
plt.savefig('recommendation_analysis.png', dpi=100, bbox_inches='tight')
print("\nVisualization saved as 'recommendation_analysis.png'")
# 9. Summary
print("\n=== Recommendation Summary ===")
print(f"Total Users: {n_users}")
print(f"Total Items: {n_items}")
print(f"Total Interactions: {(interaction_matrix > 0).sum()}")
print(f"Sparsity: {(interaction_matrix == 0).sum() / interaction_matrix.size:.2%}")
print(f"Avg interactions per user: {(interaction_matrix > 0).sum() / n_users:.2f}")
print(f"Avg interactions per item: {(interaction_matrix > 0).sum() / n_items:.2f}")
print("\nRecommendation engine setup completed!")Algorithm Comparison
- User-CF: Good for diverse preferences, but suffers from sparsity
- Item-CF: Better for stable preferences, works well with sparse data
- SVD: Captures latent factors, computationally efficient
- Neural Networks: Captures complex patterns, requires more data
Common Challenges
- Cold Start: New users/items with no interaction history
- Sparsity: Sparse interaction matrices
- Scalability: Handling millions of users and items
- Diversity: Avoiding recommendation bubbles
- Fairness: Representing all item types fairly
Deliverables
- Recommendation model
- Ranked item predictions
- Evaluation metrics
- A/B testing framework
- Deployment code
- Performance analysis