Scalability Advisor
Provides systematic guidance for scaling systems at different growth stages, identifying bottlenecks, and designing for horizontal scalability.
When to Use
- Planning for 10x, 100x, or 1000x growth
- Diagnosing current performance bottlenecks
- Designing new systems for scale
- Evaluating scaling strategies (vertical vs. horizontal)
- Capacity planning and infrastructure sizing
Scaling Stages Framework
Stage Overview
┌─────────────────────────────────────────────────────────────────────┐
│ SCALING JOURNEY │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Stage 1 Stage 2 Stage 3 Stage 4 │
│ Startup Growth Scale Enterprise │
│ 0-10K users 10K-100K 100K-1M 1M+ users │
│ │
│ Single Add caching, Horizontal Global, │
│ server read replicas scaling multi-region │
│ │
│ $100/mo $1K/mo $10K/mo $100K+/mo │
└─────────────────────────────────────────────────────────────────────┘
Stage 1: Startup (0-10K Users)
Architecture
┌────────────────────────────────────────┐
│ Single Server │
│ ┌──────────────────────────────────┐ │
│ │ App Server (Node/Python/etc) │ │
│ │ + Database (PostgreSQL) │ │
│ │ + File Storage (local/S3) │ │
│ └──────────────────────────────────┘ │
└────────────────────────────────────────┘
Key Metrics
| Metric | Target | Warning |
|---|
| Response time (P95) | < 500ms | > 1s |
| Database queries/request | < 10 | > 20 |
| Server CPU | < 70% | > 85% |
| Database connections | < 50% pool | > 80% pool |
What to Focus On
DO:
- Write clean, maintainable code
- Use database indexes on frequently queried columns
- Implement basic monitoring (uptime, errors)
- Keep architecture simple (monolith is fine)
DON'T:
- Over-engineer for scale you don't have
- Add caching before you need it
- Split into microservices prematurely
- Worry about multi-region yet
When to Move to Stage 2
- Database CPU consistently > 70%
- Response times degrading
- Single queries taking > 100ms
- Server resources maxed
Stage 2: Growth (10K-100K Users)
Architecture
┌─────────────────────────────────────────────────────────────┐
│ │
│ ┌─────────┐ ┌─────────────────────────────────┐ │
│ │ CDN │ │ Load Balancer │ │
│ └────┬────┘ └──────────────┬──────────────────┘ │
│ │ │ │
│ │ ┌──────────────┼──────────────┐ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Static │ │ App 1 │ │ App 2 │ │ App 3 │ │
│ │ Assets │ └────┬────┘ └────┬────┘ └────┬────┘ │
│ └─────────┘ │ │ │ │
│ └──────────────┼────────────┘ │
│ │ │
│ ┌──────────────┼──────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Primary │ │ Read │ │ Redis │ │
│ │ DB │───│ Replica │ │ Cache │ │
│ └─────────┘ └─────────┘ └─────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Key Additions
| Component | Purpose | When to Add |
|---|
| CDN | Static asset caching | Images, JS, CSS taking > 20% bandwidth |
| Load Balancer | Distribute traffic | Single server CPU > 70% |
| Read Replicas | Offload reads | > 80% database ops are reads |
| Redis Cache | Application caching | Same queries repeated frequently |
| Job Queue | Async processing | Background tasks blocking requests |
Caching Strategy
Request Flow with Caching:
1. Check CDN (static assets) ─► HIT: Return cached
│
2. Check Application Cache (Redis) ─► HIT: Return cached
│
3. Check Database ─► Return + Cache result
What to Cache:
- Session data (TTL: session duration)
- User profile data (TTL: 5-15 minutes)
- API responses (TTL: varies by freshness needs)
- Database query results (TTL: 1-5 minutes)
- Computed values (TTL: based on computation cost)
Database Optimization
-- Find slow queries
SELECT query, calls, mean_time, total_time
FROM pg_stat_statements
ORDER BY total_time DESC
LIMIT 20;
-- Find missing indexes
SELECT schemaname, tablename, indexrelname, idx_scan, seq_scan
FROM pg_stat_user_indexes
WHERE idx_scan = 0 AND seq_scan > 1000;
When to Move to Stage 3
- Write traffic overwhelming single primary
- Cache hit rate plateauing despite optimization
- Read replicas can't keep up with replication lag
- Need independent scaling of components
Stage 3: Scale (100K-1M Users)
Architecture
┌──────────────────────────────────────────────────────────────────────┐
│ CDN / Edge │
└──────────────────────────────────────────────────────────────────────┘
│
┌──────────────────────────────────────────────────────────────────────┐
│ API Gateway │
│ (Rate limiting, Auth, Routing) │
└──────────────────────────────────────────────────────────────────────┘
│
┌───────────────────────────┼───────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Service A │ │ Service B │ │ Service C │
│ (Users) │ │ (Orders) │ │ (Search) │
│ Auto-scale │ │ Auto-scale │ │ Auto-scale │
└───────┬───────┘ └───────┬───────┘ └───────┬───────┘
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ User DB │ │ Order DB │ │ Elasticsearch │
│ (Sharded) │ │ (Sharded) │ │ (Cluster) │
└───────────────┘ └───────────────┘ └───────────────┘
│
▼
┌───────────────────────────┐
│ Message Queue │
│ (Kafka / SQS) │
└───────────────────────────┘
Key Patterns
Database Sharding
Sharding Strategies:
1. Hash-based (user_id % num_shards)
PRO: Even distribution
CON: Hard to add shards
2. Range-based (user_id 1-1M → shard 1)
PRO: Easy to add shards
CON: Hotspots possible
3. Directory-based (lookup table)
PRO: Flexible
CON: Lookup overhead
Event-Driven Architecture
Synchronous → Asynchronous
Before:
API → Service A → Service B → Service C → Response (slow)
After:
API → Service A → Queue → Response (fast)
↓
Service B, C process async
Scaling Checklist
- Stateless application servers (no local state)
- Database read/write separation
- Asynchronous processing for non-critical paths
- Circuit breakers between services
- Distributed tracing implemented
- Auto-scaling configured with proper metrics
- Database connection pooling (PgBouncer, ProxySQL)
When to Move to Stage 4
- Need geographic distribution for latency
- Regulatory requirements (data residency)
- Single region can't handle failover
- Global user base with latency requirements
Stage 4: Enterprise (1M+ Users)
Architecture
┌─────────────────────────────────────────────────────────────────────────┐
│ Global Load Balancer │
│ (GeoDNS, Anycast, Route53) │
└─────────────────────────────────────────────────────────────────────────┘
│ │
┌────────┴────────┐ ┌───────┴────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ US-East │ │ US-West │ │ EU-West │ │ AP-South │
│ Region │ │ Region │ │ Region │ │ Region │
│ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │
│ │Services│ │ │ │Services│ │ │ │Services│ │ │ │Services│ │
│ └────────┘ │ │ └────────┘ │ │ └────────┘ │ │ └────────┘ │
│ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │
│ │Database│ │ │ │Database│ │ │ │Database│ │ │ │Database│ │
│ │(Primary)│ │ │ │(Replica)│ │ │ │(Primary)│ │ │ │(Replica)│ │
│ └────────┘ │ │ └────────┘ │ │ └────────┘ │ │ └────────┘ │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
│ │
└─────────┬─────────┘
│
Cross-Region Replication
Multi-Region Patterns
| Pattern | Consistency | Latency | Complexity |
|---|
| Active-Passive | Strong | High failover | Low |
| Active-Active | Eventual | Low | High |
| Follow-the-Sun | Strong per region | Medium | Medium |
Data Consistency Strategies
CAP Theorem Trade-offs:
Strong Consistency (CP):
- All regions see same data
- Higher latency for writes
- Use for: Financial transactions, inventory
Eventual Consistency (AP):
- Regions may have stale data briefly
- Low latency always
- Use for: Social feeds, analytics, non-critical
Causal Consistency:
- Related operations ordered correctly
- Balance of latency and correctness
- Use for: Messaging, collaboration
Enterprise Checklist
- Multi-region deployment
- Cross-region data replication
- Global CDN with edge caching
- Disaster recovery tested
- Compliance (SOC 2, GDPR, data residency)
- 99.99% SLA architecture
- Zero-downtime deployments
- Chaos engineering practice
Bottleneck Diagnosis Guide
Finding the Bottleneck
Systematic Diagnosis:
1. Where is time spent?
└─► Distributed tracing (Jaeger, Datadog)
2. Is it the database?
└─► Check slow query logs, connection pool
3. Is it the application?
└─► CPU profiling, memory analysis
4. Is it the network?
└─► Latency between services, DNS resolution
5. Is it external services?
└─► Third-party API latency, rate limits
Common Bottlenecks by Layer
| Layer | Symptoms | Solutions |
|---|
| Database | Slow queries, high CPU | Indexing, read replicas, caching |
| Application | High CPU, memory | Optimize code, scale horizontally |
| Network | High latency, timeouts | CDN, edge caching, connection pooling |
| Storage | Slow I/O, high wait | SSD, object storage, caching |
| External APIs | Timeouts, rate limits | Circuit breakers, caching, fallbacks |
Database Bottleneck Checklist
## Quick Database Health Check
1. Connection Pool
- Current connections vs max?
- Connection wait time?
- Pool exhaustion events?
2. Query Performance
- Slowest queries (pg_stat_statements)?
- Missing indexes (seq scans > 10K)?
- Lock contention?
3. Replication
- Replica lag?
- Write throughput?
- Read distribution?
4. Storage
- Disk I/O wait?
- Table/index bloat?
- WAL write latency?
Scaling Calculations
Capacity Planning Formula
Required Capacity = Peak Traffic × Growth Factor × Safety Margin
Example:
- Current peak: 1,000 req/sec
- Expected growth: 3x in 12 months
- Safety margin: 1.5x
Required: 1,000 × 3 × 1.5 = 4,500 req/sec capacity
Database Sizing
Connection Pool Size:
connections = (num_cores × 2) + effective_spindle_count
Example: 8 cores, SSD
connections = (8 × 2) + 1 = 17 connections per instance
Read Replica Sizing:
replicas = ceiling(read_traffic / single_replica_capacity)
Example: 10,000 reads/sec, 3,000/replica capacity
replicas = ceiling(10,000 / 3,000) = 4 replicas
Cache Sizing
Cache Size:
memory = working_set_size × (1 + overhead_factor)
Working set = frequently accessed data (usually 10-20% of total)
Overhead = ~1.5x for Redis data structures
Example: 10GB working set
Redis memory = 10GB × 1.5 = 15GB
Quick Reference
Scaling Decision Matrix
| Symptom | First Try | Then Try | Finally |
|---|
| Slow page loads | Add caching | CDN | Edge compute |
| Database slow | Add indexes | Read replicas | Sharding |
| API timeouts | Async processing | Circuit breakers | Event-driven |
| High server CPU | Vertical scale | Horizontal scale | Optimize code |
| High memory | Increase RAM | Fix memory leaks | Redesign data structures |
Infrastructure Cost at Scale
| Users | Architecture | Monthly Cost |
|---|
| 10K | Single server | $100-300 |
| 100K | Load balanced + cache | $1,000-3,000 |
| 1M | Microservices + sharding | $10,000-30,000 |
| 10M | Multi-region | $100,000+ |
References
