Firecrawl Self-Hosted Operations
Self-hosted Firecrawl deployment, troubleshooting, and best practices.
Host: littleblack (172.25.236.1) via ZeroTier Source: https://github.com/mendableai/firecrawl
When to Use This Skill
Use this skill when:
- Scraping JavaScript-heavy web pages that WebFetch cannot handle
- Extracting content from Gemini/ChatGPT share links
- Operating the self-hosted Firecrawl instance on littleblack
- Troubleshooting Docker container or ZeroTier connectivity issues
- Setting up new Firecrawl deployments with proper restart policies
Architecture Overview
┌─────────────────────────────────────────────────────────────────┐
│ LittleBlack (172.25.236.1) │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Client │───▶│ Scraper │───▶│ Firecrawl │ │
│ │ (curl) │ │ Wrapper :3003│ │ API :3002 │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │
│ │ │ ▼ │
│ │ │ ┌──────────────┐ │
│ │ │ │ Playwright │ │
│ │ │ │ Service │ │
│ │ │ └──────────────┘ │
│ │ │ │ │
│ │ ▼ ▼ │
│ │ ┌──────────────┐ ┌──────────────┐ │
│ │ │ Caddy :8080 │ │ Redis │ │
│ │ │ (files) │ │ RabbitMQ │ │
│ ▼ └──────────────┘ └──────────────┘ │
│ ┌──────────────┐ │
│ │ Output URL │◀── http://172.25.236.1:8080/NAME-TS.md │
│ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘Quick Reference
| Port | Service | Type | Purpose |
|---|---|---|---|
| 3002 | Firecrawl API | Docker | Core scraping engine |
| 3003 | Scraper Wrapper | Bun | Saves to file, returns URL |
| 8080 | Caddy | Binary | Serves saved markdown |
Usage
Recommended: Wrapper Endpoint
curl "http://172.25.236.1:3003/scrape?url=URL&name=NAME"Returns:
{
"url": "http://172.25.236.1:8080/NAME-TIMESTAMP.md",
"file": "NAME-TIMESTAMP.md"
}Direct API (Advanced)
curl -s -X POST http://172.25.236.1:3002/v1/scrape \
-H "Content-Type: application/json" \
-d '{"url":"URL","formats":["markdown"],"waitFor":5000}' \
| jq -r '.data.markdown'Health Checks
Quick Status
# All containers running?
ssh littleblack 'docker ps --filter "name=firecrawl" --format "{{.Names}}: {{.Status}}"'
# API responding?
ssh littleblack 'curl -s -o /dev/null -w "%{http_code}" http://localhost:3002/v1/scrape'
# Expected: 401 (no payload) or 200 (with payload)
# Wrapper responding?
curl -s -o /dev/null -w "%{http_code}" "http://172.25.236.1:3003/health"Detailed Status
# systemd services
ssh littleblack "systemctl --user status firecrawl firecrawl-scraper caddy-firecrawl"
# Docker container details
ssh littleblack 'docker ps -a --filter "name=firecrawl" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'
# Logs (live)
ssh littleblack "journalctl --user -u firecrawl -u firecrawl-scraper -u caddy-firecrawl -f"Troubleshooting
For detailed symptom-based troubleshooting (API container stopped, wrapper unresponsive, Caddy down, ZeroTier unreachable), see Troubleshooting Guide.
Bootstrap: Fresh Installation
Complete 7-step installation guide (clone, Docker config, env vars, systemd services, Caddy file server) is in Bootstrap Guide.
Best Practices
Empirically verified patterns for Docker restart policies, YAML anchors, systemd services, and health monitoring are in Best Practices.
Files Reference
| Path on LittleBlack | Purpose |
|---|---|
~/firecrawl/ | Firecrawl Docker deployment |
~/firecrawl/docker-compose.yaml | Docker orchestration (EDIT THIS) |
~/firecrawl/.env | Environment configuration |
~/firecrawl-scraper.ts | Bun wrapper script |
~/firecrawl-output/ | Saved markdown files (Caddy root) |
~/caddy | Caddy binary |
~/.config/systemd/user/ | User systemd services |
Recovery Commands Cheatsheet
# Full restart (all services)
ssh littleblack 'cd ~/firecrawl && docker compose restart'
ssh littleblack 'systemctl --user restart firecrawl-scraper caddy-firecrawl'
# Check everything
ssh littleblack 'docker ps --filter "name=firecrawl" && systemctl --user status firecrawl-scraper caddy-firecrawl --no-pager'
# Logs (last 100 lines)
ssh littleblack 'docker logs firecrawl-api-1 --tail 100'
ssh littleblack 'journalctl --user -u firecrawl-scraper --no-pager -n 100'
# Force recreate with new config
ssh littleblack 'cd ~/firecrawl && docker compose up -d --force-recreate'
# Verify restart policies
ssh littleblack 'docker inspect --format "{{.Name}}: RestartPolicy={{.HostConfig.RestartPolicy.Name}}" $(docker ps -a --filter "name=firecrawl" -q)'Related Documentation
- Firecrawl Official Docs - API reference
- Docker Compose Restart - Policy options