Logging API Requests
Overview
Implement structured API request logging with correlation IDs, performance timing, security audit trails, and PII redaction. Capture request/response metadata in JSON format suitable for aggregation in ELK Stack, Loki, or CloudWatch Logs, enabling debugging, performance analysis, and compliance auditing across distributed services.
Prerequisites
- Structured logging library: Pino or Winston (Node.js), structlog (Python), Logback with JSON encoder (Java)
- Log aggregation system: ELK Stack (Elasticsearch, Logstash, Kibana), Grafana Loki, or CloudWatch Logs
- Correlation ID propagation mechanism (middleware-injected or from incoming
X-Request-IDheader) - PII data classification for the API domain (which fields contain personal data requiring redaction)
- Log retention and rotation policy defined per compliance requirements
Instructions
- Examine existing logging configuration using Grep and Read to identify current log format, output destinations, and any structured logging already in place.
- Implement request logging middleware that captures: timestamp (ISO 8601), correlation ID, HTTP method, URL path (without query string PII), status code, response time (ms), request size, response size, and client IP.
- Generate a unique correlation ID (
X-Request-ID) for each request if not provided by the caller, and propagate it to all downstream service calls and log entries within the request scope. - Add PII redaction rules that mask sensitive fields (passwords, tokens, SSNs, email addresses) in logged request/response bodies using configurable field-path patterns.
- Implement log levels per context:
infofor successful requests,warnfor 4xx client errors,errorfor 5xx server errors with stack traces, anddebugfor request/response bodies (development only). - Configure response body logging for error responses only (4xx/5xx), capturing the error payload for debugging while skipping successful response bodies to reduce log volume.
- Add security audit logging for sensitive operations: authentication attempts, permission changes, data exports, and admin actions, tagged with
audit: truefor separate indexing. - Set up log rotation and retention policies: 30 days for application logs, 90 days for audit logs, with automatic compression of logs older than 7 days.
- Write tests verifying that PII redaction works correctly, correlation IDs propagate through nested calls, and log output matches expected JSON structure.
See ${CLAUDE_SKILL_DIR}/references/implementation.md for the full implementation guide.
Output
${CLAUDE_SKILL_DIR}/src/middleware/request-logger.js- Structured request/response logging middleware${CLAUDE_SKILL_DIR}/src/middleware/correlation-id.js- Correlation ID generation and propagation${CLAUDE_SKILL_DIR}/src/utils/pii-redactor.js- Field-level PII redaction with configurable patterns${CLAUDE_SKILL_DIR}/src/utils/audit-logger.js- Security audit event logger for sensitive operations${CLAUDE_SKILL_DIR}/src/config/logging.js- Log level, format, and output destination configuration${CLAUDE_SKILL_DIR}/tests/logging/- Logging middleware tests including PII redaction verification
Error Handling
| Error | Cause | Solution |
|---|---|---|
| Log volume overwhelming storage | High-traffic endpoint logging full request/response bodies | Log bodies only for errors; sample successful request bodies at configurable rate (1%) |
| PII leak in logs | New field added to API response containing personal data not covered by redaction rules | Maintain allowlist of loggable fields rather than blocklist; audit log output regularly |
| Correlation ID missing | Upstream service does not propagate X-Request-ID header | Generate new correlation ID when header is absent; log warning about missing upstream propagation |
| Log parsing failure | Log message contains unescaped characters breaking JSON structure | Use structured logging library that handles serialization; never concatenate user input into log strings |
| Audit log gap | Async logging dropped events during high-load period | Use synchronous logging for audit events; implement write-ahead buffer for audit trail completeness |
Refer to ${CLAUDE_SKILL_DIR}/references/errors.md for comprehensive error patterns.
Examples
Structured JSON log entry: {"timestamp":"2026-03-10T14:30:00Z","correlationId":"abc-123","method":"POST","path":"/api/users","status":201,"durationMs":45,"userId":"usr_456","audit":false} -- every field queryable in log aggregation.
Distributed tracing correlation: Propagate X-Request-ID from API gateway through 3 microservices, enabling a single Kibana query to show the complete request lifecycle across all services.
Compliance audit trail: Tag all data modification operations (POST, PUT, DELETE) with audit: true, capturing the authenticated user, modified resource ID, and change summary for SOC 2 compliance evidence.
See ${CLAUDE_SKILL_DIR}/references/examples.md for additional examples.
Resources
- Structured logging best practices (12-Factor App: Logs)
- ELK Stack documentation: https://www.elastic.co/guide/
- Pino logger: https://getpino.io/
- OpenTelemetry for distributed tracing context propagation