kafka-iac-deployment（Kafka IAC 部署）

Kafka Infrastructure as Code (IaC) Deployment

Expert guidance for deploying Apache Kafka using Terraform across multiple platforms.

When to Use This Skill

I activate when you need help with:

• Terraform deployments: "Deploy Kafka with Terraform", "provision Kafka cluster"
• Platform selection: "Should I use AWS MSK or self-hosted Kafka?", "compare Kafka platforms"
• Infrastructure planning: "How to size Kafka infrastructure", "Kafka on AWS vs Azure"
• IaC automation: "Automate Kafka deployment", "CI/CD for Kafka infrastructure"

What I Know

Available Terraform Modules

This plugin provides 3 production-ready Terraform modules:

1. Apache Kafka (Self-Hosted, KRaft Mode)

• Location: plugins/specweave-kafka/terraform/apache-kafka/
• Platform: AWS EC2 (can adapt to other clouds)
• Architecture: KRaft mode (no ZooKeeper dependency)
• Features:

- Multi-broker cluster (3-5 brokers recommended) - Security groups with SASL_SSL - IAM roles for S3 backups - CloudWatch metrics and alarms - Auto-scaling group support - Custom VPC and subnet configuration

• Use When:

- ✅ You need full control over Kafka configuration - ✅ Running Kafka 3.6+ (KRaft mode) - ✅ Want to avoid ZooKeeper operational overhead - ✅ Multi-cloud or hybrid deployments

• Variables: module "kafka" {source = "../../plugins/specweave-kafka/terraform/apache-kafka" environment = "production" broker_count = 3 kafka_version = "3.7.0" instance_type = "m5.xlarge" vpc_id = var.vpc_id subnet_ids = var.subnet_ids domain = "example.com" enable_s3_backups = true enable_monitoring = true}

2. AWS MSK (Managed Streaming for Kafka)

• Location: plugins/specweave-kafka/terraform/aws-msk/
• Platform: AWS Managed Service
• Features:

- Fully managed Kafka service - IAM authentication + SASL/SCRAM - Auto-scaling (provisioned throughput) - Built-in monitoring (CloudWatch) - Multi-AZ deployment - Encryption in transit and at rest

• Use When:

- ✅ You want AWS to manage Kafka operations - ✅ Need tight AWS integration (IAM, KMS, CloudWatch) - ✅ Prefer operational simplicity over cost - ✅ Running in AWS VPC

• Variables: module "msk" {source = "../../plugins/specweave-kafka/terraform/aws-msk" cluster_name = "my-kafka-cluster" kafka_version = "3.6.0" number_of_broker_nodes = 3 broker_node_instance_type = "kafka.m5.large" vpc_id = var.vpc_id subnet_ids = var.private_subnet_ids enable_iam_auth = true enable_scram_auth = false enable_auto_scaling = true}

3. Azure Event Hubs (Kafka API)

• Location: plugins/specweave-kafka/terraform/azure-event-hubs/
• Platform: Azure Managed Service
• Features:

- Kafka 1.0+ protocol support - Auto-inflate (elastic scaling) - Premium SKU for high throughput - Zone redundancy - Private endpoints (VNet integration) - Event capture to Azure Storage

• Use When:

- ✅ Running on Azure cloud - ✅ Need Kafka-compatible API without Kafka operations - ✅ Want serverless scaling (auto-inflate) - ✅ Integrating with Azure ecosystem

• Variables: module "event_hubs" {source = "../../plugins/specweave-kafka/terraform/azure-event-hubs" namespace_name = "my-event-hub-ns" resource_group_name = var.resource_group_name location = "eastus" sku = "Premium" capacity = 1 kafka_enabled = true auto_inflate_enabled = true maximum_throughput_units = 20}

Platform Selection Decision Tree

Need Kafka deployment? START HERE:

├─ Running on AWS?
│  ├─ YES → Want managed service?
│  │  ├─ YES → Use AWS MSK module (terraform/aws-msk)
│  │  └─ NO → Use Apache Kafka module (terraform/apache-kafka)
│  └─ NO → Continue...
│
├─ Running on Azure?
│  ├─ YES → Use Azure Event Hubs module (terraform/azure-event-hubs)
│  └─ NO → Continue...
│
├─ Multi-cloud or hybrid?
│  └─ YES → Use Apache Kafka module (most portable)
│
├─ Need maximum control?
│  └─ YES → Use Apache Kafka module
│
└─ Default → Use Apache Kafka module (self-hosted, KRaft mode)

Deployment Workflows

Workflow 1: Deploy Self-Hosted Kafka (Apache Kafka Module)

Scenario: You want full control over Kafka on AWS EC2

# 1. Create Terraform configuration
cat > main.tf <<EOF
module "kafka_cluster" {
  source = "../../plugins/specweave-kafka/terraform/apache-kafka"

  environment         = "production"
  broker_count        = 3
  kafka_version       = "3.7.0"
  instance_type       = "m5.xlarge"

  vpc_id     = "vpc-12345678"
  subnet_ids = ["subnet-abc", "subnet-def", "subnet-ghi"]
  domain     = "kafka.example.com"

  enable_s3_backups = true
  enable_monitoring = true

  tags = {
    Project     = "MyApp"
    Environment = "Production"
  }
}

output "broker_endpoints" {
  value = module.kafka_cluster.broker_endpoints
}
EOF

# 2. Initialize Terraform
terraform init

# 3. Plan deployment (review what will be created)
terraform plan

# 4. Apply (create infrastructure)
terraform apply

# 5. Get broker endpoints
terraform output broker_endpoints
# Output: ["kafka-0.kafka.example.com:9093", "kafka-1.kafka.example.com:9093", ...]

Workflow 2: Deploy AWS MSK (Managed Service)

Scenario: You want AWS to manage Kafka operations

# 1. Create Terraform configuration
cat > main.tf <<EOF
module "msk_cluster" {
  source = "../../plugins/specweave-kafka/terraform/aws-msk"

  cluster_name           = "my-msk-cluster"
  kafka_version          = "3.6.0"
  number_of_broker_nodes = 3
  broker_node_instance_type = "kafka.m5.large"

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnet_ids

  enable_iam_auth     = true
  enable_auto_scaling = true

  tags = {
    Project = "MyApp"
  }
}

output "bootstrap_brokers" {
  value = module.msk_cluster.bootstrap_brokers_sasl_iam
}
EOF

# 2. Deploy
terraform init && terraform apply

# 3. Configure IAM authentication
# (module outputs IAM policy, attach to your application role)

Workflow 3: Deploy Azure Event Hubs (Kafka API)

Scenario: You're on Azure and want Kafka-compatible API

# 1. Create Terraform configuration
cat > main.tf <<EOF
module "event_hubs" {
  source = "../../plugins/specweave-kafka/terraform/azure-event-hubs"

  namespace_name      = "my-kafka-namespace"
  resource_group_name = "my-resource-group"
  location            = "eastus"

  sku                  = "Premium"
  capacity             = 1
  kafka_enabled        = true
  auto_inflate_enabled = true
  maximum_throughput_units = 20

  # Create hubs (topics) for your use case
  hubs = [
    { name = "user-events",    partitions = 12 },
    { name = "order-events",   partitions = 6 },
    { name = "payment-events", partitions = 3 }
  ]
}

output "connection_string" {
  value = module.event_hubs.connection_string
  sensitive = true
}
EOF

# 2. Deploy
terraform init && terraform apply

# 3. Get connection details
terraform output connection_string

Infrastructure Sizing Recommendations

Small Environment (Dev/Test)

# Self-hosted: 1 broker, m5.large
broker_count  = 1
instance_type = "m5.large"

# AWS MSK: 1 broker per AZ, kafka.m5.large
number_of_broker_nodes = 3
broker_node_instance_type = "kafka.m5.large"

# Azure Event Hubs: Basic SKU
sku = "Basic"
capacity = 1

Medium Environment (Staging/Production)

# Self-hosted: 3 brokers, m5.xlarge
broker_count  = 3
instance_type = "m5.xlarge"

# AWS MSK: 3 brokers, kafka.m5.xlarge
number_of_broker_nodes = 3
broker_node_instance_type = "kafka.m5.xlarge"

# Azure Event Hubs: Standard SKU with auto-inflate
sku = "Standard"
capacity = 2
auto_inflate_enabled = true
maximum_throughput_units = 10

Large Environment (High-Throughput Production)

# Self-hosted: 5+ brokers, m5.2xlarge or m5.4xlarge
broker_count  = 5
instance_type = "m5.2xlarge"

# AWS MSK: 6+ brokers, kafka.m5.2xlarge, auto-scaling
number_of_broker_nodes = 6
broker_node_instance_type = "kafka.m5.2xlarge"
enable_auto_scaling = true

# Azure Event Hubs: Premium SKU with zone redundancy
sku = "Premium"
capacity = 4
zone_redundant = true
maximum_throughput_units = 20

Best Practices

Security Best Practices

1. Always use encryption in transit

- Self-hosted: Enable SASL_SSL listener - AWS MSK: Set encryption_in_transit_client_broker = "TLS" - Azure Event Hubs: HTTPS/TLS enabled by default

1. Use IAM authentication (when possible)

- AWS MSK: enable_iam_auth = true - Azure Event Hubs: Managed identities

1. Network isolation

- Deploy in private subnets - Use security groups/NSGs restrictively - Azure: Enable private endpoints for Premium SKU

High Availability Best Practices

1. Multi-AZ deployment

- Self-hosted: Distribute brokers across 3+ AZs - AWS MSK: Automatically multi-AZ - Azure Event Hubs: Enable zone_redundant = true (Premium)

1. Replication factor = 3

- Self-hosted: default.replication.factor=3 - AWS MSK: Configured automatically - Azure Event Hubs: N/A (fully managed)

1. min.insync.replicas = 2

- Ensures durability even if 1 broker fails

Cost Optimization

1. Right-size instances

- Use ClusterSizingCalculator utility (in kafka-architecture skill) - Start small, scale up based on metrics

1. Auto-scaling (where available)

- AWS MSK: enable_auto_scaling = true - Azure Event Hubs: auto_inflate_enabled = true

1. Retention policies

- Set log.retention.hours based on actual needs (default: 168 hours = 7 days) - Shorter retention = lower storage costs

Monitoring Integration

All modules integrate with monitoring:

Self-Hosted Kafka

• CloudWatch metrics (via JMX Exporter)
• Prometheus + Grafana dashboards (see kafka-observability skill)
• Custom CloudWatch alarms

AWS MSK

• Built-in CloudWatch metrics
• Enhanced monitoring available
• Integration with CloudWatch Alarms

Azure Event Hubs

• Built-in Azure Monitor metrics
• Diagnostic logs to Log Analytics
• Integration with Azure Alerts

Troubleshooting

"Terraform destroy fails on security groups"

Cause: Resources using security groups still exist Fix:

# 1. Find dependent resources
aws ec2 describe-network-interfaces --filters "Name=group-id,Values=sg-12345678"

# 2. Delete dependent resources first
# 3. Retry terraform destroy

"AWS MSK cluster takes 20+ minutes to create"

Cause: MSK provisioning is inherently slow (AWS behavior) Fix: This is normal. Use --auto-approve for automation:

terraform apply -auto-approve

"Azure Event Hubs: Connection refused"

Cause: Kafka protocol not enabled OR incorrect connection string Fix:

1. Verify kafka_enabled = true in Terraform
2. Use Kafka connection string (not Event Hubs connection string)
3. Check firewall rules (Premium SKU supports private endpoints)

Integration with Other Skills

• kafka-architecture: For cluster sizing and partitioning strategy
• kafka-observability: For Prometheus + Grafana setup after deployment
• kafka-kubernetes: For deploying Kafka on Kubernetes (alternative to Terraform)
• kafka-cli-tools: For testing deployed clusters with kcat

Quick Reference Commands

# Terraform workflow
terraform init          # Initialize modules
terraform plan          # Preview changes
terraform apply         # Create infrastructure
terraform output        # Get outputs (endpoints, etc.)
terraform destroy       # Delete infrastructure

# AWS MSK specific
aws kafka list-clusters # List MSK clusters
aws kafka describe-cluster --cluster-arn <arn> # Get cluster details

# Azure Event Hubs specific
az eventhubs namespace list # List namespaces
az eventhubs eventhub list --namespace-name <name> --resource-group <rg> # List hubs

Next Steps After Deployment:

1. Use kafka-observability skill to set up Prometheus + Grafana monitoring
2. Use kafka-cli-tools skill to test cluster with kcat
3. Deploy your producer/consumer applications
4. Monitor cluster health and performance