# OpenTofu + Ansible + AWS Best Practices

Comprehensive reference for building infrastructure projects with the two-phase provisioning pattern: OpenTofu creates AWS resources, Ansible configures the application layer via Docker Compose.

---

## OpenTofu

### Provider and version pinning

```hcl
terraform {
  required_version = ">= 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.0"
    }
  }

  backend "s3" {}
}
```

- Always pin `required_version` and provider versions
- Use `~>` (pessimistic constraint) for providers — allows patch updates, blocks breaking changes
- Empty `backend "s3" {}` — actual config comes from `-backend-config` at init time
- Commit the `.terraform.lock.hcl` file — ensures reproducible provider versions

### Variable design

```hcl
# GOOD: No default for required, env-specific values
variable "environment" {
  description = "Environment name (sandbox, prod)"
  type        = string
}

# GOOD: Sensible default for values that rarely change
variable "aws_region" {
  description = "AWS region to deploy resources"
  type        = string
  default     = "eu-central-1"
}

# GOOD: Sensitive with no default — forces explicit provision
variable "ses_smtp_password" {
  description = "SES SMTP password — provide on first apply via -var"
  type        = string
  sensitive   = true
}
```

**Rules:**
- Never put placeholder defaults on secrets (no `default = "changeme"`)
- Mark secrets with `sensitive = true` — prevents accidental exposure in logs/plan output
- Use descriptive `description` fields — they appear in `tofu plan` output
- Group variables by section with comments: Networking, Compute, Database, Secrets, Tags
- Use specific types: `string`, `number`, `bool`, `list(string)`, `map(string)`

### Naming conventions

```hcl
locals {
  name_prefix = "${var.project_name}-${var.environment}"
  common_tags = merge(var.tags, {
    Project     = var.project_name
    Environment = var.environment
    ManagedBy   = "opentofu"
  })
}
```

- Use `local.name_prefix` everywhere for consistent naming
- Use `local.common_tags` merged into every resource's `tags`
- Resource names: `${local.name_prefix}-<purpose>` (e.g., `myapp-sandbox-vpc`)

### Security groups: use name_prefix

```hcl
resource "aws_security_group" "ec2" {
  name_prefix = "${local.name_prefix}-ec2-"   # NOT name =
  vpc_id      = aws_vpc.main.id
  description = "App EC2 instance"

  # ... rules ...

  lifecycle {
    create_before_destroy = true
  }
}
```

- `name_prefix` (not `name`) allows `create_before_destroy` — OpenTofu creates the new SG before deleting the old one, avoiding downtime
- Always include `description` on the SG itself and on each rule
- Never use `0.0.0.0/0` for ingress unless explicitly public — use VPN CIDRs

### Lifecycle rules

```hcl
# For secrets that users update out-of-band:
lifecycle {
  ignore_changes = [value]
}

# For security groups:
lifecycle {
  create_before_destroy = true
}
```

- `ignore_changes` on SSM parameters where users provide initial values via `-var` and then manage them in AWS Console
- `create_before_destroy` on security groups and launch templates

### State management

**Per-environment state files in separate S3 buckets:**

```
backends/
  sandbox.hcl    # bucket = "myorg-tfstate-sandbox"
  prod.hcl       # bucket = "myorg-tfstate-prod"
```

```hcl
# backends/sandbox.hcl
bucket         = "myorg-tfstate-sandbox"
key            = "myapp/sandbox.tfstate"
region         = "eu-central-1"
profile        = "infra-sandbox"
dynamodb_table = "terraform-locks"
encrypt        = true
```

- Separate S3 buckets per AWS account (sandbox vs. prod)
- DynamoDB table for state locking — prevents concurrent applies
- Always `encrypt = true`
- Switch environments with `tofu init -reconfigure -backend-config=backends/<env>.hcl`

### Outputs for Ansible bridge

```hcl
output "ec2_public_ip" {
  description = "Elastic IP of the instance"
  value       = aws_eip.main.public_ip
}

output "rds_host" {
  description = "RDS endpoint hostname"
  value       = aws_db_instance.main.address
}

output "rds_secret_arn" {
  description = "Secrets Manager ARN for RDS credentials"
  value       = aws_secretsmanager_secret.rds_password.arn
}

output "ssm_prefix" {
  description = "SSM Parameter Store prefix"
  value       = "/${local.name_prefix}"
}
```

- Every value Ansible needs must be an output
- The dynamic inventory script reads these via `tofu output -json`
- Include the SSM prefix and Secrets Manager ARNs — Ansible uses them to fetch secrets

---

## AWS Security

### IMDSv2 (mandatory for Docker hosts)

```hcl
metadata_options {
  http_endpoint               = "enabled"
  http_tokens                 = "required"     # IMDSv2 only — blocks IMDSv1
  http_put_response_hop_limit = 2              # Required for Docker containers
}
```

- `http_tokens = "required"` forces IMDSv2 — prevents SSRF attacks via IMDSv1
- `http_put_response_hop_limit = 2` is essential when running Docker — containers are an extra network hop

### Encrypted storage

```hcl
# EC2 EBS
root_block_device {
  volume_size = 30
  volume_type = "gp3"
  encrypted   = true
}

# S3
resource "aws_s3_bucket_server_side_encryption_configuration" "main" {
  bucket = aws_s3_bucket.main.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
    bucket_key_enabled = true
  }
}

# RDS
resource "aws_db_instance" "main" {
  storage_encrypted = true
  # ...
}
```

- Encrypt EBS volumes, S3 buckets (SSE-KMS), and RDS storage — always
- `bucket_key_enabled = true` reduces KMS API costs for S3

### S3 public access block

```hcl
resource "aws_s3_bucket_public_access_block" "main" {
  bucket = aws_s3_bucket.main.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
```

- Always attach this to every S3 bucket — no exceptions

### Explicit private route tables

```hcl
resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id
  # No routes — private subnets have no internet access

  tags = merge(local.common_tags, {
    Name = "${local.name_prefix}-private-rt"
  })
}

resource "aws_route_table_association" "private_a" {
  subnet_id      = aws_subnet.private_a.id
  route_table_id = aws_route_table.private.id
}
```

- Never rely on the VPC's default/main route table — it may have unexpected routes
- Create an explicit private route table with NO routes for RDS subnets
- Always associate subnets to route tables explicitly

### IAM least privilege

```hcl
# GOOD: Scope SES to specific identity
resource "aws_iam_role_policy" "ses_access" {
  policy = jsonencode({
    Statement = [{
      Effect   = "Allow"
      Action   = ["ses:SendEmail", "ses:SendRawEmail"]
      Resource = "arn:aws:ses:${var.aws_region}:${data.aws_caller_identity.current.account_id}:identity/${var.ses_domain}"
    }]
  })
}

# GOOD: Scope secrets read to specific ARNs
resource "aws_iam_role_policy" "secrets_read" {
  policy = jsonencode({
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["secretsmanager:GetSecretValue"]
        Resource = [aws_secretsmanager_secret.rds_password.arn]
      },
      {
        Effect   = "Allow"
        Action   = ["ssm:GetParameter", "ssm:GetParameters"]
        Resource = [
          aws_ssm_parameter.app_secret.arn,
          aws_ssm_parameter.smtp_user.arn,
        ]
      }
    ]
  })
}
```

- Never use `Resource = "*"` for secrets access — list specific ARNs
- Scope SES to the verified identity/domain, not `*`
- Scope S3 to specific bucket ARN + `/*` pattern

### RDS hardening

```hcl
resource "aws_db_instance" "main" {
  publicly_accessible              = false
  storage_encrypted                = true
  copy_tags_to_snapshot            = true
  auto_minor_version_upgrade       = true
  performance_insights_enabled     = true
  backup_retention_period          = 7    # Days
  skip_final_snapshot              = true # false in prod
  deletion_protection              = false # true in prod
}
```

- `publicly_accessible = false` — always, RDS lives in private subnets
- `performance_insights_enabled = true` — free tier available, invaluable for debugging
- `copy_tags_to_snapshot = true` — snapshots inherit resource tags
- `skip_final_snapshot` and `deletion_protection` — toggle between sandbox/prod

---

## Secrets Management

### Pattern: auto-generated secrets

Use `random_password` and `random_id` for secrets that don't need user input:

```hcl
resource "random_password" "rds" {
  length           = 32
  special          = true
  override_special = "!#&*()-_=+"  # Avoid chars that break connection strings
}

resource "random_id" "app_secret" {
  byte_length = 32
}
```

### Pattern: Secrets Manager for database credentials

```hcl
resource "aws_secretsmanager_secret" "rds_password" {
  name_prefix             = "${local.name_prefix}-rds-password-"
  recovery_window_in_days = 0  # Sandbox: allow immediate delete/recreate
  tags                    = local.common_tags
}

resource "aws_secretsmanager_secret_version" "rds_password" {
  secret_id = aws_secretsmanager_secret.rds_password.id
  secret_string = jsonencode({
    username = var.db_username
    password = random_password.rds.result
    host     = aws_db_instance.main.address
    port     = aws_db_instance.main.port
    dbname   = var.db_name
  })
}
```

- `name_prefix` (not `name`) — allows recreation without name conflicts
- `recovery_window_in_days = 0` for sandbox (immediate delete), `7` for prod
- Store all connection details as JSON — Ansible parses them as a unit

### Pattern: SSM Parameter Store for app secrets

```hcl
resource "aws_ssm_parameter" "app_secret" {
  name  = "/${local.name_prefix}/app-secret"
  type  = "SecureString"
  value = random_id.app_secret.hex
  tags  = local.common_tags
}

# User-provided secrets: ignore_changes after first apply
resource "aws_ssm_parameter" "smtp_password" {
  name  = "/${local.name_prefix}/smtp-password"
  type  = "SecureString"
  value = var.smtp_password
  tags  = local.common_tags

  lifecycle {
    ignore_changes = [value]
  }
}
```

- Auto-generated values: no lifecycle block needed
- User-provided values: `ignore_changes = [value]` — user provides on first apply via `-var`, then manages in AWS Console
- Naming convention: `/<project>-<env>/<secret-name>`

---

## Ansible

### Dynamic inventory bridge

The `inventory.py` script reads `tofu output -json` and constructs Ansible inventory dynamically. This is the glue between the two phases.

```python
#!/usr/bin/env python3
import json, subprocess, sys, os

def get_tofu_output():
    tofu_dir = os.path.join(os.path.dirname(__file__), '..', 'opentofu')
    result = subprocess.run(
        ['tofu', 'output', '-json'],
        cwd=tofu_dir, capture_output=True, text=True, check=True
    )
    return json.loads(result.stdout)

# Build inventory from outputs — see template for full example
```

**Key design decisions:**
- Script lives in `ansible/` dir, reads from sibling `opentofu/` dir
- All tofu outputs become host variables in Ansible
- Group name matches the app name (e.g., `lightdash_server`)

### Secret fetching pattern (AWS CLI)

Use `shell` + AWS CLI for secret fetching. This avoids boto3/Python interpreter dependency issues that commonly arise with `community.aws` modules.

```yaml
# 1. Fetch from Secrets Manager (JSON bundle)
- name: Fetch RDS credentials from Secrets Manager
  shell: >
    aws secretsmanager get-secret-value
    --secret-id "{{ rds_secret_arn }}"
    --region "{{ aws_region }}"
    --profile "{{ aws_profile }}"
    --query SecretString --output text
  register: rds_secret_raw
  delegate_to: localhost
  become: false
  no_log: true
  changed_when: false

# 2. Fetch from SSM (individual values)
- name: Fetch app secret from SSM
  shell: >
    aws ssm get-parameter
    --name "{{ ssm_prefix }}/app-secret"
    --region "{{ aws_region }}"
    --profile "{{ aws_profile }}"
    --with-decryption
    --query Parameter.Value --output text
  register: ssm_app_secret
  delegate_to: localhost
  become: false
  no_log: true
  changed_when: false

# 3. Set facts from fetched secrets
- name: Set secret facts
  set_fact:
    db_user: "{{ (rds_secret_raw.stdout | from_json).username }}"
    db_password: "{{ (rds_secret_raw.stdout | from_json).password }}"
    app_secret: "{{ ssm_app_secret.stdout }}"
  no_log: true
```

**Critical rules:**
- `delegate_to: localhost` — AWS API calls run from the control machine, not the target host
- `become: false` — don't run AWS CLI as root on localhost
- `no_log: true` — on EVERY task that touches secrets (fetch, parse, set_fact, template)
- `changed_when: false` — secret reads are not changes
- `--profile "{{ aws_profile }}"` — passes the AWS profile through

### ansible.cfg

```ini
[defaults]
inventory = inventory.py
host_key_checking = False
private_key_file = ~/.ssh/gemma-infra.pem
remote_user = ubuntu
timeout = 30
stdout_callback = yaml
gathering = smart

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes
control_path = ~/.ssh/ansible-%%r@%%h:%%p
```

- `stdout_callback = yaml` — readable output instead of JSON blobs
- `gathering = smart` — cache facts, don't re-gather on every play
- `IdentitiesOnly=yes` — prevents SSH agent from offering wrong keys
- `UserKnownHostsFile=/dev/null` — VMs get replaced frequently

### Role structure

```
roles/<app-name>/
  defaults/main.yml      # Version pins, paths, non-secret config
  tasks/main.yml         # Cloud-init wait, secret fetch, deploy
  handlers/main.yml      # Container restart handlers
  templates/
    docker-compose.yml.j2
    Caddyfile.j2
    .env.j2
```

- Defaults should reference dynamic inventory variables: `"{{ rds_host }}"`, `"{{ s3_bucket }}"`
- Tasks follow the pattern: wait for cloud-init -> fetch secrets -> create dirs -> template files -> pull images -> start services -> health check
- Handlers restart individual containers, not the whole stack

### Handler pattern

```yaml
- name: restart app
  command: docker compose up -d --force-recreate <app-service>
  args:
    chdir: "{{ deploy_dir }}"

- name: restart caddy
  command: docker compose up -d --force-recreate caddy
  args:
    chdir: "{{ deploy_dir }}"
```

- Restart individual services, not `docker compose restart` (which doesn't pick up image changes)
- Use `--force-recreate` to ensure env/config changes take effect

### Dependencies

No Ansible Galaxy collections are required — secret fetching uses the AWS CLI directly (installed on the control machine), avoiding boto3/Python interpreter compatibility issues. This is the proven pattern from our production deployments.

---

## Docker Compose

### Image pinning

```yaml
services:
  app:
    image: myapp/myapp:1.2.3    # Pin exact version — NEVER use :latest
```

### Resource limits

```yaml
services:
  app:
    deploy:
      resources:
        limits:
          memory: 2G
```

- Set memory limits on every service — prevents OOM from taking down the host
- Size based on the instance type's available memory

### Health checks

```yaml
services:
  app:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
```

### Caddy reverse proxy pattern

```yaml
services:
  caddy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy-data:/data
      - caddy-config:/config
    depends_on:
      app:
        condition: service_healthy
    networks:
      - app
```

- Caddy waits for app health check to pass via `depends_on: condition: service_healthy`
- VPN-only: plain HTTP on `:80` (no TLS needed behind VPN)
- Public: use domain name in Caddyfile — Caddy auto-provisions TLS via ACME
- Only Caddy exposes ports 80/443 — app services use `expose` (internal only)
- Bridge network for inter-container communication

### .env file pattern

```
# Database (from Secrets Manager)
DATABASE_URL="postgresql://{{ db_user }}:{{ db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }}"

# App secret (from SSM)
SECRET_KEY="{{ app_secret }}"
```

- Quote all values — prevents shell interpretation of special characters in passwords
- Template renders on the remote host with `mode: 0600` for security
- Deploy task uses `no_log: true`

---

## Project Structure

### Component-grouped layout

```
<app-name>/
  opentofu/           # Phase 1: AWS infrastructure
    main.tf
    variables.tf
    outputs.tf
    backends/
      sandbox.hcl
      prod.hcl
    environments/
      sandbox.tfvars
      prod.tfvars
    .gitignore
  ansible/            # Phase 2: Application deployment
    ansible.cfg
    inventory.py
    deploy.yaml
    roles/<app-name>/
      defaults/main.yml
      tasks/main.yml
      handlers/main.yml
      templates/
        docker-compose.yml.j2
        Caddyfile.j2
        .env.j2
```

- OpenTofu and Ansible are siblings under the component directory
- NOT tool-grouped (no top-level `terraform/` and `ansible/` dirs)
- Each component has its own state — deploy independently
- Dynamic inventory connects them via `tofu output -json`

### What to commit

- `.terraform.lock.hcl` — YES (reproducible provider versions)
- `.terraform/` — NO (local provider cache)
- `*.tfstate` — NO (remote state in S3)
- `environments/*.tfvars` — YES (non-secret config)
- `backends/*.hcl` — YES (bucket names, not credentials)
- `inventory.py` — YES (dynamic inventory script)
- `ansible.cfg` — YES (project-level Ansible config)