# IaC Security Checklist: OpenTofu + Ansible Reference patterns drawn from: - `saxonia-data-infrastructure/server/opentofu/main.tf` (GOOD: S3 backend with locking, SSH-restricted firewall, SSH key auth, pinned provider) - `saxonia-data-infrastructure/server/ansible/roles/airflow/tasks/main.yml` (GOOD: 1Password secrets, no_log, 0600 .env files, deploy key, --target build) - `saxonia-data-infrastructure/server/ansible/roles/airflow/templates/docker-compose.yml.j2` (GOOD: port bound to 127.0.0.1) - Internal security report: `research/2026-03-17-server-security-data-tools/report/index.html` --- ## OpenTofu / Terraform --- ## Check: State backend uses remote storage with locking enabled - **Priority**: Important - **What to look for**: The `terraform` block's `backend` configuration — check whether it uses a remote backend (`s3`, `gcs`, `azurerm`, `remote`) or defaults to local state. For S3-compatible backends using OpenTofu >= 1.10, check for `use_lockfile = true`. For AWS S3 + DynamoDB, check for a `dynamodb_table` lock table. - **Pass condition**: Remote backend configured with state locking. No `terraform.tfstate` file committed to Git. - **Fail example**: ```hcl # No backend block — state stored locally in terraform.tfstate terraform { required_version = ">= 1.10" } ``` - **Remediation**: ```hcl # Before (insecure — local state, no locking) terraform { required_version = ">= 1.10" } # After (secure — S3-compatible remote backend with native locking) terraform { required_version = ">= 1.10" backend "s3" { bucket = "my-tfstate" key = "server/terraform.tfstate" region = "main" endpoints = { s3 = "https://nbg1.your-objectstorage.com" } use_lockfile = true # OpenTofu >= 1.10 native S3 locking (no DynamoDB needed) # ... provider-specific skip params } } ``` Also add `terraform.tfstate` and `*.tfstate.backup` to `.gitignore`. Never commit state files — they contain all resource attributes, including secrets. - **Source**: OpenTofu state management docs; saxonia `main.tf` lines 11–33 --- ## Check: Firewall restricts SSH to specific CIDRs, not 0.0.0.0/0 - **Priority**: Critical - **What to look for**: In the firewall resource (e.g., `hcloud_firewall`, `aws_security_group`, `google_compute_firewall`), find the rule for port 22. Check whether `source_ips` / `cidr_blocks` / `source_ranges` includes `0.0.0.0/0` or `::/0`. - **Pass condition**: Port 22 ingress is limited to a specific set of known IP ranges (VPN, office, bastion). `0.0.0.0/0` must not appear for SSH. - **Fail example**: ```hcl rule { direction = "in" protocol = "tcp" port = "22" source_ips = ["0.0.0.0/0", "::/0"] # CRITICAL: SSH open to the entire internet } ``` - **Remediation**: ```hcl # Before (insecure) rule { direction = "in" protocol = "tcp" port = "22" source_ips = ["0.0.0.0/0", "::/0"] } # After (secure — restrict to VPN/office CIDRs via variable) variable "allowed_ingress_cidrs" { type = list(string) description = "CIDRs allowed to reach SSH (VPN, office, bastion)" } rule { direction = "in" protocol = "tcp" port = "22" source_ips = var.allowed_ingress_cidrs } ``` If SSH must be publicly reachable, consider a bastion host or Tailscale/WireGuard VPN instead. The security report recommends removing SSH from the public firewall entirely when using a mesh VPN. - **Source**: saxonia `main.tf` lines 68–74; security report §03 §04 --- ## Check: cloud-init does not grant NOPASSWD:ALL sudo - **Priority**: Important - **What to look for**: In the `user_data` / cloud-init block (inline HCL heredoc or a rendered template file), search for `NOPASSWD` in the `sudo:` field of any user entry. Also check rendered Ansible templates that write sudoers files. - **Pass condition**: Either no sudo granted in cloud-init, or sudo is scoped to specific commands (e.g., `NOPASSWD: /usr/bin/docker`). `NOPASSWD: ALL` is a fail. - **Fail example**: ```yaml # cloud-init user_data (insecure) users: - name: ubuntu sudo: "ALL=(ALL) NOPASSWD:ALL" # Any process running as ubuntu = root ``` - **Remediation**: ```yaml # Before (insecure) users: - name: ubuntu sudo: "ALL=(ALL) NOPASSWD:ALL" # After (secure — scope to specific binaries only, or omit sudo entirely) users: - name: ubuntu groups: [docker] # Use docker group instead of sudo for Docker access # No sudo line — provision sudo rules post-boot via Ansible if needed ``` Add the user to the `docker` group for Docker access instead of granting blanket sudo. If root-level access is required for specific operations, scope it: `NOPASSWD: /usr/bin/systemctl restart myapp`. - **Source**: saxonia `main.tf` line 149; security report §04: "Never use NOPASSWD: ALL — scope to specific binaries" --- ## Check: cloud-init configures correct user groups - **Priority**: Important - **What to look for**: In the `users:` block of cloud-init, check the `groups:` list for the deploy user. Verify `docker` is present if Docker is used. Verify `sudo` is absent or intentional. - **Pass condition**: Deploy user is in `docker` group (required to run docker commands without sudo). Groups list matches the actual services running on the server. - **Fail example**: ```yaml # Missing docker group — all docker commands will fail or require sudo users: - name: ubuntu groups: [sudo] shell: /bin/bash ``` - **Remediation**: ```yaml # Before (missing docker group) users: - name: ubuntu groups: [sudo] # After (correct groups for an Airflow/Docker server) users: - name: ubuntu groups: [docker] # Required for docker compose commands without sudo shell: /bin/bash ssh_authorized_keys: - ssh-ed25519 AAAA... deployer@example.com ``` Note: adding a user to the `docker` group grants effective root access to the host via volume mounts. This is an accepted trade-off for deploy users; avoid adding application service accounts to the docker group. - **Source**: saxonia `main.tf` line 148; Docker security best practices --- ## Check: OS image is pinned to a specific version, not a rolling alias - **Priority**: Recommended - **What to look for**: In the server/VM resource, check the `image` field (Hetzner: `image`, AWS: `ami`, GCP: `source_image`). Rolling aliases like `ubuntu-latest` or `debian-12` resolve to different images over time. - **Pass condition**: Image is pinned to a specific version string (e.g., `ubuntu-22.04`) or, better, a specific AMI/image ID. Version is not `latest` or an unversioned alias. - **Fail example**: ```hcl resource "hcloud_server" "main" { image = "ubuntu-latest" # Resolves to a different image after each Ubuntu release } ``` - **Remediation**: ```hcl # Before (unversioned) image = "ubuntu-latest" # After (pinned version — change intentionally during upgrade windows) image = "ubuntu-22.04" ``` For maximum reproducibility, look up the specific image ID and pin by ID. When upgrading the OS, update the image reference in a deliberate PR. - **Source**: saxonia `main.tf` line 122 (uses `ubuntu-22.04`); infrastructure immutability best practices --- ## Check: Private network used for inter-service communication - **Priority**: Recommended - **What to look for**: If multiple servers are deployed (e.g., separate app and database servers), check whether they communicate over a private network resource (`hcloud_network`, `aws_vpc`, etc.) or over public IPs. Also check whether the firewall/security group blocks database ports from the public internet. - **Pass condition**: Database and internal services communicate over a private network. Public IP is only used for services that require internet access (web, load balancer). Database port (5432) is not open in the public firewall. - **Fail example**: ```hcl # Database connecting over public IP — no private network configured resource "hcloud_server" "db" { public_net { ipv4_enabled = true } # No network attachment } ``` - **Remediation**: ```hcl # Before (public-only) resource "hcloud_server" "db" { public_net { ipv4_enabled = true } } # After (private network for inter-service traffic) resource "hcloud_network" "internal" { name = "prod-internal" ip_range = "10.0.0.0/16" } resource "hcloud_network_subnet" "internal" { network_id = hcloud_network.internal.id type = "private" network_zone = "eu-central" ip_range = "10.0.1.0/24" } resource "hcloud_server_network" "db" { server_id = hcloud_server.db.id network_id = hcloud_network.internal.id ip = "10.0.1.10" } ``` Note from the security report: Hetzner Cloud Firewalls do **not** apply to private network interfaces. Add host-level UFW rules on private interfaces as defense-in-depth. - **Source**: security report §03: "Hetzner private network gap" --- ## Check: Server backups are enabled - **Priority**: Important - **What to look for**: In the server resource, check for a `backups` attribute. For block volumes, check for snapshot schedules. Also look for any external backup solution (restic, Velero) configured via Ansible. - **Pass condition**: `backups = true` on the server resource, OR an automated external backup solution (e.g., restic to S3) is documented and configured. Restore procedures should be tested. - **Fail example**: ```hcl resource "hcloud_server" "main" { name = "prod" backups = false # No automated backups — data loss risk } ``` - **Remediation**: ```hcl # Before (no backups) resource "hcloud_server" "main" { backups = false } # After (Hetzner automated backups — 7 daily snapshots, ~20% cost surcharge) variable "enable_server_backups" { type = bool default = true } resource "hcloud_server" "main" { backups = var.enable_server_backups } ``` For database data, also configure restic or pg_dump-based backups to object storage. The security report recommends: test restores monthly and verify dump integrity with `--stdin-from-command`. - **Source**: saxonia `main.tf` line 129 (`backups = var.enable_server_backups`); security report §08 --- ## Check: Provider version constraints are pinned - **Priority**: Recommended - **What to look for**: In the `required_providers` block, check that each provider has a version constraint. Prefer `~>` (pessimistic constraint operator) to allow patch updates while preventing major/minor version jumps. Check for missing version constraints entirely. - **Pass condition**: All providers have a version constraint of the form `~> X.Y` or `>= X.Y, < X+1.0`. No provider uses `>= X` alone (too permissive) or omits version entirely. - **Fail example**: ```hcl required_providers { hcloud = { source = "hetznercloud/hcloud" # No version constraint — will silently upgrade on tofu init } } ``` - **Remediation**: ```hcl # Before (unpinned) required_providers { hcloud = { source = "hetznercloud/hcloud" } } # After (pinned with pessimistic constraint — allows patch updates only) required_providers { hcloud = { source = "hetznercloud/hcloud" version = "~> 1.60" } } ``` After setting constraints, commit the `.terraform.lock.hcl` file to version control. This locks exact provider checksums and prevents supply-chain attacks via provider registry compromise. - **Source**: saxonia `main.tf` lines 4–8; OpenTofu provider locking docs --- ## Ansible --- ## Check: Secrets fetched from 1Password or a vault — never hardcoded - **Priority**: Critical - **What to look for**: Search Ansible playbooks, `vars/`, `group_vars/`, and `host_vars/` for plaintext secrets: passwords, API keys, fernet keys, database connection strings. Check if `op read`, `ansible-vault`, or SOPS is used. Check that `.env.j2` templates reference Ansible variables that come from a vault call — not inline strings. - **Pass condition**: All secrets are fetched at runtime from 1Password CLI (`op read`), Ansible Vault, or SOPS. No plaintext secrets appear in any YAML, Jinja2 template, or inventory file. - **Fail example**: ```yaml # vars/main.yml (CRITICAL: hardcoded secrets) airflow_fernet_key: "wC3DN6MfXP1abcdefghij123=" airflow_pg_password: "supersecret123" ``` - **Remediation**: ```yaml # Before (insecure — hardcoded in vars) airflow_fernet_key: "wC3DN6MfXP1abcdefghij123=" # After (secure — fetch from 1Password at deploy time) - name: Fetch Airflow fernet key from 1Password command: op read "op://Vault Name/Item Name/fernet_key" register: airflow_fernet_key changed_when: false no_log: true become: false delegate_to: localhost ``` For teams not using 1Password, use `ansible-vault encrypt_string` for secrets in vars files, or SOPS for `.env` files encrypted in Git. - **Source**: saxonia `tasks/main.yml` lines 54–93; security report §08 --- ## Check: no_log: true on all tasks that handle secrets - **Priority**: Important - **What to look for**: Find Ansible tasks that call `op read`, register a variable containing a secret, or use `template` / `copy` to write a `.env` file. Verify that `no_log: true` is set on each such task. Also check the task that renders the `.env.j2` template. - **Pass condition**: Every task that fetches, registers, or writes a secret has `no_log: true`. This prevents secret values from appearing in Ansible output, CI logs, or AWX/Tower job history. - **Fail example**: ```yaml # Missing no_log — secret value printed to stdout on every run - name: Fetch Airflow fernet key from 1Password command: op read "op://Vault/Airflow/fernet_key" register: airflow_fernet_key changed_when: false # no no_log: true — output will contain the fernet key ``` - **Remediation**: ```yaml # Before (insecure — logs secret value) - name: Fetch Airflow fernet key from 1Password command: op read "op://Vault/Airflow/fernet_key" register: airflow_fernet_key changed_when: false # After (secure) - name: Fetch Airflow fernet key from 1Password command: op read "op://Vault/Airflow/fernet_key" register: airflow_fernet_key changed_when: false no_log: true become: false delegate_to: localhost ``` Note: `no_log` suppresses all task output. When debugging, use a separate debug task with `no_log: false` that only prints a non-secret indicator (e.g., key length). - **Source**: saxonia `tasks/main.yml` lines 58, 65, 72, 79, 87; Ansible security best practices --- ## Check: .env files have mode 0600, config files mode 0644 - **Priority**: Important - **What to look for**: In Ansible `template` or `copy` tasks that write `.env` files, `.env.vpn` files, or any file containing secrets, check the `mode` attribute. Config files that don't contain secrets (e.g., `docker-compose.yml`) should be 0644. - **Pass condition**: Files containing secrets: `mode: "0600"`. Non-secret config files: `mode: "0644"`. Owner and group should be the deploy user (not root). - **Fail example**: ```yaml # .env file world-readable - name: Render .env template: src: .env.j2 dest: /opt/airflow/.env owner: ubuntu group: ubuntu mode: "0644" # World-readable — any process can read secrets ``` - **Remediation**: ```yaml # Before (insecure — 0644 for secret file) - name: Render .env template: src: .env.j2 dest: /opt/airflow/.env mode: "0644" # After (secure — 0600 for secret file, 0644 for non-secret config) - name: Render .env template: src: .env.j2 dest: /opt/airflow/.env owner: ubuntu group: ubuntu mode: "0600" # Owner read/write only - name: Render docker-compose.yml template: src: docker-compose.yml.j2 dest: /opt/airflow/docker-compose.yml owner: ubuntu group: ubuntu mode: "0644" # Non-secret config — readable by group/world is fine ``` - **Source**: saxonia `tasks/main.yml` lines 96–121 --- ## Check: Dedicated deploy key used for Git clone, not a personal SSH key - **Priority**: Important - **What to look for**: In Ansible tasks that use the `git` module, check the `key_file` parameter. Verify it points to a named deploy key file (e.g., `github_deploy_ed25519`) rather than a generic `id_rsa` or `id_ed25519`. In OpenTofu, check that a separate CI deploy SSH key resource exists (`hcloud_ssh_key.deploy_ci`). - **Pass condition**: A dedicated read-only deploy key is used for Git operations. The key is named to indicate its purpose and scope. Personal SSH keys are not used on servers. - **Fail example**: ```yaml - name: Clone repo git: repo: git@github.com:org/repo.git dest: /opt/app key_file: /home/ubuntu/.ssh/id_ed25519 # Personal key — rotates with person, not role ``` - **Remediation**: ```yaml # Before (personal key) - name: Clone repo git: repo: git@github.com:org/repo.git key_file: /home/ubuntu/.ssh/id_ed25519 # After (dedicated deploy key) - name: Clone or update repo git: repo: "{{ repo_url }}" dest: "{{ repo_dir }}" version: "{{ repo_branch }}" force: true key_file: /home/ubuntu/.ssh/github_deploy_ed25519 # Deploy-specific key accept_hostkey: true become_user: ubuntu ``` Generate a separate Ed25519 key pair per repo. Add the public key as a read-only deploy key in GitHub/GitLab. Store the private key path in Ansible vars. This key can be revoked independently of any team member's personal key. - **Source**: saxonia `tasks/main.yml` line 36; security report §04 --- ## Check: Docker build uses --target to avoid baking secrets into image layers - **Priority**: Important - **What to look for**: Find Ansible tasks (or CI steps) that run `docker build`. Check whether `--target` is used to build a specific multi-stage target. The concern is a Dockerfile with a `dev_build` (no secrets) and `prod_build` (secrets baked in via ARG/ENV) stage — the Ansible task should always build `dev_build`; secrets are injected at runtime via `.env`, not at build time. - **Pass condition**: `docker build` uses `--target dev_build` (or equivalent non-secret stage). Secrets are not passed as `--build-arg` to any task. The production image contains no secret values in its layers. - **Fail example**: ```yaml # Bakes fernet key into image layer — extractable via docker history - name: Build Airflow image command: > docker build --build-arg fernet_key="{{ airflow_fernet_key.stdout }}" -t airflow:latest /opt/airflow ``` - **Remediation**: ```yaml # Before (insecure — secrets in build args become image layer metadata) - name: Build Airflow image command: > docker build --build-arg fernet_key="{{ airflow_fernet_key.stdout }}" -t airflow:latest /opt/airflow # After (secure — build only the secrets-free target; inject secrets at runtime via .env) - name: Build Airflow Docker image (dev_build target) command: > docker build --target dev_build -t {{ airflow_image_name }}:{{ airflow_image_tag }} {{ airflow_repo_dir }} changed_when: true ``` The corresponding Dockerfile should have a `dev_build` stage that installs dependencies only, and a `prod_build` stage that is **not** built via Ansible (or is avoided entirely in favor of runtime `.env` injection). - **Source**: saxonia `tasks/main.yml` lines 44–50; comment: "Build the dev_build target to avoid baking secrets into image layers"