Terraform Best Practices: Infrastructure That Doesn't Break at 3am

Infrastructure as Code (IaC) is supposed to turn cloud provisioning into a deterministic, version-controlled process. Yet without disciplined module boundaries and state management, teams frequently encounter corrupted state files, accidental destructive updates, and panicked midnight outages.

1. Remote State Backend & Concurrency Locking

Never store Terraform state files in Git. Aside from exposing sensitive credentials and database connection strings in plain text, multiple engineers running terraform apply simultaneously will corrupt your state file.

On AWS, configure an S3 bucket with server-side KMS encryption and versioning, paired with DynamoDB for state locking:

terraform {
  required_version = ">= 1.8.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.40"
    }
  }

  backend "s3" {
    bucket         = "arvaan-corp-terraform-state-prod"
    key            = "networking/vpc/terraform.tfstate"
    region         = "ap-south-1"
    encrypt        = true
    dynamodb_table = "terraform-state-locks"
  }
}

2. Reduce Blast Radius: The Monolithic State File Anti-Pattern

Putting your VPC, RDS databases, EKS clusters, and IAM roles inside a single main.tf state file is an operational disaster waiting to happen. An inadvertent parameter change in a security group could trigger a recreational destruction of your production database.

Decompose state files by lifecycle and volatility:

• Foundation (Slow-moving): VPC, subnets, routing tables, NAT Gateways (changed once a year).
• Data Tier (High-criticality): RDS, DynamoDB, S3 buckets with termination protection enabled.
• Compute Tier (Frequent changes): Kubernetes nodes, ECS task definitions, autoscaling groups.

3. Automated CI/CD Plan Reviews (Atlantis / Spacelift)

Humans should never run terraform apply from their laptops. In mature engineering organizations, Terraform is executed strictly via CI pipelines:

1. A developer submits a pull request updating a Terraform module.
2. CI automatically runs tflint, checkov (security policy auditing), and terraform plan.
3. The plan output is commented directly onto the PR for peer review.
4. Upon approval and merge, the pipeline applies the plan against production.