Remote State & Locking - Terraform

Remote State

In a team environment, local state files are problematic because:

Multiple team members with different state copies
No central source of truth
Risk of concurrent modifications
State file on individual laptops is a security risk

Remote State stores the state file in a centralized, shared location that the entire team accesses.

Why Remote State Matters

Problem: Local State

Developer A            Developer B
  state.tfstate       state.tfstate
  (old)               (new)
   ↓                    ↓
  [Apply]            [Apply]
   Resource destroyed   Resource modified
   CONFLICT!

Solution: Remote State

Developer A            Developer B
        ↓                 ↓
    [Remote State]
    [state.tfstate]
    (source of truth)
   ← Both read/write same state

Remote State Benefits

Benefit	Description
Shared access	All team members see same infrastructure
Locking	Prevents concurrent modifications
Audit trail	History of state changes (with backends like S3)
Backup	State in cloud storage, not on laptop
Security	Sensitive data protected at rest and in transit
CI/CD friendly	Automation can access shared state
Disaster recovery	Restore state if local files lost

Configuring Remote State

AWS S3 Backend:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Azure Backend:

terraform {
  backend "azurerm" {
    resource_group_name  = "tfstate"
    storage_account_name = "tfstate12345"
    container_name       = "tfstate"
    key                  = "prod.tfstate"
  }
}

Google Cloud Backend:

terraform {
  backend "gcs" {
    bucket = "my-tf-state-bucket"
    prefix = "terraform/state"
  }
}

Setting Up S3 Backend

Step 1: Create S3 bucket

aws s3 mb s3://my-terraform-state --region us-east-1

Step 2: Enable versioning

aws s3api put-bucket-versioning \
  --bucket my-terraform-state \
  --versioning-configuration Status=Enabled

Step 3: Enable encryption

aws s3api put-bucket-encryption \
  --bucket my-terraform-state \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      }
    }]
  }'

Step 4: Block public access

aws s3api put-public-access-block \
  --bucket my-terraform-state \
  --public-access-block-configuration \
  "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Step 5: Create DynamoDB table for locks

aws dynamodb create-table \
  --table-name terraform-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

State Locking

When multiple team members run Terraform simultaneously, they can corrupt the state. State Locking ensures only one person can modify infrastructure at a time.

How Locking Works

User A: terraform plan
  ↓
  Lock acquired (stored in DynamoDB)
  
User B: terraform plan
  ↓
  Blocked! Waiting for lock...
  
User A: terraform apply
  ↓
  Infrastructure updated
  ↓
  Lock released

User B: Proceeds with plan
  ↓
  Lock acquired

Locking Configuration

With S3 backend (DynamoDB lock):

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"  # Lock backend
    encrypt        = true
  }
}

Lock Troubleshooting

Force unlock (if locked by failed process):

terraform force-unlock LOCKID

View lock status:

# For AWS S3 backend
aws dynamodb scan \
  --table-name terraform-locks \
  --region us-east-1

Get current lock:

terraform state pull | grep WorkspaceEnlocked

Migration to Remote State

Step 1: Prepare Remote Backend

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
  }
}

Step 2: Initialize Migration

# Save local state first
cp terraform.tfstate terraform.tfstate.backup
 
# Initialize with new backend
terraform init
# Answer "yes" to copy state to remote

Step 3: Verify Migration

# Check remote state
terraform state list
 
# Remote state is now used
terraform plan

Step 4: Delete Local State (Optional)

rm terraform.tfstate
rm terraform.tfstate.backup

Working with Remote State

Reading State

# Download state to inspect
terraform state pull
 
# List resources in remote state
terraform state list
 
# Show specific resource attributes
terraform state show aws_instance.web

Modifying State

# Don't modify state manually! Use Terraform instead
# But if absolutely necessary:
terraform state mv aws_instance.web aws_instance.server
terraform state rm aws_instance.old_server

Backup Strategy

Enable state backup in AWS:

terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "prod/terraform.tfstate"
    
    # Automatic backups by S3 versioning
    # Also created prior states with .backup suffix
  }
}

Manual backup:

# Download current state
terraform state pull > terraform.tfstate.backup
 
# Download specific version from S3
aws s3api get-object \
  --bucket my-terraform-state \
  --key prod/terraform.tfstate \
  terraform.tfstate.v2

Accessing Remote State from Other Stacks

Read outputs from another stack's remote state:

data "terraform_remote_state" "core" {
  backend = "s3"
  config = {
    bucket = "my-terraform-state"
    key    = "core/terraform.tfstate"
    region = "us-east-1"
  }
}
 
resource "aws_instance" "app" {
  subnet_id = data.terraform_remote_state.core.outputs.subnet_id
  # Reference outputs from core stack
}

Best Practices

Practice	Reason
Always use remote state for teams	Prevents state conflicts
Enable encryption	Protect sensitive data (passwords, keys)
Enable backend locking	Prevent concurrent modifications
Backup state regularly	Disaster recovery
Version your bucket	Track state history
Block public access	Security
Use separate backends per environment	Isolation (dev, staging, prod)
Document backend configuration	Team consistency
Restrict backend access	IAM policies for backend buckets
Never commit state files to Git	Keep .gitignore up to date

Security Hardening

# KMS encryption
terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
    dynamodb_table = "terraform-locks"
  }
}

Multi-Environment Setup

terraform/
  dev/
    terraform.tf   # backend "s3" key: dev/
    main.tf
  staging/
    terraform.tf   # backend "s3" key: staging/
    main.tf
  prod/
    terraform.tf   # backend "s3" key: prod/
    main.tf

Each environment has its own state file in S3, preventing accidental modifications to production.