G
GuideDevOps
Lesson 17 of 28

Stateful Inspection & Connection Tracking

Part of the Networking Basics tutorial series.

Stateful inspection is what separates modern firewalls from simple packet filters. It's the ability to understand connections, not just individual packets.

Stateless vs Stateful Firewalls

Stateless Firewall (Packet Filter)

Examines each packet independently:
"Is this packet allowed by my rules?"

Rule: Allow UDP port 53
✓ Allows all UDP packets to port 53
✓ Allows all UDP packets FROM port 53
(Even unsolicited responses!)

Problem: Can't distinguish request from response

Stateful Firewall (Connection-Aware)

Understands the connection lifecycle:
"Is this packet part of an established connection?"

Rule: Allow DNS queries
✓ Allows DNS request from PC to 8.8.8.8:53
✓ Automatically allows response from 8.8.8.8:53 back
✓ Automatically closes when done
✗ Blocks unsolicited packets from 8.8.8.8:53

Smarter: tracks actual conversations

Connection States

Modern firewalls track several states for each connection:

TCP Connection States

StateMeaningExample
NEWConnection just startingSYN packet received
ESTABLISHEDConnection activeACK received, data flowing
RELATEDRelated to existing connectionFTP data connection
INVALIDNot matching any connectionRandom SYN from nowhere
CLOSE_WAITConnection close initiatedFIN received, awaiting FIN back
TIME_WAITClosed, ports reservedCleanup phase
LISTENWaiting for connectionsServer port open

UDP Pseudo-States

Since UDP has no connection handshake:

No Connection: Packet not part of recent flow
NEW: First packet in a flow
ESTABLISHED: Recent packets in same flow

ICMP States

NEW: Echo request (ping)
ESTABLISHED: Echo reply (response)

How Stateful Inspection Works

Example: PC makes DNS query

Step 1: PC initiates DNS query
  Source: 192.168.1.100:54321 TCP
  Dest: 8.8.8.8:53 UDP
  State: NEW
  Firewall: "Check rules... DNS queries allowed"
  Action: ACCEPT
  Firewall adds to conntrack: [192.168.1.100:54321 ← → 8.8.8.8:53]

Step 2: DNS server responds
  Source: 8.8.8.8:53
  Dest: 192.168.1.100:54321
  State: RELATED (part of connection)
  Firewall: "I'm tracking this! Response to outbound query"
  Action: ACCEPT (automatic, no rule needed)

Step 3: Connection timeout
  No packets in 30 seconds
  Firewall: "Connection idle, removing from table"
  Action: DELETE from conntrack

Connection Tracking Table

The firewall maintains a connection tracking table (conntrack):

# View conntrack table (Linux)
cat /proc/net/nf_conntrack
 
# Example output:
# ipv4     2 tcp      6 118 ESTABLISHED src=192.168.1.100 dst=8.8.8.8 sport=54321 dport=53 src=8.8.8.8 dst=192.168.1.100 sport=53 dport=54321 [ASSURED] use=1
 
# Meaning:
# Protocol: TCP
# State: ESTABLISHED
# Timeout: 118 seconds
# Source: 192.168.1.100:54321
# Destination: 8.8.8.8:53
# Reply: 8.8.8.8:53 → 192.168.1.100:54321

Firewall Rule Evaluation with Stateful Inspection

Traditional Stateless Approach:

Rule 1: Allow TCP 80 inbound
Rule 2: Allow TCP 443 inbound
Rule 3: Deny all else

Problem: Response traffic needs separate rules
Internet: "Here's your HTTP response"
Firewall: "I don't have a rule for source port 80..."
Action: DROP (breaks web browsing!)

Stateful Approach:

Rule 1: Allow TCP 80 inbound? NO
Rule 2: Is this ESTABLISHED traffic? YES
Action: ACCEPT (automatic)

Result: Works! Responses auto-allowed

Conntrack Tuning

Conntrack Table Size:

# Check current size
cat /proc/sys/net/netfilter/nf_conntrack_max
 
# Increase if needed (default often too small)
sudo sysctl -w net.netfilter.nf_conntrack_max=1000000

Connection Timeouts:

# Check TCP timeout
cat /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established
# Typical: 432000 seconds (5 days)
 
# Check UDP timeout
cat /proc/sys/net/netfilter/nf_conntrack_udp_timeout
# Typical: 30 seconds
 
# Adjust for your environment
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=1800
# Changed to 30 minutes

Stateful Firewall Rules (iptables)

Accept established connections:

# Allow NEW connections on port 22 (SSH)
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j ACCEPT
 
# Allow established/related connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
 
# Drop invalid packets
iptables -A INPUT -m state --state INVALID -j DROP

Modern syntax (nftables):

# Allow established/related
nft add rule inet filter input ct state established,related accept
 
# Allow new connections on SSH
nft add rule inet filter input tcp dport 22 ct state new accept
 
# Drop invalid
nft add rule inet filter input ct state invalid drop

Connection Tracking and NAT

Stateful inspection works with NAT:

Internal PC: 192.168.1.100:54321
NAT Device: 203.0.113.50
External Server: 8.8.8.8:53

PC sends DNS query:
  192.168.1.100:54321 → 8.8.8.8:53

NAT rewrites:
  203.0.113.50:10001 → 8.8.8.8:53

Server responds:
  8.8.8.8:53 → 203.0.113.50:10001

NAT conntrack looks up:
  10001 belongs to 192.168.1.100:54321

NAT rewrites response:
  8.8.8.8:53 → 192.168.1.100:54321

PC receives response: "Got my DNS answer!"

Critical: NAT and stateful inspection depend on connection tracking!

Connection Tracking Limitations

Problem 1: Table Overflow

High connection rate
Table fills up
New connections rejected
→ Gateway becomes bottleneck

Solution:

  • Increase nf_conntrack_max
  • Monitor table usage
  • Tune timeouts

Problem 2: Embedded IP Addresses Some protocols send IP addresses in application data:

FTP: "Connect to 192.168.1.100:12345"
Firewall conntrack doesn't know this is an embedded IP
Firewall needs FTP-specific ALG (Application Layer Gateway)

Problem 3: Asymmetric Routes

Outbound: A → NAT → B (tracked)
Return: B → Router2 → C → A
(Different path, NAT device doesn't see return)

Firewall doesn't track the return
Connection breaks

Stateful Inspection in Kubernetes

Network policies use stateful inspection concepts:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          tier: frontend
    ports:
    - protocol: TCP
      port: 80

How it works:

Pod A (frontend) → Pod B (web) on port 80
→ Conntrack marks: ESTABLISHED
← Pod B responses automatically allowed

Stateful Inspection in Load Balancers

Modern load balancers use stateful inspection:

Client: 203.0.113.100:54321
Load Balancer: Tracks connection
Select backend: 10.0.1.50

Conntrack entry:
203.0.113.100:54321 ← → 10.0.1.50:80
(masquerades as load balancer)

All packets in flow routed to same backend

Monitoring Connection State

Check established connections:

# Linux
ss -tan | grep ESTAB
 
# macOS
netstat -an | grep ESTABLISHED

Monitor conntrack live:

# Watch conntrack statistics
watch -n 1 'cat /proc/net/nf_conntrack_stat'
 
# Count by state
cat /proc/net/nf_conntrack | awk '{print $3}' | sort | uniq -c
 
# Typical output:
# 1000 [ESTABLISHED]
# 50 [TIME_WAIT]
# 20 [RELATED]

Best Practices

✓ Always enable stateful inspection on firewalls ✓ Monitor conntrack table size and hit rate ✓ Tune timeouts appropriately for your workload ✓ Use ESTABLISHED,RELATED rule to avoid explosion of rules ✓ Drop INVALID packets to prevent attacks ✓ Consider splitting conntrack load on high-traffic systems ✓ Document any protocol-specific ALG requirements ✓ Test NAT + stateful inspection combinations thoroughly

Key Concepts

  • Stateless firewall checks rules on every packet
  • Stateful firewall understands connection lifecycle
  • Connection states: NEW, ESTABLISHED, RELATED, INVALID
  • Conntrack table tracks all active connections
  • Automatic return traffic allowed without extra rules
  • NAT + stateful required to work together correctly
  • Timeouts clean up stale connections
  • Efficiency: Stateful rules much simpler than stateless