G
GuideDevOps
Lesson 10 of 10

GitOps Best Practices

Part of the GitOps tutorial series.

Setting the Gold Standard for GitOps

Implementing ArgoCD or Flux is only step one. True GitOps is a cultural paradigm shift. If you simply use ArgoCD to deploy massive, unstructured, untested YAML files, you are just performing CIOps with a prettier UI.

Follow these golden rules to unlock the true stability of GitOps.

1. Strictly Decouple App and Config Repositories

Never place your Kubernetes YAML manifests (or Helm charts) in the exact same repository as your application source code.

As discussed in the Workflows tutorial, maintaining a dedicated Infrastructure Git Repository prevents catastrophic CI/CD circular loops and allows you to enforce completely different security and organizational permissions for code developers versus site reliability engineers.

2. Ban kubectl apply in Production

To enforce GitOps, you must break the habit of manual terminal administration.

The Git repository must be the exclusive mechanism for changing the cluster state. To physically enforce this, revoke kubectl apply, create, edit, and delete cluster roles from all human developers the moment GitOps goes live.

Developers should only possess get, list, and watch permissions for reading logs and debugging. If they attempt an imperative change, the Kubernetes RBAC system should firmly deny them.

3. Avoid Vaulting Secrets in Plain Text

Just because "Git is the single source of truth" does not mean you should commit DATABASE_PASSWORD=SuperSecret123! directly into your Git repository.

A GitOps workflow must safely handle secrets. You have two Primary strategies:

  1. Sealed Secrets / Mozilla SOPS: Encrypt the secret locally using strong cryptography, commit the encrypted garbage string to Git, and let a Kubernetes operator natively decrypt it inside the cluster using a master private key.
  2. External Secrets Operator (ESO): Keep the secret completely out of Git. Store it in AWS Secrets Manager or HashiCorp Vault. In your Git repository, commit an ExternalSecret manifest that tells Kubernetes: "Reach out to AWS dynamically at runtime to fetch the database password."

4. Aggressively Enable Auto-Healing (Drift Reversal)

Because "Git is Truth," reality cannot be permitted to disagree with it.

If a rogue script or a chaotic microservice mutates a deployment directly on the cluster, that constitutes Configuration Drift.

You must configure your GitOps operator to be aggressive. In ArgoCD, enable the selfHeal: true flag on all Application definitions. This ensures that any change made outside the Git pipeline is aggressively overwritten and destroyed within minutes. It forces teams to respect the Git flow.

5. Don't Templatize Too Early

When teams migrate to GitOps, they often attempt to convert 100% of their plain Kubernetes YAML directly into highly complex Helm charts or Kustomize overlays on day one.

This over-engineering creates a massive barrier to entry for junior engineers, creating complex bugs where templates render incorrectly and crash the deployment.

Start with raw, plain-text YAML. ArgoCD and Flux read plain YAML flawlessly. Only introduce Kustomize or Helm when you are duplicating the exact same YAML manifest across multiple environments (e.g., Staging vs Production) and strictly need to abstract out differences like replica counts or RAM limits.

6. Treat the Infrastructure Repo as Code

Because the Infrastructure Repository generates your production website, it must be treated with the exact same rigor as a Python or Node.js codebase.

  1. Pull Requests Required: Do not allow direct commits to the main branch. All changes must go through a PR and receive human approval.
  2. Pre-Commit Linting: Enforce tools like kubeval or kube-linter in GitHub Actions on the PR. If someone attempts to deploy a YAML file containing a syntax typo or a missing API version, the CI pipeline should block the PR before ArgoCD ever has the chance to parse it.
  3. Run Dry-Runs: CI checks on the PR should run helm template or kustomize build to render the final, compiled YAML dynamically and echo it back to the PR comments. This allows reviewers to easily see exactly what changes the abstract templates will apply to production.