GitOps + ArgoCD

GitOps + ArgoCD Guidelines

MANDATORY FOR ALL AI AGENTS: Every Kubernetes-bound deployment in this platform MUST be GitOps-driven. The Git repository is the single source of truth for cluster state — no kubectl apply from laptops, no manual Helm installs, no console-edited manifests. ArgoCD reconciles cluster → Git, not the other way. Any drift between cluster and Git is either auto-healed or flagged as a fault. CI builds artifacts (images, charts); GitOps deploys them by writing a new commit, not by calling the cluster API.

Project-agnostic: code samples use placeholder names (platform-payment-service, platform-gitops), domain (example.com), namespace (platform), and registry (ghcr.io/example). Replace with your project’s actual values.

Reference implementation: ArgoCD 3.0.x (3.x is GA since 3.0, May 2025) + Kustomize 5.4.x + Sealed-Secrets 0.27.x + Argo Rollouts 1.8.x. Helm 3.16+ for third-party charts only. ApplicationSet controller bundled with ArgoCD.

Companion guides:

• api-gateway.md — the gateway is deployed via ArgoCD too; its config repo is a GitOps subject.
• security.md — secrets management, RBAC, SSO/OIDC for the ArgoCD UI/API.
• obs-tracing.md — ArgoCD emits Prometheus metrics + OTEL spans for sync events.
• adr.md — GitOps tool choice, repo topology, promotion strategy, secrets strategy are ADR-worthy.
• arch-doc-spec.md §Deployment View — GitOps controllers are first-class deployment topology actors.

Companion commands:

• /arch-threat — threat-model the GitOps control plane (ArgoCD is an attack surface).
• /arch-deps — audit chart + image dependencies referenced in the GitOps repo.

Table of Contents

1. Principles
2. CI/CD Boundary
3. Repository Topology
4. ArgoCD Architecture
5. Application CRD
6. Project CRD + RBAC
7. Sync Policies
8. Sync Waves + Hooks
9. App-of-Apps Pattern
10. ApplicationSet Pattern
11. Multi-Environment with Kustomize
12. Helm in GitOps
13. Promotion Strategy
14. Secrets Management
15. Image Update Automation
16. Progressive Delivery (Argo Rollouts)
17. Drift Detection + Notifications
18. Health Checks + Resource Customizations
19. Multi-Cluster
20. Disaster Recovery + Bootstrap
21. Observability
22. Anti-Patterns
23. Pre-Merge Checklist

1. Principles

GitOps is defined by four properties (OpenGitOps v1.0.0). Every rule below derives from one of them.

1. Declarative. All desired state is described declaratively (YAML manifests, Helm values, Kustomize overlays). No imperative scripts (kubectl create, helm install) in the deploy path.
2. Versioned + immutable. Git is the source of truth; every change is a commit with author + diff + signature. Tags and SHAs are immutable; branches are pointers.
3. Pulled automatically. A controller (ArgoCD) running in the cluster pulls state from Git on a loop. The cluster reconciles toward Git — not the other way around.
4. Continuously reconciled. Drift between Git and cluster is detected and either auto-healed or alerted. Manual kubectl edit is reverted unless explicitly excluded.

Derived rules:

• No human writes to the cluster API for desired state. All changes go through Git PRs.
• CI never calls kubectl apply. CI writes a commit to the GitOps repo; ArgoCD applies it.
• Cluster credentials never leave the cluster. Developer laptops have read-only kubeconfig at best. Write access via ArgoCD UI/API only.
• Rollback = git revert. Not helm rollback, not kubectl rollout undo — those create drift.
• Secrets are encrypted at rest in Git. Plaintext secrets in a GitOps repo is a security incident.
• Every environment maps to one or more Git refs. No “fix it live, sync to Git later.” If it’s not in Git, it doesn’t exist.

2. CI/CD Boundary

The hardest line to hold. CI and CD have separate concerns and separate repos.

Hard rule: CI MUST NOT have cluster credentials. The only artifact CI produces that the cluster sees is a Git commit in the GitOps repo.

2.1 The “image promotion” handoff

CI produces a tagged image; it must record that tag somewhere ArgoCD reads. Two valid patterns:

A. CI writes the GitOps repo directly (push pattern):

1
# .github/workflows/release.yml (excerpt)
2
- name: Update GitOps manifest
3
  run: |
4
    git clone https://${{ secrets.GITOPS_TOKEN }}@github.com/example/platform-gitops
5
    cd platform-gitops
6
    yq -i '.images[0].newTag = "${{ steps.meta.outputs.version }}"' \
7
       envs/dev/payment-service/kustomization.yaml
8
    git commit -am "chore(dev): bump payment-service to ${{ steps.meta.outputs.version }}"
9
    git push

B. ArgoCD Image Updater polls the registry (pull pattern, see §15). CI does nothing extra; the updater writes the commit.

Pattern A is simpler and explicit (every deploy = a clear CI commit). Pattern B keeps CI dumber but adds a new controller to operate. Choose one per env tier — pattern A is fine for dev, B is convenient for many microservices.

3. Repository Topology

Two valid layouts. Pick one in an ADR and stick to it.

3.1 Split: app repos + one GitOps repo (recommended)

1
example/platform-payment-service     ← app code, Dockerfile, Helm chart source (if any)
2
example/platform-billing-service     ← app code, ...
3
example/platform-gitops              ← all Kubernetes manifests, all environments

Why split. App developers stay in app repos; platform team owns deploy topology. PRs to platform-gitops are reviewed by SRE. CI from any app repo writes its tag bump as a PR to platform-gitops.

Layout of platform-gitops:

1
platform-gitops/
2
├── README.md
3
├── bootstrap/                          # ArgoCD itself + cluster-wide controllers
4
│   ├── argocd/
5
│   │   ├── kustomization.yaml
6
│   │   └── values.yaml
7
│   ├── sealed-secrets/
8
│   ├── argo-rollouts/
9
│   └── ingress-nginx/
10
├── apps/                               # ApplicationSet + Application manifests
11
│   ├── root-app.yaml                   # app-of-apps entry (§9)
12
│   ├── platform-services.yaml          # ApplicationSet (§10)
13
│   └── infra-controllers.yaml
14
├── base/                               # Kustomize bases (env-neutral)
15
│   ├── payment-service/
16
│   │   ├── deployment.yaml
17
│   │   ├── service.yaml
18
│   │   ├── kustomization.yaml
19
│   │   └── ...
20
│   └── billing-service/
21
├── envs/                               # per-env overlays
22
│   ├── dev/
23
│   │   ├── payment-service/
24
│   │   │   └── kustomization.yaml
25
│   │   └── ...
26
│   ├── staging/
27
│   └── prod/
28
└── projects/                           # AppProject CRs (RBAC) (§6)
29
    ├── platform-dev.yaml
30
    ├── platform-staging.yaml
31
    └── platform-prod.yaml

3.2 Mono: app + manifests in same repo

Only viable for very small platforms (≤ 3 services). Risks: app PRs collide with deploy PRs; CI must guard against feedback loops (a tag bump must NOT trigger another CI run).

3.3 Env branches vs env folders

Env folders (recommended). One repo, one branch (main), folders per env: envs/dev/, envs/staging/, envs/prod/. Promotion = a PR that copies a manifest fragment from envs/dev/ to envs/staging/. Easy to diff, easy to audit.

Env branches (anti-pattern in most cases). One branch per env. Tempting but: branches drift, merge conflicts are constant, git log per env is fragmented. Use only if your platform has hard separation rules (e.g., different teams own different envs and you want branch protection rules per env).

4. ArgoCD Architecture

ArgoCD runs in the cluster it manages (and optionally manages remote clusters too — see §19).

Components:

Install ArgoCD via the official Helm chart or upstream Kustomize manifest. Pin a specific version. Track the upgrade per CVE in bootstrap/argocd/.

4.1 Bootstrap chicken-and-egg

ArgoCD manages itself, but cannot install itself. Bootstrap order:

1. Cluster created (Terraform / eksctl / gke / kubeadm).
2. kubectl apply the ArgoCD manifest once, manually from a runbook (this is the only sanctioned manual apply).
3. Apply the root app (apps/root-app.yaml) — an Application pointing at bootstrap/argocd/ itself.
4. From now on, ArgoCD upgrades itself via Git commits. Manual step never repeats unless the cluster is destroyed.

The bootstrap script (scripts/bootstrap.sh) lives in platform-gitops and is documented in the README.

5. Application CRD

The unit of deployment in ArgoCD.

1
apiVersion: argoproj.io/v1alpha1
2
kind: Application
3
metadata:
4
  name: payment-service-dev
5
  namespace: argocd
6
  finalizers:
7
    - resources-finalizer.argocd.argoproj.io   # delete child resources on app delete
8
spec:
9
  project: platform-dev
10
  source:
11
    repoURL: https://github.com/example/platform-gitops
12
    targetRevision: main
13
    path: envs/dev/payment-service
14
  destination:
15
    server: https://kubernetes.default.svc
16
    namespace: platform
17
  syncPolicy:
18
    automated:
19
      prune: true
20
      selfHeal: true
21
      allowEmpty: false
22
    syncOptions:
23
      - CreateNamespace=true
24
      - PrunePropagationPolicy=foreground
25
      - ApplyOutOfSyncOnly=true
26
    retry:
27
      limit: 5
28
      backoff:
29
        duration: 5s
30
        factor: 2
31
        maxDuration: 3m
32
  revisionHistoryLimit: 10

Critical fields:

• project: AppProject reference for RBAC (§6). Never default in production.
• targetRevision: branch (mutable, e.g., main) for dev; tag or commit SHA (immutable) for prod. Mutable refs in prod = audit failure.
• syncPolicy.automated.prune: true: required for true GitOps. Without prune, deleted manifests stay alive in the cluster.
• syncPolicy.automated.selfHeal: true: reverts manual kubectl edit drift. Disable only for resources legitimately mutated outside Git (rare).
• finalizers: ensures children are GC’d on app delete. Forgetting this leaves orphans.

5.1 Multi-source applications

ArgoCD 2.6+ supports multiple sources (one Helm chart + one values repo). Useful when you deploy a third-party chart you don’t fork:

1
spec:
2
  sources:
3
    - repoURL: https://prometheus-community.github.io/helm-charts
4
      chart: kube-prometheus-stack
5
      targetRevision: 65.5.0
6
      helm:
7
        valueFiles:
8
          - $values/envs/prod/kube-prometheus-stack/values.yaml
9
    - repoURL: https://github.com/example/platform-gitops
10
      targetRevision: main
11
      ref: values

Pin chart targetRevision to an exact version. Never * or latest.

6. Project CRD + RBAC

AppProjects scope what an Application can deploy and who can manage it. Defense-in-depth against a compromised app developer or a mis-routed PR.

1
apiVersion: argoproj.io/v1alpha1
2
kind: AppProject
3
metadata:
4
  name: platform-prod
5
  namespace: argocd
6
spec:
7
  description: Production services
8
  sourceRepos:
9
    - https://github.com/example/platform-gitops
10
  destinations:
11
    - server: https://kubernetes.default.svc
12
      namespace: platform
13
    - server: https://kubernetes.default.svc
14
      namespace: platform-jobs
15
  clusterResourceWhitelist:
16
    - group: ''
17
      kind: Namespace
18
  namespaceResourceWhitelist:
19
    - group: '*'
20
      kind: '*'
21
  namespaceResourceBlacklist:
22
    - group: ''
23
      kind: ResourceQuota
24
    - group: ''
25
      kind: LimitRange
26
  roles:
27
    - name: developer
28
      description: Read + manual sync of dev/staging only — NOT prod
29
      policies:
30
        - p, proj:platform-prod:developer, applications, get, platform-prod/*, allow
31
        - p, proj:platform-prod:developer, applications, sync, platform-prod/*, deny
32
      groups:
33
        - platform-dev-engineers
34
    - name: sre
35
      description: Full control
36
      policies:
37
        - p, proj:platform-prod:sre, applications, *, platform-prod/*, allow
38
      groups:
39
        - platform-sre
40
  signatureKeys:
41
    - keyID: ABCDEF0123456789
42
  orphanedResources:
43
    warn: true

• sourceRepos restricts which Git repos this project may reference. Prevents an Application in platform-prod from secretly pulling from a fork.
• destinations restricts which clusters + namespaces. Combined with clusterResourceWhitelist, prevents privilege escalation via cluster-scoped resources.
• signatureKeys requires commits in the source repo to be GPG-signed by an allow-listed key. Strongly recommended for prod.
• orphanedResources.warn: true flags resources in the target namespace not managed by any Application — surfaces drift.

SSO + group mapping is configured in argocd-rbac-cm ConfigMap. Map IdP groups (Okta/Keycloak/AAD) to project roles. Never use the built-in admin account beyond bootstrap.

7. Sync Policies

Three modes:

7.1 PruneLast + IgnoreDifferences

Some resources have fields the cluster mutates (e.g., Deployment.spec.replicas when HPA is active). Without telling ArgoCD, every reconcile will fight the HPA:

1
spec:
2
  ignoreDifferences:
3
    - group: apps
4
      kind: Deployment
5
      jsonPointers:
6
        - /spec/replicas

Use sparingly. Each ignoreDifferences entry is a known divergence between Git and cluster — document why.

7.2 Sync options that matter

8. Sync Waves + Hooks

Order matters when one manifest depends on another (CRD before CR, namespace before deployment, secret before pod).

8.1 Sync waves

Annotate resources with argocd.argoproj.io/sync-wave. Lower waves apply first. Default wave is 0. Negative is allowed.

1
metadata:
2
  annotations:
3
    argocd.argoproj.io/sync-wave: "-1"   # CRDs first

Typical pattern:

• -2 Namespaces, ResourceQuotas
• -1 CRDs, ClusterRoles
• 0 ConfigMaps, Secrets, ServiceAccounts (default)
• 1 Services, Deployments, StatefulSets
• 2 Ingress, HPAs
• 3 Smoke tests (Jobs as PostSync hooks)

8.2 Hooks

Run a Job at a specific phase. Hook = a manifest annotated with argocd.argoproj.io/hook:

Delete policies for hooks: HookSucceeded, HookFailed, BeforeHookCreation. Default leaves the Job around; use HookSucceeded to clean up after success.

1
apiVersion: batch/v1
2
kind: Job
3
metadata:
4
  name: payment-db-migrate
5
  annotations:
6
    argocd.argoproj.io/hook: PreSync
7
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
8
spec:
9
  template:
10
    spec:
11
      restartPolicy: Never
12
      containers:
13
        - name: liquibase
14
          image: ghcr.io/example/platform-payment-service:1.4.2
15
          args: ["liquibase", "update"]

Liquibase + GitOps pairing. Liquibase runs as a PreSync hook (or as an initContainer — debatable). The image of the hook MUST match the image of the deployment about to roll out. Use the same targetRevision for both. See liquibase.md for migration discipline.

9. App-of-Apps Pattern

One root Application points at a folder of other Application manifests. Bootstrap installs the root; the root brings up everything else.

1
apps/
2
├── root-app.yaml              # ← the only app you apply manually
3
└── children/
4
    ├── platform-services.yaml # ApplicationSet
5
    ├── infra-controllers.yaml
6
    └── monitoring.yaml

1
apiVersion: argoproj.io/v1alpha1
2
kind: Application
3
metadata:
4
  name: root
5
  namespace: argocd
6
spec:
7
  project: platform-bootstrap
8
  source:
9
    repoURL: https://github.com/example/platform-gitops
10
    targetRevision: main
11
    path: apps/children
12
    directory:
13
      recurse: true
14
  destination:
15
    server: https://kubernetes.default.svc
16
    namespace: argocd
17
  syncPolicy:
18
    automated:
19
      prune: true
20
      selfHeal: true

Trade-off vs ApplicationSet: app-of-apps is simpler to reason about and easier for cross-team review. ApplicationSet is more powerful for fan-out (one template → N apps). Use both: app-of-apps at the root, ApplicationSets for service families.

10. ApplicationSet Pattern

Templated, generator-driven Application creation. Eliminates copy-paste of N near-identical Application manifests.

10.1 Generators

10.2 Per-env per-service fan-out

1
apiVersion: argoproj.io/v1alpha1
2
kind: ApplicationSet
3
metadata:
4
  name: platform-services
5
  namespace: argocd
6
spec:
7
  generators:
8
    - matrix:
9
        generators:
10
          - list:
11
              elements:
12
                - env: dev
13
                  cluster: https://kubernetes.default.svc
14
                  namespace: platform
15
                - env: staging
16
                  cluster: https://kubernetes.default.svc
17
                  namespace: platform-staging
18
                - env: prod
19
                  cluster: https://prod-cluster.example.com
20
                  namespace: platform
21
          - git:
22
              repoURL: https://github.com/example/platform-gitops
23
              revision: main
24
              directories:
25
                - path: base/*
26
  template:
27
    metadata:
28
      name: '{{path.basename}}-{{env}}'
29
    spec:
30
      project: 'platform-{{env}}'
31
      source:
32
        repoURL: https://github.com/example/platform-gitops
33
        targetRevision: main
34
        path: 'envs/{{env}}/{{path.basename}}'
35
      destination:
36
        server: '{{cluster}}'
37
        namespace: '{{namespace}}'
38
      syncPolicy:
39
        automated:
40
          prune: true
41
          selfHeal: true

Adding a new service = drop a folder in base/ and envs/<env>/; the ApplicationSet generates 3 Application CRs automatically.

10.3 Preview environments via PR generator

1
generators:
2
  - pullRequest:
3
      github:
4
        owner: example
5
        repo: platform-payment-service
6
        labels: ["preview"]
7
      requeueAfterSeconds: 60
8
template:
9
  metadata:
10
    name: 'preview-payment-{{number}}'
11
  spec:
12
    source:
13
      repoURL: https://github.com/example/platform-payment-service
14
      targetRevision: '{{head_sha}}'
15
      path: deploy/preview
16
    destination:
17
      namespace: 'preview-{{number}}'

One ephemeral env per PR, gone when the PR closes. Watch resource quotas.

11. Multi-Environment with Kustomize

Kustomize is the default rendering engine for first-party manifests. Helm only for third-party charts (see §12).

11.1 Layout

1
base/payment-service/
2
├── deployment.yaml
3
├── service.yaml
4
├── configmap.yaml
5
├── hpa.yaml
6
└── kustomization.yaml

1
apiVersion: kustomize.config.k8s.io/v1beta1
2
kind: Kustomization
3
resources:
4
  - deployment.yaml
5
  - service.yaml
6
  - configmap.yaml
7
  - hpa.yaml
8
commonLabels:
9
  app.kubernetes.io/name: payment-service
10
  app.kubernetes.io/part-of: platform

1
apiVersion: kustomize.config.k8s.io/v1beta1
2
kind: Kustomization
3
namespace: platform
4
resources:
5
  - ../../../base/payment-service
6
patches:
7
  - path: deployment-patch.yaml
8
    target:
9
      kind: Deployment
10
      name: payment-service
11
images:
12
  - name: ghcr.io/example/platform-payment-service
13
    newTag: 1.4.2
14
configMapGenerator:
15
  - name: payment-service-config
16
    behavior: merge
17
    literals:
18
      - SPRING_PROFILES_ACTIVE=prod
19
      - LOG_LEVEL=INFO
20
replicas:
21
  - name: payment-service
22
    count: 6

11.2 Rules

• Base is environment-neutral. No env-specific values. No replica counts, no image tags pinned in base (use a placeholder).
• Image tag is owned by overlay. Never in base. CI / Image Updater writes the overlay file.
• No vars: between envs. Each overlay is fully self-contained.
• commonLabels only in base unless an env truly needs an extra label.
• No symbolic links across overlays — ArgoCD repo-server may not follow them in all storage backends.

12. Helm in GitOps

Helm is fine for third-party charts you don’t author (Prometheus, cert-manager, ingress-nginx, Postgres operator). Avoid Helm for first-party services — Kustomize handles the same patterns with less ceremony.

12.1 Pin chart versions

1
spec:
2
  source:
3
    repoURL: https://prometheus-community.github.io/helm-charts
4
    chart: kube-prometheus-stack
5
    targetRevision: 65.5.0     # exact version, never '*'
6
    helm:
7
      releaseName: monitoring
8
      valueFiles:
9
        - $values/envs/prod/kube-prometheus-stack/values.yaml
10
      parameters:
11
        - name: prometheus.prometheusSpec.retention
12
          value: 30d

12.2 Render vs install

ArgoCD calls helm template at sync time. It does NOT write a Helm release secret. If your chart has post-install hooks that depend on a Helm release object, you have to either move them to ArgoCD hooks (§8.2) or use the Replace=true sync option.

12.3 OCI charts

1
source:
2
  repoURL: oci://ghcr.io/example/charts
3
  chart: platform-shared
4
  targetRevision: 2.1.0

OCI registries supported since ArgoCD 2.7. Authenticate via repository secret with the OCI scheme.

13. Promotion Strategy

How does a tag tested in dev reach prod?

13.1 Folder copy (recommended)

PR copies (or bumps) the image tag in envs/staging/<svc>/kustomization.yaml from the value in envs/dev/<svc>/kustomization.yaml. Same for staging → prod.

Pros: full audit (one PR = one promotion); easy to diff; reviewable.

Cons: a few seconds of manual work per promotion. (Often the right cost.)

13.2 Tag-based promotion via Image Updater

Image Updater (§15) sees a new tag matching prod-* and writes the bump. CI tags an image prod-1.4.2 only after staging soak passes.

Pros: zero-touch promotion.

Cons: implicit; auditors must read updater rules to understand why a deploy happened.

13.3 PR automation (Kargo, argo-promotion)

External controller opens promotion PRs automatically when upstream env stays healthy for N minutes. Use only when you have ≥ 4 environments and manual PRs become noise. Adds a new tool to the operate list.

13.4 Never:

• Sync a manifest with a latest tag in prod.
• Re-tag an image in the registry (e.g., move prod-stable to a new digest). Breaks ArgoCD’s diff because the manifest didn’t change but the runtime did.

14. Secrets Management

Plaintext secrets MUST NOT live in Git. Three viable patterns; pick one per project in an ADR.

14.1 Sealed Secrets (Bitnami)

Cluster has a controller with a private key. Developers encrypt secrets with the public key into SealedSecret CRs, commit them, and the controller decrypts to a real Secret in the cluster.

kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets \
  --format yaml < secret.yaml > sealed-secret.yaml

Pros: simple, self-contained, cluster-only trust boundary. Cons: private key is the disaster recovery artifact — back it up offline. If lost, every sealed secret in Git is unreadable.

14.2 External Secrets Operator (ESO)

Cluster has a controller that pulls from an external store (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, Azure Key Vault) and projects them as Kubernetes Secrets. Git holds only a reference (ExternalSecret CR).

1
apiVersion: external-secrets.io/v1
2
kind: ExternalSecret
3
metadata:
4
  name: payment-service-db
5
spec:
6
  refreshInterval: 1h
7
  secretStoreRef:
8
    name: aws-secrets-manager
9
    kind: ClusterSecretStore
10
  target:
11
    name: payment-service-db
12
  data:
13
    - secretKey: PASSWORD
14
      remoteRef:
15
        key: platform/prod/payment-service/db
16
        property: password

Pros: best when org already has a secrets store; clear separation of duties; rotation is upstream. Cons: adds an external dependency on the secrets store; cluster needs IAM access.

14.3 SOPS + age/PGP

Files in Git are encrypted with SOPS using age or PGP keys. ArgoCD has no native SOPS support — you need a plugin (argocd-vault-plugin, helm-secrets, or a custom CMP).

Pros: works in plain Git, no extra controller. Cons: adds a config management plugin to the repo-server; key management still required.

14.4 Hard rules

• Never commit a plaintext Secret CR, even in a private repo. Defense-in-depth.
• Rotate the sealing/decryption key on a schedule and on staff offboarding.
• Audit access to the secrets store. ESO config is fine in Git; the IAM role behind it isn’t (back it with Terraform).
• CI/CD systems get short-lived OIDC tokens, not long-lived secret store credentials.

See security.md §Secrets Management for the threat model.

15. Image Update Automation

ArgoCD Image Updater watches container registries and writes tag bumps back to the GitOps repo.

1
# Annotations on the Application
2
metadata:
3
  annotations:
4
    argocd-image-updater.argoproj.io/image-list: payment=ghcr.io/example/platform-payment-service
5
    argocd-image-updater.argoproj.io/payment.update-strategy: semver
6
    argocd-image-updater.argoproj.io/payment.allow-tags: regexp:^1\.[0-9]+\.[0-9]+$
7
    argocd-image-updater.argoproj.io/write-back-method: git
8
    argocd-image-updater.argoproj.io/git-branch: main

Update strategies:

Write-back methods:

• argocd — directly patches the Application (transient, not GitOps; not recommended).
• git — commits to the GitOps repo (true GitOps; recommended).

The updater needs a Git write credential. Store it as a Secret in argocd-image-updater namespace. Use a deploy key with write access scoped to the GitOps repo only, not a personal token with org-wide access.

Limit scope. Updater should be enabled per Application that opts in via annotation, not cluster-wide. Don’t let an updater bug auto-promote a half-baked image to prod.

16. Progressive Delivery (Argo Rollouts)

Deployment does a rolling update. For canary, blue/green, or analysis-gated rollouts use Argo Rollouts — a CRD that replaces Deployment.

1
apiVersion: argoproj.io/v1alpha1
2
kind: Rollout
3
metadata:
4
  name: payment-service
5
spec:
6
  replicas: 6
7
  selector:
8
    matchLabels:
9
      app.kubernetes.io/name: payment-service
10
  template:
11
    metadata:
12
      labels:
13
        app.kubernetes.io/name: payment-service
14
    spec:
15
      securityContext:
16
        runAsNonRoot: true
17
        runAsUser: 10001
18
        seccompProfile:
19
          type: RuntimeDefault
20
      containers:
21
        - name: payment
22
          # Prefer a distroless (or non-root) base image so the container has no shell/root user.
23
          image: ghcr.io/example/platform-payment-service:1.4.2
24
          ports:
25
            - containerPort: 8080
26
          securityContext:
27
            allowPrivilegeEscalation: false
28
            readOnlyRootFilesystem: true
29
            capabilities:
30
              drop:
31
                - ALL
32
  strategy:
33
    canary:
34
      canaryService: payment-service-canary
35
      stableService: payment-service-stable
36
      trafficRouting:
37
        nginx:
38
          stableIngress: payment-service
39
      steps:
40
        - setWeight: 10
41
        - pause: { duration: 5m }
42
        - analysis:
43
            templates:
44
              - templateName: success-rate
45
            args:
46
              - name: service-name
47
                value: payment-service-canary
48
        - setWeight: 30
49
        - pause: { duration: 10m }
50
        - setWeight: 60
51
        - pause: { duration: 10m }
52
        - setWeight: 100

1
apiVersion: argoproj.io/v1alpha1
2
kind: AnalysisTemplate
3
metadata:
4
  name: success-rate
5
spec:
6
  args:
7
    - name: service-name
8
  metrics:
9
    - name: success-rate
10
      interval: 1m
11
      successCondition: result[0] >= 0.99
12
      failureLimit: 3
13
      provider:
14
        prometheus:
15
          address: http://prometheus.monitoring:9090
16
          query: |
17
            sum(rate(http_server_requests_seconds_count{
18
              service="{{args.service-name}}", status!~"5.."
19
            }[2m])) /
20
            sum(rate(http_server_requests_seconds_count{
21
              service="{{args.service-name}}"
22
            }[2m]))

• Analysis runs Prometheus queries (or NewRelic / Datadog / Wavefront) and aborts the rollout on failure.
• Traffic routing integrates with NGINX, Istio, Linkerd, AWS ALB, SMI, Traefik.
• Blue/green strategy is one alternative; canary is more common for stateless HTTP services.

ArgoCD treats Rollout as a healthy first-class resource via the bundled Lua health check. No special config beyond installing the Rollouts controller via GitOps.

17. Drift Detection + Notifications

17.1 Detection

ArgoCD reconciler diffs every Application every 3 minutes (configurable). States:

17.2 Notifications

1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: argocd-notifications-cm
5
  namespace: argocd
6
data:
7
  service.slack: |
8
    token: $slack-token
9
  template.app-degraded: |
10
    message: |
11
      :rotating_light: {{.app.metadata.name}} is *{{.app.status.health.status}}*
12
      Project: {{.app.spec.project}}
13
      Repo: {{.app.spec.source.repoURL}}
14
      Revision: {{.app.status.sync.revision}}
15
  trigger.on-degraded: |
16
    - description: Application degraded
17
      send: [app-degraded]
18
      when: app.status.health.status == 'Degraded'
19
  subscriptions: |
20
    - recipients: [slack:platform-alerts]
21
      triggers: [on-degraded, on-sync-failed]

Per-app subscription via annotation:

1
metadata:
2
  annotations:
3
    notifications.argoproj.io/subscribe.on-degraded.slack: platform-payment-alerts

Bake critical triggers (sync-failed, degraded, deleted) into project-default subscriptions. Per-team channels via annotation.

18. Health Checks + Resource Customizations

Default health checks exist for Deployment, StatefulSet, DaemonSet, Pod, Service, Ingress, Job, PVC, ReplicaSet, Application, Rollout. For custom CRDs (operators), ArgoCD shows them as Healthy immediately — wrong.

Add a Lua health check:

1
data:
2
  resource.customizations.health.cert-manager.io_Certificate: |
3
    hs = {}
4
    if obj.status ~= nil then
5
      if obj.status.conditions ~= nil then
6
        for i, condition in ipairs(obj.status.conditions) do
7
          if condition.type == "Ready" and condition.status == "False" then
8
            hs.status = "Degraded"
9
            hs.message = condition.message
10
            return hs
11
          end
12
          if condition.type == "Ready" and condition.status == "True" then
13
            hs.status = "Healthy"
14
            hs.message = condition.message
15
            return hs
16
          end
17
        end
18
      end
19
    end
20
    hs.status = "Progressing"
21
    hs.message = "Waiting for certificate"
22
    return hs

Test Lua snippets with the argocd admin CLI before committing.

19. Multi-Cluster

One ArgoCD installation, N target clusters. Register each cluster:

argocd cluster add my-prod-context --name prod-eu-west-1 --grpc-web

Creates a Secret in the argocd namespace with the kubeconfig. The Application’s destination.server field references the cluster API URL.

19.1 Hub-and-spoke

19.2 ApplicationSet cluster generator

1
generators:
2
  - clusters:
3
      selector:
4
        matchLabels:
5
          env: prod

ArgoCD labels each registered cluster Secret; the generator fans out one Application per matching cluster. Add a cluster → it appears automatically.

20. Disaster Recovery + Bootstrap

ArgoCD is recoverable from Git plus a small set of inputs:

20.1 DR runbook outline

# 1. Provision cluster
terraform apply -target=module.cluster

# 2. Bootstrap ArgoCD
kubectl apply -k bootstrap/argocd

# 3. Restore Sealed Secret key (if Sealed Secrets used)
kubectl apply -f /secure-backup/sealed-secrets-key.yaml

# 4. Wait for ArgoCD CRDs
kubectl wait --for=condition=Established crd/applications.argoproj.io --timeout=300s

# 5. Apply root app — everything else reconciles
kubectl apply -f apps/root-app.yaml

Test this runbook on a fresh cluster at least quarterly. A DR plan that has never been executed is a hypothesis, not a plan.

21. Observability

ArgoCD exposes Prometheus metrics on every controller. Scrape and alert on:

Tracing: ArgoCD 2.13+ emits OTEL spans for repo-server rendering, controller reconcile, and API calls. Wire to the same OTEL collector as the application services. See obs-tracing.md.

Logs: JSON to stdout. Ship to the same log pipeline as workloads. Field trace_id correlates a sync event with the downstream service rollout.

22. Anti-Patterns

1. kubectl apply from a laptop or CI to a managed cluster. Defeats GitOps. CI pushes to Git only.
2. helm install / helm upgrade outside ArgoCD. Same as above — creates state ArgoCD doesn’t track.
3. Mutable image tags in prod (:latest, :main, :stable). Disables rollback via git revert (the tag still points to a newer digest).
4. Plaintext Secret in Git, even in a private repo. Always sealed, ESO-referenced, or SOPS-encrypted.
5. selfHeal: false on first-party services without an explicit reason. Defeats reconciliation.
6. Disabling prune because “I’m scared of deletions.” Then GitOps doesn’t actually mirror Git — you have an additive system, not a sync.
7. Per-env branches without a process (dev, staging, main branches). Drift between branches is inevitable; promotion becomes merge-conflict resolution.
8. One giant Application for “everything in this namespace.” Slow diffs, all-or-nothing syncs, painful rollback. Split per service or per chart.
9. No AppProject (everything in default). Any compromised Application can deploy anywhere.
10. No finalizers on Applications. Deleting an Application leaves orphan resources.
11. Manual Replace=true as the default sync option. Breaks SSA-managed resources and operator semantics. Use only for resources that genuinely cannot be patched.
12. Image Updater with org-wide PAT. Use a deploy key scoped to the GitOps repo only.
13. CI doing kubectl rollout undo as the rollback mechanism. Rollback is git revert → ArgoCD reapplies prior state.
14. Skipping ignoreDifferences for HPA-managed spec.replicas. Reconciler fights HPA forever; CPU burns; alerts page on phantom drift.
15. No sealing-key backup for Sealed Secrets. One controller pod loss + no backup = every encrypted secret in Git is rubble.
16. Promotion via re-tagging the same image in the registry. Manifest didn’t change → ArgoCD sees no diff → no sync → confusion.
17. ArgoCD running inside the workload cluster it manages, with no DR plan. Cluster outage = no recovery path.
18. App-of-apps where the root has selfHeal: false. A drifted child can’t be auto-recovered. Root must self-heal.
19. Sync waves abused for “deploy order” when waves are for resource-kind order. If service B must come up after service A is Healthy, split into separate Applications and use a notification trigger or Argo Workflows — not waves.
20. Webhook from Git to ArgoCD over public internet without verification. Use the shared secret + IP allowlist. Otherwise an attacker can force-sync.

23. Pre-Merge Checklist

For any PR to the GitOps repo:

• Target file path matches the env tier the PR claims (envs/dev/... vs envs/prod/...).
• Image tag is immutable (SHA digest or semver tag), not a branch name or latest.
• No plaintext secrets, tokens, or private keys in the diff (grep -E 'BEGIN .* PRIVATE KEY|sk-[a-z]|ghp_|AKIA').
• No new Application references a repo outside the project’s sourceRepos allow-list.
• No new Application deploys to a destination outside the project’s destinations allow-list.
• automated.prune: true and selfHeal: true unless an in-repo comment explains the exception.
• finalizers: [resources-finalizer.argocd.argoproj.io] present on Application.
• Sync waves only used for resource-kind order, not service order.
• If adding a CRD, the corresponding CR uses sync wave > the CRD’s wave.
• If adding a hook, hook-delete-policy is set (no orphan Jobs).
• Pod/container security context hardened: non-root containers + readOnlyRootFilesystem + drop ALL caps (runAsNonRoot, allowPrivilegeEscalation: false, seccompProfile: RuntimeDefault).
• If adding ignoreDifferences, the comment explains which controller is mutating the field.
• Helm targetRevision is pinned to an exact version (no *, no latest).
• CHANGELOG / commit body explains what is being deployed and why (link to upstream PR or issue).
• Promotion to prod has at least one CODEOWNER (SRE) approval.
• No simultaneous bump of multiple services in the same PR (one concern per commit; easier rollback).
• For prod, targetRevision is a tag or SHA — never a branch.
• For prod, signed commits are required if AppProject has signatureKeys configured.

For any code change in an app repo that triggers a deploy:

• CI does NOT call kubectl, helm install, or any cluster API.
• CI’s only deploy-time output is a commit/PR to the GitOps repo.
• If a DB migration is required, the corresponding Liquibase changeset is committed and a PreSync hook is configured (§8.2).
• Health check endpoints (/actuator/health/liveness, /readiness) return correct status from the new image — ArgoCD reads them.
• Image is signed (cosign / Notary). AppProject can enforce signature verification with kyverno or policy-controller as a sync hook.

References

• OpenGitOps principles: https://opengitops.dev/
• ArgoCD docs: https://argo-cd.readthedocs.io/
• ApplicationSet generators: https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators/
• Argo Rollouts: https://argo-rollouts.readthedocs.io/
• Sealed Secrets: https://sealed-secrets.netlify.app/
• External Secrets Operator: https://external-secrets.io/
• Kustomize: https://kustomize.io/