GitOps Workflow Explained: A Beginner's Guide to Git-Centric Continuous Delivery

GitOps is revolutionizing the way we approach continuous delivery by leveraging Git as the single source of truth for both infrastructure and application configuration. This guide serves as an introduction for developers and platform engineers who are eager to enhance their deployment processes by making them safer, more auditable, and repeatable. Here, you will discover core GitOps principles, a step-by-step workflow, comparisons of popular tools, and best practices for getting started.

What is GitOps?

High-level definition

GitOps uses Git repositories as the canonical source of truth for declarative system configurations. Automated reconciliation ensures that runtime environments, such as Kubernetes clusters, align with this desired state. Essentially, your Git repository acts as a blueprint, while an operator (a controller) guarantees that the actual state matches your blueprint.

How GitOps Differs from Traditional CI/CD

• Push-based CI/CD: Traditional Continuous Integration/Continuous Deployment (CI/CD) pipelines build artifacts and push changes directly to environments. Control and deployment logic reside within the CI system.
• Pull-based GitOps: The desired state is stored in Git. An agent (GitOps operator) running within the cluster pulls this desired state, reconciling the cluster to match it. This method provides a clearer separation of concerns and minimizes credential exposure outside the cluster.

Key Benefits in Simple Terms

• Auditability: Each deployment is tracked as a Git commit or pull request, creating a clear, versioned audit trail.
• Easy Rollback: Revert a Git commit or merge to roll back the environment to a previous state.
• Collaboration: Employ familiar Git workflows, including pull requests and code reviews, for changing infrastructure and applications.
• Repeatability: Declarative configs ensure consistent environments across clusters.

For further insights, you can refer to Weaveworks’ GitOps overview.

Core Principles of GitOps

1. Declarative Desired State: Store system configuration as declarative files such as Kubernetes manifests, Helm charts, or Kustomize overlays, focusing on what instead of how.

2. Git as the Source of Truth: All production changes must be documented in Git as commits or PRs to ensure traceability and effective collaboration using Git tools.

3. Automated Reconciliation (Pull Model): A GitOps operator, such as Argo CD or Flux, continuously compares the desired state in Git with the live cluster and applies changes to achieve alignment.

4. Continuous Verification: Automated tests, policy checks, and runtime monitoring validate that changes are valid and that the environment remains healthy both before merges and post-reconciliation.

These principles highlight Git as the remote control for your cluster, allowing automation to do its work effectively.

GitOps Workflow — Step-by-step

Here’s a concrete workflow illustrating a typical GitOps deployment, using a simple example of updating a container image tag in a Kubernetes Deployment:

1. Developer Makes a Change Locally:

   • Clone the Git repo with application manifests.
   • Edit the Deployment to update the container image tag (e.g., my-app:v1.2.0 to my-app:v1.3.0).

   Example snippet (Kubernetes Deployment):

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: my-app
   spec:
     replicas: 2
     template:
       spec:
         containers:
         - name: my-app
           image: ghcr.io/example/my-app:v1.3.0
   

2. Open a Pull Request (PR):

   • Push a feature branch and create a PR, providing a description of the change and linking related issues.

3. CI Runs Tests and Validation:

   • CI jobs lint YAML, run unit tests, and optionally execute policy checks. If any checks fail, the PR is updated accordingly.

4. Merge to the Main Branch:

   • After code review and passing checks, merge the PR. This commit now reflects the desired state for the cluster.

5. GitOps Operator Detects Changes:

   • The operator (Argo CD or Flux) either polls Git or responds to notifications to pull the latest commit.

6. Operator Applies Changes to the Cluster:

   • The operator evaluates the cluster against the Git desired state and applies any necessary changes.

7. Observability and Verification:

   • Post-reconciliation, the operator ensures resource health, integrating with monitoring tools to trigger smoke tests and progressive delivery strategies like canaries.

8. Rollback via Git:

   • To rollback, revert the commit or merge a PR that restores the previous configuration, prompting the operator to reconcile the cluster back.

Popular GitOps Tools and Ecosystem

Two leading open-source GitOps operators dominate the Kubernetes ecosystem: Argo CD and Flux. Both focus on continuous reconciliation from Git but exhibit distinct features.

Comparison: Argo CD vs. Flux

• Argo CD Getting Started
• Flux Getting Started

Benefits and Common Challenges

Benefits Recap

• Faster and safer deployments using familiar Git workflows.
• Enhanced collaboration through pull requests and reviews.
• Simple rollbacks via Git reverts.
• Consistent, repeatable environments.

Operational and Cultural Challenges

• Initial Setup Complexity: Configuring controllers, repository layouts, and access controls requires effort.
• Secrets Management: Ensure secrets are not stored in plain-text within Git; adopt a secure solution.
• Drift: Misconfigured operators or manual changes can cause discrepancies, if not detected.
• Learning Curve: Teams must familiarize themselves with declarative configuration and Git workflows.

When GitOps May Not Be Ideal

• Teams reliant on imperative tooling may struggle to transition to a declarative model.
• Environments with strict protocols against storing configuration in Git can face challenges, though signed/protected repos offer some mitigation.

Practical Mitigations

• Start small with a single app and cluster.
• Utilize secure secrets management (e.g., Sealed Secrets, SOPS with KMS, or Vault).
• Integrate automated tests and policy checks early.

Getting Started — Simple Hands-On Example

Prerequisites

• Git hosting account (GitHub, GitLab).
• Development Kubernetes cluster (kind, Minikube, or a managed cluster).
• kubectl installed and configured.

High-Level Quickstart (Argo CD or Flux)

1. Choose an operator and refer to its quickstart:
   • Argo CD Getting Started
   • Flux Documentation
2. Bootstrap or create an App that points to your Git repo containing manifests.
3. Commit a sample application manifest to the repo.
4. Modify an image tag or replica count, open a PR, merge, and observe the operator reconcile the change.

Sample Minimal Workflow Commands

# Clone sample repo
git clone [email protected]:example/gitops-sample.git
cd gitops-sample
# Edit manifests/my-app/deployment.yaml to update the image tag
git add .
git commit -m "Bump my-app image to v1.3.0"
git push origin feature/bump-image
# Open PR via your Git hosting UI, then merge when checks pass

Security and Compliance Considerations

Secrets Management

• Avoid storing plain-text secrets in Git. Utilize the following:
  • Sealed Secrets: Encrypt secrets before committing.
  • SOPS + KMS: Secure files in Git decrypted by the operator or CI.
  • HashiCorp Vault: Manage dynamic secrets out-of-band.

Applying Least Privilege and RBAC

• Limit the GitOps operator’s permissions. Implement Kubernetes RBAC to scope service accounts appropriately.

Ensuring Git Provenance and Signed Commits

• Enforce commit signing and branch protection to secure changes to vital branches.

Implementing Policy Enforcement

• Use policy as code (e.g., OPA/Gatekeeper) to maintain security and compliance during or prior to reconciliation.

Best Practices and Tips for Beginners

Start Small and Iterate

• Initiate with one application and a single dev cluster, building confidence prior to scaling.

Repository Strategies

• Consider a monorepo with environment overlays for simplicity.
• Alternatively, adopt multiple repos for stricter separation and access control.

Automate Tests and Policy Checks

• Integrate security scans and validation checks into your CI process before merging.

Monitor Drift and Failures

• Set up alerts for synchronization failures and health check regressions.

Document and Train the Team

• Develop runbooks detailing the GitOps flow, PR process, and rollback procedures.

Design with Separation of Concerns

• Utilize architecture patterns to maintain a clear separation between platform deployment concerns and application logic.

Troubleshooting Common Issues

Sync Failures and Reconciliation Errors

• Inspect operator logs or utilize kubectl describe and kubectl get events to diagnose issues.

Drift Detection and Manual Fixes

• Manual changes in the cluster can cause drift. Update Git to reflect any necessary adjustments.

Secrets Failing to Decrypt

• Ensure key setups, KMS permissions, or SOPS configurations are appropriately configured.

Operator Permission Errors

• Verify RBAC settings to ensure the service account has the required permissions.

Useful Commands

# Check Argo CD app status
argocd app get my-app
# View Flux logs
kubectl -n flux-system logs deploy/flux-controller
# Inspect events for a namespace
kubectl get events -n my-app-namespace

Conclusion and Further Resources

GitOps transforms deployments into safer, auditable, and collaborative processes using Git as a single source of truth and automating reconciliation with a pull-based operator. This approach simplifies rollbacks and encourages consistent, repeatable practices.

Next Steps:

1. Follow a quickstart with Argo CD or Flux using a small Kubernetes cluster and sample Git repo:

   • Argo CD Getting Started
   • Flux Documentation

2. First GitOps Deployment Checklist:

   1. Create a Git repo for declarative manifests.
   2. Include a minimal app manifest (Deployment + Service).
   3. Install Argo CD or Flux in a dev cluster.
   4. Configure the operator to monitor the repo.
   5. Make a small change via a PR and merge.
   6. Verify reconciliation and app status.

Further Reading:

• Weaveworks GitOps overview: link
• CNCF GitOps Working Group: link
• Argo CD documentation: link
• Flux documentation: link

For additional guidance on integrating Windows workloads, check out this Windows Containers Docker Integration Guide.

Try out a guided quickstart with either tool (Argo CD or Flux), keep your secrets secure, and don’t hesitate to share your experiences or queries in the comments.