The Basics of GitOps and Secure Deployment in Kubernetes

Today, I would like to discuss secrets and GitOps in the K8S world.Some of my friends who are not DevOps/SRE engineers and most developers asked me too many times how to store Secrets/API keys/environment variables within the deployment process. I hope this article will help to understand with a trivial explanation of the basics of how-to store secrets in the deployment process in the GitOps Way.

First things first

Plan of our journey:

• a small introduction about GitOps
• deploy k8s cluster ( EKS )
• encrypt sensitive data ( SOPS )
• deploy ArgoCD ( GitOps tool )
• deploy our application in the GitOps way
• inference

GitOps and why it is so important

Let's imagine you have some Git repo and deploy your application into Kubernetes via Jenkins, AKA IaC style (Infrastructure as a code). At first look, this architecture is magnificent. We have stored our code in repo, and we deploy it via any CI\CD tool. But why is GitOps so major? The answer to that question lies in how Kubernetes API works. When you apply something into Kubernetes, the API will make you aware that the syntax is acceptable or not. If it's OK, it will give you the lowdown that your resources are declaratively written. Kubernetes will deploy everything you declared. But the first problem is that k8s does not guarantee that and the second one is that if somebody deletes/change/update information about any component or resource, you will never know until it is late.The GitOps way implies that what is in the repo will be in Kubernetes. It literally syncs the states between your repo and Kubernetes. If an unspecified person changes something in Kubernetes ( delete or update config map, for example ), the GitOps tool will sync state and overwrite changes. It works in both ways.

It's easy with EKS

I will cover the deployment process with tool. I assume that you are already familiar with AWS and with K8S, so I will not explain much on it.
First of all, you need to install eksctl.

For security reasons, it is not a best practice to use AWS access keys. That's why we are going to use IAM Roles to have the ability to use the AWS KMS service within the Kubernetes pods. Hence, we need to create an AWS IAM Policy for KMS keys and assign this Policy to our EKS Worker group, which we will discuss later in this article. I have prepared two git repo for this reason. You are free to use them.

So what we are going to do next is:

• Create IAM role and put ARN reference into the EKS config
• Create KMS key
• Deploy EKS cluster with KMS Policy

You should see something like this in the KMS section of the AWS console.

How to encrypt the data with SOPS

In this section, we will discuss how to encrypt the data with SOPS. I've drawn a small explanation of how works with encryption and the KMS keys. In a few words, SOPS is a tool written by Mozilla to encrypt YAML\JSON and other formats by encryption only values and not keys. More information about this tool you can find .To install SOPS, you need to follow this command. We will encrypt with SOPS the official Jenkins helm chart values.yaml and push it to the repo.

ArgoCD - Declarative GitOps CD for Kubernetes

is a tool which we will deploy into Kubernetes and which will do our deployment process. I've prepared a simple schema of how ArgoCD will do the job.Argo CD is un-opinionated about how secrets are managed and let's you decide how to handle this problem independently by allowing you to customize the deployment workflow with any tool. You can read more about it , but in a few words, you need to do two things:1. It would be best if you injected the tool via custom tooling. You will have two options:

• Adding Tools Via Volume Mounts
• Build Your Own Image ( which I )

ARG ARGOCD_VERSION=v1.7.8

FROM argoproj/argocd:${ARGOCD_VERSION}

ARG SOPS_VERSION="v3.6.1"

# Switch to root for the ability to perform install
USER root

# Install tools needed for your repo-server to retrieve & decrypt secrets, render manifests 
# (e.g. curl, awscli, gpg, sops)
RUN apt-get update && \
    apt-get install -y \
        curl \
        awscli \
        gpg && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    curl -o /usr/local/bin/sops -L //github.com/mozilla/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux && \
    chmod +x /usr/local/bin/sops

# Switch back to non-root user
USER argocd

2. Register a new plugin in argocd-cm

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
data:
  configManagementPlugins: |
    - name: sops
      init:
        command: ["sh", "-xc"]
        args: ["sops -d values_encrypted.yaml > values_decrypted.yaml"]
      generate:
        command: ["sh", "-xc"]
        args: ["helm template -n ${ARGOCD_APP_NAMESPACE} ${ARGOCD_APP_NAME} -f values_decrypted.yaml ."]

Let's deploy the ArgoCD with our .Now you can login with your new password into UI via your Web browser

Last but not least - Application deployment!

Full description of declarative you can read from , but the only thing I would like to mention.

1. plugin -

So let's deploy our and check if the password is correct with our encryption.

Inference

As you can see from above, the secured deployment process is not rocket science. There a variety of options on how to do that and how to manage it. The conclusion from my side is - don't be afraid of trying something new.
I hope this article explained the basics of GitOps and secure deployment.
Few links to read:

• - Alternative CD tool
• - there are a lot of them.