How to Streamline CI/CD With Octopus and TeamCity

It's no secret that without clear and organized processes in place, developers may struggle to collaborate effectively, leading to delays in delivering software updates. A few years ago, the Social Discovery Group team faced the challenge of a suboptimal CI/CD process. At that time, the team used TeamCity and Octopus, each with its strengths. For instance, Octopus is convenient for deployments, while TeamCity is good for automated tests and sufficiently convenient for project builds. To construct a comprehensive and visually appealing CI/CD pipeline that is maximally convenient and flexible in configuration, it is necessary to use a mix of tools. The code was stored in a local repository on Bitbucket for several projects. The SDG team studied the issue and decided to optimize the process using the existing tools.

Key optimization goals:

1. Automatic build and deployment from TeamCity to specified environments.
2. Naming builds: "release" is added to the master branch to differentiate builds by names.
3. Automatic build and deployment when pushing to corresponding branches on respective test environments of respective services.
4. Establishing a process where deployment to test and development environments must be completed before deployment to staging and then to production. This was implemented in Octopus.

The SDG team decided to use TeamCity for builds and automated tests, and Octopus for deployments.

What was implemented in TeamCity:

1. TeamCity allows the use of three agents in the free version, which was sufficient for the SDG team. They installed a new agent, added it to the pool, and applied it to their templates.
2. At the time of using the latest version of TeamCity, the team was working on Ubuntu Server. The screenshot shows the additional plugins used by the team:

   
   

3. From the tools, the team added plugins, such as Allure 2.14.0 for report creation and Nuget 5.5.1.
4. To simplify the execution of similar tasks, the SDG team created several templates for different types of deployments: NuGet and services.
5. Each of these templates included several steps, reflected in the screenshots below.

The deployment for NuGet looked as follows:

It's worth noting that depending on whether the branch is the master or not, "-release" was added to the release (steps 3, 4).

The deployment for services can be seen below:

For each service, corresponding variables were substituted based on system variables (service name, %build.number%, and others).

An example of the Docker Build step is presented in the screenshot:

Each project repository contained the corresponding Dockerfile.

The differences between steps 4 and 5, as mentioned earlier, were as follows:

The %deploymentTarget% variable served as the Environment(s) parameter, to which the corresponding stage(s) in Octopus were passed during deployment (e.g., Test, Dev). When changes were pushed to the respective branches (configured) of development teams, builds and software deployments to the corresponding test environments were automatically performed. The settings are visible in the screenshot below. To connect with Octopus, two global parameters needed to be added: octopus.apiKey and octopus.url

Additionally, the SDG team connected a common NuGet repository and Container Registry for all projects in the Connections section.

Furthermore, SDG recommends configuring email notifications in the Email Notifier section, setting up backups in the Backup section, creating necessary groups, assigning appropriate roles, and adding users to the required groups. The main setup is completed, and in conclusion, the team recommends regularly checking for updates and updating TeamCity once a month.

Next, the Social Discovery Group team moved on to configuring Octopus. This article will not describe the installation details, basic user rights settings, and other aspects, because you can easily do them by yourself. The team immediately addressed the lifecycle, which is configured in the Library section. In the screenshot below, you can see an example flow of the SDG team:

Then, the team created all necessary variable groups by themes in Variable Sets. For each variable, values were set and dependencies on the environment, targets and target roles (tags) were established. An example is shown in the screenshot below:

The clusters in Kubernetes served as targets, and the target roles were tags attached to the corresponding clusters or computer environments. All this can be configured in the Infrastructure section.

Projects could also be grouped, and a convenient dashboard could be set up to display services, stages, and versions deployed on them.The deployment process for SDG looked as follows: all test stages were combined into one step, and a common template was created for them, similarly for the stage and live stages.

The screenshot below shows how this looked for the SDG team:

On the right, the previously described Lifecycle was selected. The Deploy a Package stage included fairly simple default settings.

For the Deploy Raw Kubernetes Yaml stage, the SDG team used universal self-written Yaml templates. In this example - Kubernetes Script, is explained in more detail below. Corresponding parameters marked in red were also substituted. It's worth noting that necessary global variable groups were connected in the Variables->Variable Sets menu, and project-specific variables were set in the Variables->Project menu, which had higher priority.

In this article, the SDG team decided to skip such details as adding a logo to the project, setting up triggers, or other minor details. Let's focus on two important menu items: 1 - Releases, where you can always see the version and creation date of a particular release; this information is also displayed on the project dashboard, 2 - Variables->Preview, where you can see which variables will be substituted for the corresponding stage.

Moving on to the most important part - deployment of Yaml templates in Kubernetes clusters. They were created in the Library->Step Templates section. Below, the SDG team presented a screenshot using their parameters. For each parameter, you could choose a tag, type, and default value, as well as add a description, which is highly recommended.

The code in this case looked as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: '#{Octopus.Project.Name | ToLower}'
  namespace: #{Octopus.Environment.Name | ToLower}    
  labels:
    Octopus.Kubernetes.DeploymentName: '#{Octopus.Project.Name | ToLower}-#{Octopus.Environment.Name | ToLower}'
spec:
  replicas: #{Replicas}
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
  selector:
    matchLabels:
      Octopus.Kubernetes.DeploymentName: '#{Octopus.Project.Name | ToLower}-#{Octopus.Environment.Name | ToLower}'
  template:
    metadata:
      labels:
        Octopus.Kubernetes.DeploymentName: '#{Octopus.Project.Name | ToLower}-#{Octopus.Environment.Name | ToLower}'
    spec:
      volumes:
#{if usesidecar}
        - name: dump-storage
          persistentVolumeClaim:
            claimName: dumps-#{Octopus.Environment.Name | ToLower}
#{/if}
#{if MountFolders}
#{each folder in MountFolders}
        - name: volume-#{folder | ToBase64 | Replace "\W" X | ToLower}
          hostPath:
            path: #{folder}
            type: DirectoryOrCreate
#{/each}
#{/if}
        - name: logs-volume
          hostPath:
            path: #{LogsDir}
            type: DirectoryOrCreate
        - name: appsettings
          secret:
            secretName: #{Octopus.Project.Name | ToLower}
#{if Secrets}
#{each secret in Secrets}
        - name: #{secret.name}
          secret:
            secretName: #{secret.name}
#{/each}
#{/if}
#{if usesidecar}
        - name: diagnostics
          emptyDir: {}
        - name: dumps
          configMap:
            name: dumps
            defaultMode: 511    
#{/if}
      containers:
      - name: #{Octopus.Project.Name | ToLower}-#{Octopus.Environment.Name | ToLower}-container
        image: #{DockerRegistry}/projectname.#{Octopus.Project.Name | ToLower}:#{Octopus.Release.Notes}
        #{if resources}
        resources:
        #{each resource in resources}
          #{resource.Key}:
            #{each entry in resource.Value}
            #{entry.Key}: #{entry.Value}
            #{/each}
        #{/each}
        #{/if}
        ports:
        - name: http
          containerPort: 80
          protocol: TCP
        env:
        - value: "Development"
          name: "ASPNETCORE_ENVIRONMENT"
        - name: DD_ENV
          value: "#{Octopus.Environment.Name | ToLower}"
        - name: DD_SERVICE
          value: "#{Octopus.Project.Name | ToLower}-#{Octopus.Environment.Name | ToLower}"
        - name: DD_VERSION
          value: "1.0.0"
        - name: DD_AGENT_HOST
          value: "#{DatadogAgentHost}"
        - name: DD_TRACE_ROUTE_TEMPLATE_RESOURCE_NAMES_ENABLED
          value: "true"
        - name: DD_RUNTIME_METRICS_ENABLED
          value: "true"
        volumeMounts:
#{if usesidecar}
        - name: dump-storage
          mountPath: /tmp/dumps
#{/if} 
#{if MountFolders}
#{each folder in MountFolders}
        - mountPath: #{folder}
          name: volume-#{folder | ToBase64 | Replace "\W" X | ToLower}
#{/each}
#{/if}
        - mountPath: #{LogsDir}
          name: logs-volume
#{if usesidecar}
        - name: diagnostics
          mountPath: /tmp
#{/if}
        - name: appsettings
          readOnly: true
          mountPath: /app/appsettings.json
          subPath: appsettings.json
#{if Secrets}
#{each secret in Secrets}
        - name: #{secret.name}
          readOnly: true
          mountPath: #{secret.mountPath}
          subPath: #{secret.subPath}
#{/each}
#{/if}
        readinessProbe:
          httpGet:
            path: hc
            port: http
            scheme: HTTP
          initialDelaySeconds: #{InitialDelaySeconds}
        imagePullPolicy: IfNotPresent
        securityContext: {}
#{if usesidecar}
      - name: sidecar
        image: '#{DockerRegistry}/monitor:3'
        command:
          - /bin/sh
        args:
          - '-c'
          - while true; do . /app/init.sh; sleep 1m;done
        env:
          - name: USE_MEMORY
            value: '2048'
          - name: PROJECT
            value: "#{Octopus.Project.Name | ToLower}-#{Octopus.Environment.Name | ToLower}"
        resources: {}
        volumeMounts:
          - name: diagnostics
            mountPath: /tmp
          - name: dump-storage
            mountPath: /tmp/dumps
          - name: dumps
            mountPath: /app/init.sh
            subPath: init.sh
      shareProcessNamespace: true
#{/if}
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: environment
                operator: In
                values:
                - "#{Node}"
---
apiVersion: v1
kind: Service
metadata:
  name: #{Octopus.Project.Name | ToLower}
  namespace: #{Octopus.Environment.Name | ToLower}
  labels:
    Octopus.Kubernetes.DeploymentName: '#{Octopus.Project.Name | ToLower}-#{Octopus.Environment.Name | ToLower}'
spec:
  type: ClusterIP
  selector:
    Octopus.Kubernetes.DeploymentName: '#{Octopus.Project.Name | ToLower}-#{Octopus.Environment.Name | ToLower}'
  ports:
  - name: http
    port: 80
    targetPort: http
    protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    cert-manager.io/cluster-issuer: "letsencrypt-cluster-issuer"
    cert-manager.io/renew-before: '#{LetsencryptRenewBefore}'
    kubernetes.io/ingress.class: nginx
  #{if IngressAnnotations}
  #{each annotation in IngressAnnotations}
    #{annotation.Key}: #{annotation.Value}
  #{/each}
  #{/if}
  name: #{Octopus.Project.Name | ToLower}
  namespace: #{Octopus.Environment.Name | ToLower}
  labels:
    Octopus.Kubernetes.DeploymentName: '#{Octopus.Project.Name | ToLower}-#{Octopus.Environment.Name | ToLower}'
spec:
  tls:
#{if ExternalHost}
#{each host in ExternalHost}
  - hosts:
      - #{host}
    secretName: #{Octopus.Project.Name | ToLower}-#{host | ToBase64 | Replace "\W" X | ToLower}-tls
#{/each}
#{/if}
  rules:
#{if ExternalHost}
#{each host in ExternalHost}
  - host: '#{host}'
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: #{Octopus.Project.Name | ToLower}
            port:
              name: http
#{/each}
#{/if}
#{if usesidecar}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: dumps
  namespace: #{Octopus.Environment.Name | ToLower}
data:
  init.sh: |-
    #!/usr/bin/env bash
    mem=$(ps aux | awk '{print $6}' | sort -rn | head -1)
    mb=$(($mem/1024))
    archiveDumpPath="/tmp/dumps/$PROJECT-$(date +"%Y%m%d%H%M%S").zip"
    fullPathGc="/tmp/$PROJECT-$(date +"%Y%m%d%H%M%S").dump"
    echo "mem:" $mb" project:" $PROJECT "use:" $USE_MEMORY
    if [ "$mb" -gt "$USE_MEMORY" ]; then
    export USE_MEMORY=$(($USE_MEMORY*2))
    pid=$(dotnet-dump ps | awk '{print $1}')
    dotnet-dump collect -p $pid -o $fullPathGc
    zip $fullPathGc.zip $fullPathGc
    mv $fullPathGc.zip $archiveDumpPath
    rm $fullPathGc
    fi
#{/if}

All variables in Octopus were specified in the following format in the code: '#{Octopus.Project.Name | ToLower}', where the last part indicates converting to lowercase.

The last configuration file was created to automatically save the state of .NET services when they reached a certain memory usage limit. This significantly helped identify memory leaks during development and promptly fix the services.

Finally, the service dashboard looked as follows:

Development and testing teams found it very convenient to work with this dashboard.

Optimization results:

1. The SDG team built an efficient CI/CD process that significantly improved the speed and convenience of development. The team worked quite long within this process framework.
2. SDG also introduced automated tests into the convenient TeamCity tool and automated service deployments.
3. Through the user-friendly Octopus dashboard, the team configured access rights and could manage deployments and environments.

Subsequently, SDG implemented many other features in Octopus. For example, automatic shutdown of clusters at night on a schedule.

However, the pursuit of perfection knows no bounds. The Social Discovery Group team advanced their development by mastering Azure DevOps. They set up a process in Azure DevOps within one ecosystem on Helm, even more comprehensive and efficient. That will be covered in the next article.

We would love to hear about your experience in setting up and optimizing CI/CD using Octopus and TeamCity. Share your insights and tips!