Shifting Containers Left: On the Quest for Reproducible Development Environments

Photo by Casey Horner on Unsplash

The adding functionality part of coding happens on the left and is typically performed by Developers. These people value development environments that take care of the mundane (and more). Coding, building, running tests, iterating. Hey, if it works on my machine, what can go wrong?

The delivering functionality piece happens on the right, typically performed by people with more of an Ops acumen, living on the right. What they focus on is the ability to consistently and efficiently deploy software (including their dependencies) and keeping them running.

CI/CD folks live throughout the pipeline, promoting the product from development to deployment and operations. They, too, value repeatability. There is nothing worse than trying to introduce a single, tiny code change, only to realize that the entire build tool set has been patched twice since the previous push of some functionality.

# dev stage
FROM golang:1-alpine AS build-env
WORKDIR /go/src/example.com/mypackage
RUN apk add \
        ca-certificates \
        gcc musl-dev \
        git
COPY . .
RUN go get -d -v ./... \
    && go test -short -timeout 30s ./... \
    && go install -v ./...

# build stage
FROM alpine
RUN apk add \
        ca-certificates
WORKDIR /go/bin/
COPY --from=build-env /go/bin/ /go/bin/

Beyond the cyber security aspect (less code means less potential for exploitation), the mere size reduction should be a good incentive. From 359 MB down to... 5 MB when this article was written.

Now, would a developer want to develop in a Continuous Integration build environment? This proposal sounds questionable, as Continuous Integration environments are really meant for machines, not for humans.

If we look at the problem from the Ops side and go left, we drop a number of Python modules . Keep moving left: working in a directory cluttered with Python modules is not pleasing. Just think about the resulting 

.gitignore

requirements.txt

Mind however that we do not want the CI/CD tool to micromanage the promotion from development to testing, otherwise the said promotion cannot be performed manually. Otherwise, whenever we are closely iterating on that part of the pipeline, we would start copy-pasting steps from the CI/CD tool, say Gitlab's 

.gitlab-ci.yml

Let's picture the development cycle when something does not work quite well once in a lambda function. The developer wants to iterate quickly by introducing a small change locally, then push, then inspect, then code again.

Quickly means without checking in code and pushing it through the pipeline, so without necessarily involving the CI/CD tool. After all, even developers of the pipeline itself want to be able to promote environments manually before asking a CI/CD tool to automate the process. So our CI/CD tool is our best friend, but right there and now it gets in the way of getting the job done.

WWOD - What would an Ops do?

Hang on a minute. Orchestration of the development environment. Is that really necessary? Let's take a few steps back. What would an alternative solution look like? Do mind that we are not focusing on environment-specific solutions, ala  for Kubernetes.

docker run --rm -it -v ${HOME}:/home/user \
		   my-development-image:latest /bin/bash

• The gymnastic of what needs to run inside. Code editor with at least syntax highlighting? Compiler or interpreter? Command-line linter with common configuration settings throughout the team? git hooks to enforce the said common development practices?
• As opposed to what needs to run outside the container. A full GUI-based IDE? The same compiler, interpreter, linter, git hooks as listed in the previous point?
• The components sitting outside the container need to be efficiently installed, version controlled and updated.
• And then the 

  my-development-image

   image needs to be maintained and refreshed when needed.

That is basically what VS Code's 

Remote - Containers

The language support and other SDLC tooling kind of VS Code extensions that we want available to the (containerized) development environment are defined in 

.devcontainer.json

"extensions": [
	"ms-python.python",
        "mhutchie.git-graph"
],

With that in place, where do we store the configuration of the development environment, as far as the piece of software under development is concerned? Think API keys, Cloud credentials, etc. The configuration of the development environment can be specified in a 

.env

zsh

The 

.env

env

Coming back to the aforementioned AWS lambda function. Our development environment can now benefit from docker-compose orchestration that will make sure the lambda function always sees a in front of it, for its persistence needs. That is, provided that 

.devcontainer/docker-compose.yml

version: '3.5'
services:    
  controller:
    build:
      context: ..
      dockerfile: Dockerfile
    volumes: 
      - /home/myuser/git/the-app:/workspace
      - /home/myuser/.aws:/home/developer/.aws
    links:
      - dynamodb
    environment:
      - SLACK_WEBHOOK
    command: sleep infinity

.services/controller/environment

SLACK_WEBHOOK

  dynamodb:
    image: amazon/dynamodb-local
    restart: unless-stopped
    volumes:
      - /data/db

Remote - Containers

.devcontainer/devcontainer.json

        "dockerComposeFile": "docker-compose.yml",
        "service": "controller",

Notice that we keep a

workspaceFolder

workspaceMount

    "workspaceFolder": "/workspace",    

All in all, this is what our

devcontainer.json

{
  "name": "Python 3",
  "dockerComposeFile": "docker-compose.yml",
  "service": "controller",
  "postCreateCommand": "npm install",
  "extensions": [
    "ms-python.python",
    "mhutchie.git-graph"
  ],
  "settings": {
    "terminal.integrated.shell.linux": "/bin/bash",
    "python.pythonPath": "/usr/local/bin/python",
    "python.linting.enabled": true,
    "python.linting.pylintEnabled": true,
    "python.linting.pylintPath": "/usr/local/bin/pylint"
  },
  "workspaceFolder": "/workspace",	
}

• CI/CD credentials and the place they are stored are unlikely to be the same between a developer's machine and a CI/CD agent/runner
• the Docker images may be built outside of Docker Compose, as opposed to inside

In the specific case of this Python AWS lambda function, keeping the same

Dockerfile

Dockerfile

Next use case to stare at? The advent of ARM as desktop platform, should the universe (and Apple) decide so. Being able to code/build/test on an ARM platform with all the local bells and whistles, but from a non-local, non-target hardware platform (some developer laptop) may come extremely handy.