How to Get the Most Out of Heroku CI

by Tyler HawkinsMay 5th, 2020

Too Long; Didn't Read

Heroku offers a service called Heroku CI for developers hosting and deploying their code through Heroku. Heroku allows developers to run test suites and other jobs on each pull request created in their projects. These jobs must pass before merging the code changes into the master branch. There are many great tools for creating CI/CD pipelines, such as Travis CI and GitLab. In this article, we’ll go through the basic setup for getting up and running with Heroku’s Heroku service.

Companies Mentioned

Coin Mentioned

featured image - How to Get the Most Out of Heroku CI

Continuous integration and continuous delivery (CI/CD) are best practices in today’s software engineering development process.

Continuous integration (CI) allows developers to automate running test suites and other jobs on each pull request created in their projects. These jobs must pass before merging the code changes into the master branch. This creates confidence in the master version of the code and ensures that one developer doesn’t break things for every other developer working out of the same codebase.

Continuous deployment (CD) facilitates deploying changes into production immediately when new code is merged into the master branch. Gone are the days of only releasing code once per quarter, per month, or per week. By releasing code early and often, developers can deliver value to their customers at a faster pace. This strategy also makes it easier to identify issues in production and to pinpoint which commit introduced them.

There are many great tools for creating CI/CD pipelines. is a popular open-source tool, and even comes with its own CI/CD features. offers a service called , which makes it a viable choice for developers already hosting and deploying their code through Heroku.In this article, we’ll go through the basic setup for getting up and running with Heroku CI, and then explore some advanced features like parallel test runs and automated browser tests.

Demo App

For this article, I’ve created a pun generator app. Dads everywhere, unite! The app is incredibly straightforward: With a click of a button, the app outputs a dad joke on the screen. To keep the code simple, I’ve created it with plain HTML and vanilla JS. The frontend is served by a Node.js and Express server.

Pun generator app

You can find all of the .

Test Setup

To help bootstrap my app, I cloned the example Node.js app from Heroku in their . I then wrote some HTML and added some JavaScript to handle the button click and the pun generation. I chose as my unit testing framework, and I wrote tests using Kent Dodds’ .

I added an NPM script so that I can run my tests by entering the command

npm test

in my terminal. Running my tests locally generates output that looks like the following:

Test output

Basic CI Setup

Now that I have a test suite that I can run locally, I thought it would be nice if I could have it run each time I have new code to merge into my master branch. A CI/CD pipeline can automate that for me! The explain the setup in greater detail, so I’d recommend following the instructions found there, but here are the basic steps I followed:

Pushed my code to a repo in GitHub
Created a Heroku app for that repo
Created a Heroku pipeline
Connected the pipeline to my GitHub repo
Enabled Heroku CI in the pipeline settings (In order to do this, you have to provide a credit card, because Heroku CI does come with for using it.)

Fairly easy! Next, I created a new branch in my repo, added some new code, pushed it to that branch, and then opened up a pull request to merge my new branch into the master branch.

This is where the magic happens. At this point, I could see a section in my pull request in GitHub showing “checks” that need to pass. These “checks” are jobs running in the CI pipeline. In the screenshot below, you should notice the job for

continuous-integration/heroku

GitHub pull request waiting on Heroku CI pipeline to pass

When I then hopped over to the Heroku pipeline dashboard, I could view the progress of the job as it ran my tests:

Heroku CI job in progress

Once the job finished, I could then see a green checkmark back in GitHub as shown in the screenshot below:

GitHub pull request after Heroku CI pipeline passes

Now, I could merge my branch into the master branch with confidence. All the tests were passing, as verified by my Heroku CI pipeline.

Requiring Checks to Pass in GitHub

As a side note, you should notice in my GitHub screenshot above that the

continuous-integration/heroku

check is required to pass. By default, checks are not required to pass. Therefore, if you’d like to enforce passing checks, you can set that up in the settings for your specific repo.

Require status checks to pass before merging

Parallel Test Runs

Now that we’ve covered the basic setup for getting started with Heroku CI, let’s consider a more advanced scenario: What if you have a large test suite that takes awhile to run? For organizations that have a large code base and have been writing tests for a long time, it’s common to see a test suite take 5–10 minutes to run. Some test suites take more than an hour to run! That’s a lot of time to wait for feedback and to merge your code.CI pipelines should be quick so that they are painless to run. If you do have a large test suite, Heroku CI offers the ability to across multiple dynos. By running your tests in parallel, you can significantly cut down the time it takes to run the whole suite.

To set up parallel test runs, all you need to do is specify in your

app.json

file the

quantity

of dynos you want to run. I chose to use just two dynos, but you can use as many as you want! You can also specify the

size

of the dynos you use.

By default, your tests run on a single "performance-m" dyno, but you can increase or decrease the size of the dyno if you're trying to control costs. In my case, I chose the smallest dyno that Heroku CI supports, which is the "standard-1x" size.

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 2,
          "size": "standard-1x"
        }
      }
    }
  }
}

Now, when I added new code and created a new pull request, I could see my Heroku CI job was running on two dynos. For my tiny test suite of only three unit tests, this was definitely overkill. However, this kind of setup would be extremely useful for a larger time-consuming test suite. It’s important to note that , so make sure the test runner you choose for your app includes this capability.

Two dynos being used during a Heroku CI job

Automated Browser Tests with Cypress

In addition to running unit tests, you may also want to run integration tests and end-to-end tests on your app. and are popular end-to-end test frameworks, both of which are industry standard. The nice thing about Cypress for frontend developers is that you write your tests in JavaScript, so you don’t need to learn Java like you would for Selenium.Let’s take a look at how we could configure Cypress to run a few end-to-end tests on the pun generator app and then include those tests in our Heroku CI pipeline.First, I installed a few necessary dependencies by running

npm install --save-dev cypress cross-env start-server-and-test

Second, I added some more NPM scripts in my

package.json

file so that it looked like this:

"scripts": {
  "cypress:open": "cypress open",
  "cypress:run": "cypress run --browser chrome",
  "cypress:test": "start-server-and-test start //localhost:5000 cypress:run",
  "start": "cross-env PORT=5000 node index.js",
  "test": "jest"
},

Third, I wrote a small Cypress test suite to test that the button in my app works correctly:

describe('Pun Generator App', () => {
  beforeEach(() => {
    cy.visit('//localhost:5000')
  })

  it('adds a dad joke to the page when the button is clicked', () => {
    cy.contains('Click me for a terrible pun').click()
    cy.contains('My ceiling isn’t the best, but it’s up there.')
  })

  it('tells the user when it is all out of puns', () => {
    const button = cy.contains('Click me for a terrible pun')
    for (let i = 0; i < 20; i++) {
      button.click()
    }
    cy.contains("I'm all out of puns!")
  })
})

I could now run

npm run cypress:test

locally to verify that my Cypress setup is working properly and that my end-to-end tests pass.

Finally, I modified my

app.json

file to include a new test script and appropriate buildpacks for Heroku CI to use. It's important to note that for JavaScript apps, Heroku CI uses the

npm test

command. If you don't specify a test script in the

app.json

file, then Heroku CI will just use the test script specified in your

package.json

file.

However, I wanted Heroku CI to use a custom script that ran both Jest and Cypress as part of the test, so I wrote an override test script in

app.json

{
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 1,
          "size": "standard-1x"
        }
      },
      "buildpacks": [
        { "url": "//github.com/heroku/heroku-buildpack-google-chrome" },
        { "url": "heroku/nodejs" }
      ],
      "scripts": {
        "test": "npm test && npx cypress install && npm run cypress:test"
      }
    }
  }
}

Unfortunately, I hit a snag on this last step. After several hours of reading, researching, and troubleshooting, I discovered that . The recommend using the

--headless

option rather than the deprecated default

Xvfb

option.

However, while running Cypress inside of the Heroku CI pipeline, it still tries to use

Xvfb

. Using previous versions of Cypress and older (and deprecated) Heroku stacks like "cedar-14" yielded no better results.

It would appear that either Heroku or Cypress (or both) have some issues to address! Hopefully users running end-to-end tests with Selenium fare better than I did when trying to use Cypress.

Other Heroku CI Features

Now that we’ve discussed two of the main features, running tests in parallel and running browser tests, let’s briefly look at a few other features of Heroku CI.

In-Dyno Databases

If your application relies on a database, then you’ll likely need to use that database during testing. Heroku CI offers , which are databases that are created inside of your test dynos during the CI pipeline test. These databases are ephemeral. This means that they only exist for the duration of the test run, and they’re much faster than a normal production-ready database because the database queries don’t pass over the network. These two benefits help your test suites to finish quicker, which speeds up your feedback loop and keeps your costs down.

Environment Variables

If you need to specify any non-confidential , you can add them to your

app.json

file like so:

{
  "environments": {
    "test": {
      "env": {
        "NON_SECRET_VARIABLE": "abcd1234"
      }
    }
  }
}

You would typically place private secrets in an

.env

file which you tell Git to ignore so that isn't checked into your source control. That way you aren't storing those values in your repo.

Heroku CI adheres to this same principle by allowing you to store private environment variables directly in the Heroku CI Pipeline Dashboard rather than exposing them in the

app.json

file.

Debugging the CI Process

If you are running into issues while setting up your Heroku CI pipeline, you can use the

heroku ci:debug

command directly in your terminal to create a test run based on your project's last local commit. This allows you to inspect the CI environment and gives you greater insight into possible test setup problems.

This command is especially helpful if you know that your tests are passing outside of the Heroku CI environment but failing when run in the Heroku CI pipeline. In this case, the issue likely lies in the CI setup itself.

Limitations

Although Heroku CI has a lot to offer, it does have some limitations. First, unlike other CI/CD tools such as Travis CI that are platform agnostic, you must host your app on Heroku dynos and use Heroku Pipelines in order to use Heroku CI. If you’re already a Heroku user, this of course isn’t a problem, and is actually a great benefit, because testing with Heroku CI is about as close as you can get to modeling a production environment for apps deployed through Heroku. However, it does mean that users of other platforms won’t be able to consider switching to Heroku CI without moving a lot of their other infrastructure to Heroku.Second, as mentioned above during my browser testing experiment, Heroku CI doesn’t currently seem to be compatible with Cypress.Third, .For other limitations, you can consult .

Conclusion

By now you should be comfortable with the basics of Heroku CI and understand some of the advanced features as well. For further questions, you can always .Once you’ve chosen your test tools and ensured their compatibility with Heroku CI, getting up and running should be a breeze. With Heroku CI you can create a software development system that enables high confidence and extreme productivity.And now, without further ado, here are some more puns from our app: