Origin story
Most modern companies rely on a whole host of technologies and tools to run their operations, and we are no different. Our stack includes CircleCI, GitHub and GitHub Actions, GCP, Kubernetes, Jira, and much more. Since our feature rollout process involves a wide range of tools working in tandem, we needed a solution that could talk to all of them, from posting a pull request to deploying to production. That’s when our engineers had a thought: if we’re already spending most of our day in Slack, why not manage our pipeline and deployments there too?
And so “Bender the Great” was born. It’s our in-house automated CI/CD tool that makes our lives extremely easy and connects to every tool we need to get the job done.
So what is Bender?
Bender started life based on an open-source GitHub bot called Hubot, connected to Slack through their legacy API. We recently migrated to a Slack app using Bolt for JavaScript, and now it hangs out with us in Slack while we chat away and get things done. It’s connected to the various tools in our stack either through their webhooks or public APIs. Bender’s memory is a datastore on GCP that manages all the features we’re about to walk through.
We have a dedicated repository for Bender’s code, which the team maintains and contributes to.
Benefits of an automated CI/CD
Without a system like this in place, developers would need to carry out most of these tasks manually. First, the team would have to agree on how to communicate when a PR is ready for review and track any feedback or comments. Notifications would often get missed, and the back-and-forth of commenting could drag the process out. Once the PR is merged and the test suite is triggered, the developer would need to monitor the pipeline by hand, and once it passes, likely run a CLI command to deploy or upload the build image into the Kubernetes cluster. And at every stage, the Jira ticket status would need to be updated manually.
In our case, all of those pain points are already handled by Bender. Communication around PRs is posted to Slack automatically, with the relevant people mentioned for quicker responses. GitHub Actions, together with Bender’s queue system, ensures each feature is thoroughly tested and deployed to staging for manual testing. From there, deploying to production is as simple as sending a public command to Bender in Slack.
So let’s walk through a typical deployment of a frontend service feature. On average, it takes about 35 minutes to reach production:
- PR approval: 5–15 minutes, depending on the size of the PR and whether there are comments to address
- GitHub image build: 8–10 minutes, depending on which service is being deployed
- E2E suite run: 15–20 minutes, depending on whether there’s a failure and a re-run is needed
During those 35 minutes, the only actions the developer takes are merging the PR and deploying to production.
Another major benefit of running our own automated system is the ability to override certain steps when needed to fast-track a deployment.
Now let’s take a closer look at what our process looks like in detail.
Posting a PR
Our journey starts with a PR. There are a lot of ways companies communicate that PRs are ready for review and manage the comments and changes needed for approval.
We have a dedicated Slack channel where we post PRs that are ready for review, tagging the relevant team members to check them. The fun begins when someone comments on the posted PR. Since Bender is connected to GitHub webhooks and keeps a reference to the associated Slack thread, it’s able to (but not limited to):
- Ping the PR creator when someone comments
- Ping the commenter if someone uses their GitHub handle (we keep a map between GitHub and Slack handles)
- Ping someone (or a team) if they were requested for review
- Post GitHub Action failures with links to the workflow
By posting a PR to the code review channel, you instantly get a way to communicate with the team. You’ll see if any actions failed that block the merge, and you’ll know when the PR is merged, thanks to dedicated emojis. This ties in with our GitHub CODEOWNERS setup, where Bender can ping assigned teams even if they weren’t explicitly tagged, ensuring the right people are always up to date on changes made within their scope, even when those changes come from another team.
Another neat feature is that Bender is connected to Jira. When you create a PR with a valid ticket ID in the title, it picks it up and moves the ticket to the “Code review” stage. The same applies to all the next steps in the flow.
When the PR is approved, Bender tags the author in the PR thread to let them know.
To tie the whole flow together, here’s the journey a PR takes across our services.
Changing E2E code
Changes to the E2E code are a bit special: they don’t trigger a service deployment but get picked up once a new E2E run starts. We post a message with the author in the deployment channel so everyone can track the changes, include a link to the commit with the squash message title, and immediately move the referenced Jira ticket to closed.
Deployment to dev and staging environments
We also have a dedicated channel for the deployment statuses of the different services, where we can send commands to Bender to manage them. Our services have three environments:
- dev: where we run our full E2E suite
- staging: where we then do manual QA on our features
- production
Each environment has an alphabet emoji assigned to it, which gets added to the different threads to indicate where the action took place.
When a PR is merged, we have a dedicated “Bender” step in the release workflow, so the PR creator gets pinged that the service is now deployed or added to the deployment queue. Whenever the deployment pipeline is clear, Bender picks up the currently available services from the queue and triggers CircleCI to run the E2E suite based on the deployed services. If the E2Es don’t pass on the first run, we get pinged about the failed scenarios and Bender manages the restart of the failures.
Once a service is deployed, Bender fetches the diff of the service release, finds all the tickets that were part of the deployment, and advances them to the “On dev” stage.
Once the E2E run succeeds, Bender moves all the deployed services to the staging environment for manual testing, then picks up the next batch of available services to deploy to dev ♻️. After each service deployment to staging, the Jira tickets found are advanced to the “On staging” stage.
Our flow here starts after the PR is merged: the image build GitHub Action finishes, and we call the Bender API to start the deployment.
Gathering approvals and releasing to production
We don’t do automated releases. We prefer people to manage their own services and features, and Bender helps a lot with that. Since Bender can generate diffs between environments, we use that capability when preparing for production releases.
It starts with a diff thread in the deployment channel, then asking Bender whether the service is ready to be deployed. Bender finds the diff between the production and staging releases and builds a Slack reply tagging everyone with commits waiting to be deployed. A number emoji from 1️⃣ to 9️⃣ is added to the message, indicating how many approvals are needed before the deployment. The people with changes then add a 👍 to that same message, and when the number of 👍 matches the number emoji, Bender pings the author of the thread that the service is ready to be deployed.
With another quick command to Bender, we can deploy the service(s) to production 🚀. Bender goes through the services one by one, deploying each and tagging the authors who had changes in the success message. And just like with the other environments, once the deployment is done, Bender advances the Jira tickets to “On production.” From there, another Jira automation kicks in to close the parent tickets if needed.
Keeping track of history
Another important part of any deployment is the ability to roll back a failure. Bender is connected to the Kubernetes cluster and can pull the last 10 deployed versions of a service. We can then ask Bender to roll back to a specific version, and it handles the deployment of that image again. This applies to all three environments.
Quick environment variable overrides
Sometimes we need to adjust environment variables set in the charts, but we don’t want to go through the entire PR → approval → deployment-to-three-envs process. For this case, we have a command where Bender can set the environment variable for a service and restart it with the new values, without touching any code.
The downside is that we now have two sources of truth: the original value in the values.yaml in code, and the value in Bender if it exists. To manage that, we run regular cleanups where we set the new permanent values in code and remove them from the Bender repository.
Final thoughts
There’s so much more to Bender than what we’ve covered here, enough to fill several more posts. It’s been a real time-saver in ways I couldn’t have imagined when I first joined the team. Having a single tool connected to all the technologies we use, and able to take action across them, makes developers’ lives significantly easier and lets us focus on what really matters: delivering features with quality and speed to our customers.
