Localizer: An adventure in creating a reverse tunnel/tunnel manager for Kubernetes

Be sure to subscribe to my blog for more content.

Before we get into the details of what localizer is and how it came to be, it’s crucial that we look at what developer environments were and the motivations behind the ones I create.

What is the Goal of a Developer Environment?

Scaling Up

Introducing Kubernetes

While at Azuqua, I identified the need to run Kubernetes to ease cloud resources’ scalability and improve the developer experience. At the same time, this drastically decreased the developer experience. While this may seem contradictory at first, it’s essential to consider the multiple ways that developer experience presents itself. While at first, it may seem like it’s just tooling to test/write code, it’s a combination of testing code, writing code, and deploying that code. With the Docker Compose environment at StayMarta, we made the test/build cycle incredibly simple but shifted the deploy aspects onto a bespoke team, the DevOps model. That approach works for small teams, but as you grow, this quickly doesn’t scale. If you can’t deploy code efficiently, the developer experience is frustrating and promptly turns to an unhealthy relationship with the team responsible for that cycle.

So, how exactly does Kubernetes make this better then?

The ability to reproducibly deploy helps both deployment confidence and the amount of time needed to get from nothing to running in production. However, it’s not without its faults. Whenever you start moving deployment tooling onto developers, you’ve decreased the developer experience. It’s unavoidable because you’re introducing net new materials, DSLs, and more for the developers to learn. While KinD and minikube are great projects, they all suffer from needing developers to understand how to work with Kubernetes. You need to be able to build a Docker image, push the docker image into the cluster, delete the application pod, verify if it’s even using the correct tag, wait for Kubernetes to recreate the container, and make sure you’ve configured an ingress route to access it outside of your cluster or use kubectl port-forward to access it. The second that breaks, you're now required to dig into why service connectivity isn't working, why your Docker image isn't in the containerd cache, or other not so easily solved areas. While to someone who's worked with Kubernetes for years now, this isn't very difficult, this is hardly achieving the "ease of use" goal I have.

How do we make developing on a Kubernetes developer environment easier?

[…] an open source tool that lets you run a single service locally, while connecting that service to a remote Kubernetes cluster.

https://www.telepresence.io/discussion/overview

It seemed perfect, a tool that enabled your local machine to act as if it were running in Kubernetes and allow your services to be targeted from inside of Kubernetes. We rolled it out at Azuqua after initial tests showed it worked well, and we went with it. Unfortunately, it didn’t last.

The Problem With Current Local Service Development Tooling

When I started developing a new development environment for Outreach, I again tried to use Telepresence as the solution to bridge the gap between our local cluster and our local machine. Unfortunately, it didn’t work. We did not have a well-defined service discovery mechanism to work around the bugs, we had a much larger engineering team, and almost all services needed to talk to each other. We found more and more edge cases with Telepresence, and our developer experience was suffering. Our NPS score for our developer environment was at a low of -26, just because of Telepresence!

For comparison, a good NPS is generally +30…

It was pretty clear that Telepresence was not going to solve our use-cases. It was time to look into alternatives.

What are other alternatives out there?

Introducing Localizer

Default: Creating Tunnels

Running localizer on my home Kubernetes Cluster (it's not limited to just local developer environments)

When you run localizer without any arguments, it automatically creates port-forwards for all Kubernetes Services (this is important), a loopback-interface powered IP address and adds them to /etc/hosts. Out of the box, this allows code to communicate with Kubernetes services despite being outside of the cluster.

Expose: Creating a Reverse Tunnel

List: Listing the status of the tunnel(s)

Running localizer list on my home cluster (Hi Ghost 👋)

The arguably most useful command that localizer provides is list. It enables developers to view the various tunnels running and provides insight into the health and status.

How Localizer Solved Our Problems (sort of)

The lack of abstractions is a fundamental problem for Kubernetes right now, and I look forward to writing tooling that can help developers focus on the business problems and not waste time on the details.

Looking to the Future

Special Thanks: Mark Lee for editing!

Originally published at https://blog.jaredallard.me on December 4, 2020.

Software Engineer/SRE Hybrid obsessed with containers, Kubernetes, and everything in-between.