Last week I gave myself the challenge to create something in a week. Not a mockup, but a complete usable solution with documentation and enough tests to make it feel solid. I have been looking at DigitalOcean for some time, and while browsing their documentation I came across a sample script that would help you set up multiple ‘droplets’ (DO speak for server instances) on a HAProxy.
I found the script quite useful, but it was quite cumbersome to use. It required Ruby, you still had to set up HAProxy, and according to the documentation “there are tons of error cases that aren’t being checked”. So this seemed like a good project to take on. My idea before starting was to simply port it to Go, and of course check and fix up all the corner cases, but as you can see, it turned out a bit differently. Furthermore this was interesting, since it was some time since I last had dived into the “http” part of Go.
Below is a breakdown of my work each day.
Work Log
Monday
Most time Monday I spent researching. This is my first reverse proxy, so I studied HAProxy, nginx, the existing `doproxy` script to work out which features would bring the most value.
I saw the manual aspect of `doproxy` as the point I wanted to make a difference. I would love to make a proxy that was able to provision and de-provision droplets automatically. I realized that to do that the most effective way, I would need to “be” the proxy, so I could react to query times and live metrics.
Another aspect that “annoyed” me with the current solutions was that most servers need a “soft” restart to apply settings, leading to various tricks to make the restart seamless. So I decided that another main feature should be hot configuration reload, as far as it was possible.
I then spent some time to research if there were any existing applications or libraries it would make sense to base my code on. After all, I don’t want to write something that already existed. However, beside some very simple reverse proxies I didn’t really find anything that would work.
Tuesday
I spent most of the day getting simple configuration determined and set up. I considered various configuration formats, but settled on TOML, which I have used previously, and had a slight preference to compared to YAML.
For configurations that are meant to be hand-edited to some extent, I tend to stay away from JSON, since there is no comments and the syntax is more complicated than necessary for something like this.
Other than configuration I got at basic reverse proxy setup as well as a framework for backends and load balancers.
At this point, I decided that I would leave out “provisioning” for the time being. While it is an important part of what will make this program great, it was not essential for the first version. It would still be possible to add this feature later, so in the interest of finishing something useful this week I put that on hold.
Wednesday
This was the first time I got the various elements “tied together”. A lot of time was spent trying to simplify the interfaces and still make it work.
I also implemented health checks, as well as round-trip timing, inventory saving and similar things.
Thursday
Once most features were tied together and mostly running, I started writing configuration tests as well as documenting how the parts fit together, as well as setting up Travis CI testing.
To give myself an idea of the work required, I also wrote configuration for the provisioning, since that gave me a good idea of what would be needed to be done to implement this.
Friday
The first part of today was spent creating an end-to-end test of the reverse proxy, with mocked parts in between. The rest of the day I have spent documenting and finishing up a “0.1.0” release.
This was mostly to shave of most of the rough edges for potential users, as well as getting rid of the worst inconsistencies.
What went right?
IntelliJ. This is the first project, where I exclusively used IntelliJ, and it was a great help. The golang plugin has really improved a lot, and even though it takes some time to set up (on each machine), the help it provides is significant. Especially the code aware renamer is a huge new feature.
Structs -> Interfaces. In many cases the interfaces started out as structs, that were later abstracted into interfaces, and embedding to embed common interface logic, so it doesn’t have to be duplicated between interface implementations.
Configuration first. I find that creating the configuration first helps me a lot, since it helps me keep the user perspective. If I write the configuration that would make sense to me as a user before implementing the code, it helps me keep the configuration sensible and un-bloated. Usually a few iterations are required, but it helps in the end.
Cut features. It was clear rather early, that the full scope of my ideas wasn’t feasible within a week. In the end, cutting the features meant that a usable product was finished this week, and not a half-complete, test- or documentation-less mess.
What went wrong?
Real world testing. This is still a big missing thing. Right now the code only has a “works on my machine” seal of approval, so that makes me a bit uncomfortable releasing it. But luckily I assume that any users of my program are good at helping in debugging.
Locks in subobjects. A thing that has come up on my “annoying list” is mutex-locking subobjects of objects. If a struct holds another struct, where a member requires a mutex lock, it is tricky to remember. The only “solution” is to have “getter” and “setter” functions, which I also dislike when done wrong.
One-package project? At various points I considered, if I should split some functionality into separate packages, but I haven’t found any good way, where it would result in a huge clump is microscopic packages, which IMO is much worse than a single package server. So my “package-sense” is tingling, but for now it has been told to shut up.
Conclusion
It has been a very fun project, and I will continue to work on it. Next on my list is adding automatic provisioning to the proxy and a lot of other features that have come up. I have put them on the project page. If you have any other requests, feel free to add them. However, remember, this is not meant to be a complete reverse proxy. You can still put another reverse proxy in from of it, if you need advanced features like URL-routing, etc.
You can follow further development on the Project Page: https://github.com/klauspost/doproxy