EXPEDIA GROUP TECHNOLOGY — SOFTWARE
Mittens — Warming up Your Application
Mittens is an open-source tool that can be used as a warm-up routine for HTTP applications over both REST and gRPC
At Hotels.com™, part of Expedia Group™, we deploy our services on Kubernetes. Many of those services are deployed to production multiple times per week while Kubernetes restarts pods and applications on a regular basis.
When an application starts as part of a deploy, redeploy, or restart, the very first requests are expected to be slow.
Although there are many reasons that contribute to this and to various extents (e.g. class loading, lazy initialization of connection pools and thread pools, SSL handshake), having a warmup routine that sends dummy requests before the application receives any traffic is a common practice to reduce the impact of the so called “cold-start” issue.
Despite the fact that many tools exist for making either HTTP/REST or gRPC calls most of them only support REST, and it’s not trivial to run them on Kubernetes or as linked containers.
What is Mittens?
Mittens is a tool that can be used as a warm-up routine for HTTP applications over both REST and gRPC.
Its main features are summarised below:
- Sends requests continuously for a predefined amount of time
- Supports REST and gRPC
- Supports concurrent requests
- Supports placeholders for random elements in requests
- Provides files or/and endpoints that can be used as liveness/readiness probes in Kubernetes
You can run it as a standalone command-line tool, as a linked Docker container, or even as a sidecar on Kubernetes.
Implementation and how to use
Mittens is written in Go. It uses default libraries for HTTP calls and makes use of reflection and of the grpcurl tool for gRPC calls.
The application receives a number of command-line flags. You can read the documentation for more context.
Mittens has been open-sourced under the Apache 2.0 License.
How we use Mittens at Hotels.com and Expedia Group?
Mittens run as a sidecar container next to the main container in our application pods. It usually runs for a time-window between 1 and 2 minutes.
A useful feature for many application owners has been the placeholders for random elements. This can be used to test the same endpoint with different combinations and values for parameters, hence avoiding cache hits. Another common use case for applications serving a booking website is to be able to generate and send requests to urls that include valid dates.
It is worth mentioning that a successful run of the tool is not a requirement for our applications to start. This decision has been taken by having the following in mind:
- The primary goal of Mittens is to mitigate latency issues for the first requests and not to verify that the application is ready/healthy.
- Sidecars are not yet first-class citizens, at least not in our Kubernetes version (this feature is expected to be released soon). This means that we don’t have a proper sidecar container lifecycle that would take care of starting and stopping the warmup container.
Do I need to be on Kubernetes to use it?
Being on Kubernetes is not a requirement for running the tool. As long as your services are containerised you can run Mittens as a linked container. In fact, this is how we used to run it before moving to Kubernetes! You can also run it as a standalone cmd application, for testing purposes.
Credits
The initial implementation of Mittens was written by Daniel Albuquerque and Andy Stanton. When I joined the team and we started working towards open-sourcing it, none of us had much experience with Go. Peter Reisinger’s help was invaluable as he brought the project into its current shape.
