Kubernetes Validating Webhooks

As new engineers onboard to teams within my organization, one area that is commonly confusing is Kubernetes admission control. This confusion is due to a lack of clear documentation and examples. For my team, pairing is a great way to knowledge transfer these concepts but I would like to offer my take on developing these resources in a series of posts.

Within this post, I hope to describe what validating webhooks are, how they come in handy to customize your Kubernetes clusters and provide a simple, tested example to begin working to a deployable implementation. The default documentation from Kubernetes is available here and provides great context but little implementation.

So, what is a validating webhook? Put simply, a validating webhook is an endpoint Kubernetes can invoke prior to persisting resources in ETCD. This endpoint should return a structured response indicating whether the resource should be rejected or accepted and persisted to the datastore.

I look at validating webhooks very much like microservices in that they should fulfill a single purpose. I prefer to have many webhooks with simple validation purposes as opposed to one with all my validations crammed inside. This can become a maintenance nightmare as the next engineer attempts to update validation logic.

Simple Use Case

To demonstrate validating webhooks let’s assume we have a business requirement where all namespaces created in our cluster must contain a label indicating team ownership. If a label is not provided, we should reject the namespace creation with a meaningful message to allow an engineer to resubmit the request without outside intervention.

Valid Namespace

Invalid Namespace

Example Implementation

For this particular post we will leverage Golang as the language of choice for our endpoint. This is not a limitation as any language can be used to create the endpoint. This will be further clarified as we deploy this implementation. If you wish to quickly jump to the example code, it is available here.

Go application structure is hotly debated. In this example we will be colocating our handlers in the main.go. Personally, I believe this simplifies the implementation and gives a single file for other engineers to maintain.

To begin implementing this webhook I start with a simple happy path test, which defines the requirements we need to implement in a handler method. We must expose an endpoint that accepts a POST, with an AdmissionReview request payload. This AdmissionReview struct contains an AdmissionRequest which in turn contains the raw bytes of the resource we are validating.

Based on our assertions, you can see our implementation must return a nested AdmissionResponse struct with a bool flag communicating the validation result. In our happy path test, we assume this will be true. The Namespace and Metadata struct have been implemented in our main.go file along with skeleton implementations of the handler and main function.

Now for a negative case. Note we are not providing labels and should expect a validation response of false and a meaningful message describing the rejection reason. I also included several test cases for invalid request payloads you can view here.

So, now that we have a skeleton implementation and a bunch of failing tests let’s implement the meat of our handler. Typically, I would include more logging here around inputs and results but I will omit for clarity. Outside of parsing logic, the main responsibility of this function is to set the allowed flag based on our label checks. This occurs on line 64 to 69 below. Once this is complete our tests should pass and we can move on to implementing the base server for the handler.

Kubernetes requires that your custom admission webhook endpoints are served over HTTPS. This requires that we provide a certificate and key file to leverage the Go http.ListenAndServeTLS function. Typically, we generate a certificate signing request within the target cluster, retrieve the certificate and key to set as a secret and mount it in the appropriate path. I will cover deployment more in a future post.

I hope this initial post will help you to understand and implement your own validating webhook. There is much more to cover in deployment and functional testing which will be the focus of my next post.