Enabling HTTPS in Kubernetes with Cert Manager & Let’s Encrypt for free!

Enabling HTTPS in Kubernetes with Cert Manager? Why do we need HTTPS at all?

Not long ago, there weren’t many sites/servers on the web with the HTTPS implemented. Even browsers were not particularly “interested” in which kind of protocol was used to talk to a website. Nowadays, having HTTPS is a standard, and browsers will be very harsh when it comes to warning you that the site you are visiting is “not secure” if the protocol used is not HTTPS.

What does this mean for you as a client or a server owner? Would you like your important data (e.g., financial, personal, or “just” your tokens or session details) to be easily reachable to anyone? With plain HTTP, all the data between the client (e.g., browser) and the HTTP server is transmitted as plain text, which makes sniffing, “man in the middle,” and other attacks very possible, so your browser is right: you should not feel safe while using plain HTTP!

What does your server need for HTTPS? 

Two things are needed for all kinds of TLS server configurations:

A certificate issued by a trusted certificate authority (CA)

A private key (PK) paired with the certificate

Traditionally, you would purchase those things and then handle PK with care (just as any other secret). You would also need to pay attention to the expiration dates of certificates and renew them accordingly – this part typically requires manual intervention or some level of DIY automation. Backups and access rights are an issue too.

How is HTTPS enabled in Kubernetes Ingress?

Kubernetes (k8s) makes the task of exposing a service to an outside world very easy. You only need to create an Ingress object that will point to a service, which will point to a pod with your application. Below you can see what our sample application looks like in ArgoCD:

It’s a simple nginx deployment that contains the following:

A single pod

Service that points to that pod

Ingress object that points to that service

From the ArgoCD network perspective, it looks like this:

In this setup, we have an Ingress object containing a special TLS section that points to a k8s Secret that should contain our certificate and a private key mentioned above.

This Kubernetes Secret needs to exist and be populated with a valid, trusted certificate and a corresponding private key. In our example, you can see that we do not have the Secret – yet! If we had this Secret populated, our Ingress would handle HTTPS traffic, and that would be it – HTTPS would work! But, to populate the Secret, you will need to obtain a certificate and a private key from CA, which is not a trivial task if done manually.

Can issuing/renewing TLS certificates be automated?

Our goal is to automate obtaining and renewing certificates from CA and storing them in a Kubernetes Secret needed for Ingress HTTPS to work. The Cert Manager (CM) is a tool that will help us achieve this. This is how official documentation describes CM:

“The Cert-manager is a powerful and extensible X.509 certificate controller for Kubernetes and OpenShift workloads. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date and will attempt to renew certificates at a configured time before expiry.”

Therefore, the CM will obtain a certificate for you from an Issuer of your choice. In our case, we will use the increasingly popular issuer Let’s Encrypt(LE). To achieve this, the CM talks to the LE using the ACME protocol. The ACME protocol, among other things, specifies how ownership of a domain is proven to an Issuer. There are several mechanisms to do so, but in our case, we will use the HTTP-01 challenge. All of this, including resolving a challenge, will be done automatically for us by the CM.

Installing the Cert Manager in our k8s cluster

Official installation instructions from the CM site offer a few ways to install it in your cluster. You might use the provided static YAML-s or a Helm chart. Since we use the ArgoCD and GitOps principles, we will download the static file and put it in our GitOps repo, and the CM will be installed in our cluster automatically. If we take a look at the user interface of ArgoCD, we will see something like this (not everything fits the picture):

Many objects have been installed, but we need a few more things! So far, only the CM objects have been installed, but to teach our CM to talk to the LE, we need to create two more objects of a kind: ClusterIssuer, which will instruct the CM how to talk with the LE. 

But wait, why two objects? Well, the LE provides us with two Issuers:

Staging

Used for development, experiments, and initial setup. Unlike the production issuer, the staging issuer allows us to make many more requests to it. Still, it will issue an invalid certificate because the root CA of the certificate is not generally trusted.

Production

Should be used after we have proven that the staging issuer works. It will issue valid, trusted certificates. The number of requests for this one is limited.

The YAML-s for these two issuers should be deployed to our cluster too. The staging and production issuers are given respectively in the code below. The difference is in the name and the server URLs. The objects are of ClusterIssuer kind. As for the CRD-s, they are provided during the CM installation.

At this point, we have the following set up:

Application exposed to the web using k8s Ingress

Ingress pointing to a non-existent Secret that should contain a certificate and the Private Key

The Cert Manager configured with staging and production of the Let’s Encrypt Issuers

HTTPS certificate instruction

For HTTPS to work, we need one more thing: to instruct the CM to obtain a certificate for our Ingress. It is done by just putting appropriate annotation to our Ingress object. Please note that this annotation’s value matches the staging issuer name we created a step ago.

As soon as this happens, the CM will pick up the Ingress, see that it is pointing to a non-existent TLS secret, and start a challenge to create a Secret with a certificate and PK. It looks like this:

And if we scroll to the right, we will see even more objects:

You will notice that the Ingress object has a child object of a kind: the certificate and some other objects required to obtain the certificate. At this point, the challenge will be performed. During the challenge, a few more objects will be created, including a temporary pod, service, and its Ingress, which are needed to complete the challenge. Once the challenge is successfully done, all the temporary objects are removed from the cluster, and a secret containing a certificate is created. The challenge-solving temporary pod, service, and Ingress are marked in the previous picture.

Once all of it is completed successfully, feel free to change the issuer annotation in Ingress to point to the production LE issuer like this:

The issuance of a production certificate should happen now. Secret containing TLS data is recreated automatically containing production data:

and we have our HTTPS running!

If we examine the certificate, we will notice that it is valid for a few months only, but no worries! The CM will renew it when the time comes!

Final thoughts about enabling HTTPS in Kubernetes with Cert Manager

Well, this was not that simple, right? It might appear that many steps are needed to configure all of this, but please note that most of the steps are either one-time (installation of the CM and the Issuers) or automated (the creation of objects needed to complete the challenge). 

Once we have the Cert Manager and the Issuers configured, the only thing needed to obtain and maintain certificates is to put a single annotation on an Ingress object once! Can it be simpler than that? 

So, to summarize:

The Cert Manager installation in k8s is quite straightforward.

The Cert Manager documentation is very detailed and well-organized. 

It requires no storage (PV) nor a database of any kind. It is a k8s-friendly application that keeps its state in its Kubernetes objects (in etcd). It basically means you don’t have to worry about losing data or disaster recovery. All certificates can be reissued easily, and you do not need to do a thing for it to happen. Without automation, you would have to keep a backup of your Private Key and a certificate.

No need to worry about certificate expiration.

The private key is created by the cluster and stored in it, so it will never leave it, which increases security.

Most browsers and technologies, including Java language and many others, trust the certificates issued by the LE.

All of the tech mentioned above is free, including certificates!

Bonus tips

The Cert Manager is Prometheus-friendly. It exposes important metrics, including the expiration of certificates, so alerts can be created just in case something goes wrong with certificate renewal. There is a nice Grafana dashboard out there that can display useful metrics: https://grafana.com/grafana/dashboards/11001-cert-manager/.

For debugging an issuing process, you may find reasons for problems if they occur (e.g., some network policies are in the way of solving a challenge) by paying attention to the statuses and events of the following k8s objects:

challenges.acme.cert-manager.io

orders.acme.cert-manager.io

certificaterequests.cert-manager.io

certificates.cert-manager.io

A temporary pod is created by the Cert Manager to receive the HTTP–01 challenge and is destroyed by the CM when the challenge is finished. An existing network policy may prevent the HTTP–01 challenge from being solved by preventing Ingress traffic from being routed to the temporary pod. For this to be resolved, a special network policy needs to be created to allow traffic to the pod. 

It is possible to have more control of the fields of the certificate and other options using annotations: https://cert-manager.io/docs/usage/ingress/#supported-annotations