Exposing Kubernetes Services via Ingress: Part 2

Prerequisites

Before getting started, here are some prerequisites.

• Kubernetes Cluster
• Cert-Manager Installed (follow Part 1)
• Domain name
• IP Pointing to Domain name (Assigned to the Contour Ingress Controller)

Introduction

Secure Sockets Layer (SSL) is an effective way to secure our sites from malicious actors. Having an SSL cert means that attackers cannot snoop into the communication between the server and the client rendering traditional Man in the Middle (MiTM) attacks useless.

Lets Encrypt

Nowadays, many SSL vendors in the wild provide SSL certs at different prices. One particular vendor is Lets Encrypt, managed by the Electronic Frontier Foundation. They provide free SSL certificates to anyone with a domain name and a public IP address that can actually be verified.

Once you verify the details, you can acquire an SSL Certificate for free. The maximum duration of an SSL cert is 90 days (3 months), after which you need to renew it to keep it active.

They also provide automated mechanisms to help you renew your certificate without worrying about it. Let's start configuring our certificate.

For Certificate Provisioning, we'll use cert-manager by Jetstack. It's a CNCF project specifically developed to be used in Kubernetes Environments. It can provision certificates from Let's Encrypt (free) and other paid certificate issuers like Sectigo, Digicert, etc.

Installing & Configuring Cert Manager

To install cert-manager, we'll use Helm Charts. Run the commands below to add the Helm repo for cert-manager and install it.

1. Add Helm Repository

helm repo add jetstack https://charts.jetstack.io

2. Update the Helm Repository

helm repo update

3. Install Cert-Manager

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.9.1 \
  --set installCRDs=true

Once it's installed, wait for the pods to transition to a running state.

k get po -n cert-manager

NAME                                      READY   STATUS    RESTARTS      AGE
cert-manager-55658cdf68-xd6bh             1/1     Running   0  		22h
cert-manager-cainjector-967788869-rj4gm   1/1     Running   0  		29h
cert-manager-webhook-6668fbb57d-hzvrt     1/1     Running   0  		22h

Provisioning Certificate

We'll use a wildcard certificate for our domain. There are two options to set the issuer. They are:

a. ClusterIssuer (Available All Over the Cluster)

ClusterIssuer can be accessed from anywhere within the Cluster. It's namespace agnostic, meaning Certificates can be generated in any namespace using the Cluster Issuer.

b. Issuer (Available Only in the Deployed Namespace)

An issuer can only be accessed from the namespace it is deployed in, issuer for namespace A cannot be accessed from namespace B, C, etc.

I'll go with the ClusterIssuer as I'll deploy different applications over different namespaces using a Wild Card SSL Certificate, so having a single Issuer satisfies my requirements.

To get a wildcard certificate, we need to perform DNS validation for the domain we choose to issue a certificate for. Each cloud provider provides a specific way of verifying the domain, the steps provided below are generic and may work for most of them.

Setting up the Certificate

Step 1: Create an API Token and Put It in a Variable

export token="some_token_123.abc"

Step 2: Base64 Encode the Token

echo $token | base64

Step 3: Create a Secret Object in the Cert-Manager Namespace

apiVersion: v1
kind: Secret
metadata:
  name: dns-verification
data:
  access-token: $(replace everything after the dollar with base64 encoded token)
type: Opaque

Now let's create a ClusterIssuer.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: wildcard-certificate
spec:
  acme:
  	#Using the staging server will help avoid lockouts after too many cert generation attempts
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: certificate@example.com
    privateKeySecretRef:
      name: private-key-wildcard-certificate
    solvers:
      - dns01:
          cloudprovider:
            tokenSecretRef:
              name: dns-verification
              key: access-token

Now create the Certificate.

apiVersion: cert-manager.k8s.io/v1alpha2
kind: Certificate
metadata:
  name: wildcard-certificate
spec:
  secretName: dns-verification
  issuerRef:
    name: wildcard-certificate
    kind: ClusterIssuer
  commonName: "*.example.com"
  dnsNames:
    - example.com
    - "*.example.com"
  acme:
    config:
      - dns01:
          provider: cloudprovider
        domains:
          - example.com
          - "*.example.com"

Routing Traffic to the httpd Deployment

We created the httpd deployment in the last article. We'll now make it available through https as well.

To turn the HTTP deployment to https, simply add the tls secret entry into the ingress manifest like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: contour
  name: httpd
  #Ingress Namespace
  namespace: default
spec:
  tls:
  - hosts:
    - httpd.example.com
    secretName: wildcard-certificate
  rules:
  #Provide name of domain here
  - host: httpd.example.com
    http:
      paths:
      - backend:
          service:
          	#Provide your service name here
            name: httpd-service
            #Provide your service port here
            port:
              number: 80
        path: /
        pathType: Prefix

Let's check the status of the ingress. We should see the https port show up as well.

NAME     CLASS    HOSTS                        ADDRESS        PORTS     AGE
httpds   <none>   httpds.example.com   192.168.22.9   80, 443   1d

Now let's run a curl command to check.

curl https://httpds.example.com

curl https://httpds.wildcard.example.com

*   Trying 192.168.22.9:80...
* Connected to httpd.wildcard.example.com (192.168.22.9) port 80 (#0)
> GET / HTTP/2
> Host: httpd.wildcard.example.com
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/2 200 OK
< date: Fri, 09 Sep 2022 09:01:15 GMT
< server: envoy
< last-modified: Mon, 11 Jun 2007 18:53:14 GMT
< etag: "8a-412a5v8a73a81"
< accept-ranges: bytes
< content-length: 45
< content-type: text/html
< x-envoy-upstream-service-time: 3
< vary: Accept-Encoding
<
<html><body><h1>It works!</h1></body></html>
* Connection #0 to host httpds.wildcard.example.com left intact

Troubleshooting

Rate Limit

If the certificate keeps getting requested for a similar domain name more than 5 times a week, then we won't be able to request a certificate for a whole week. Please use the staging server to generate the certificate first, and once successful, use the production server.

(This is applicable for the production Let's Encrypt certificate)

Events:
  Type     Reason     Age                   From          Message
  ----     ------     ----                  ----          -------
  Normal   Issuing    7m30s (x27 over 17h)  cert-manager  Issuing certificate as Secret was previously issued by ClusterIssuer.cert-manager.io/example-wildcard-certificate
  Normal   Reused     7m30s (x27 over 17h)  cert-manager  Reusing private key stored in existing Secret resource "dns-verification"
  Warning  Failed     7m30s                 cert-manager  The certificate request has failed to complete and will be retried: Failed to wait for order resource "example-wildcard-certificate" to become ready: order is in "errored" state: Failed to create Order: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: *.example.com: see https://letsencrypt.org/docs/duplicate-certificate-limit/

There's no way to bypass this, but a workaround exists where you can add another DNS name during certificate generation. But make sure that the DNS name is a derivative of the domain you are issuing the certificate for.

Wrapping Up

We learned to provision certificates for our applications deployed in Kubernetes using cert-manager. We also learned about a recurring problem during certificate generation and its workaround.

Hope you learned from this article. Stay tuned for more on Ingress Controllers.