How to Make Istio SSL Offloading Work with Nginx

In many corporate system infrastructures, it’s very important for the information to be encrypted end-to-end, to be protected from potential vulnerabilities. We’ve learned from our experience that creating a fully secure setup is essential. The main part of the diagram that we will focus on today will be the traffic going from the Nginx proxy to Istio’s HTTPS port. Keep in mind that, even if it’s not compulsory to have a full HTTPS connection between Nginx and Istio, there are applications that won’t work if you don’t use SSL offloading in front (Keycloak, for example).

Getting started with Istio

In order to set everything up, we will use the setup from our previous article regarding local development environments (be sure to have at least 4GB of RAM configured).

Installing Istio

Let’s download Istio using the official docs (the latest Istio version as of writing this article is 1.6.5):

$ curl -L https://istio.io/downloadIstio | sh -

$ sudo mv istio-1.6.5/bin/istioctl /usr/local/bin/

Now let’s install it on our cluster:

$ istioctl install --set addonComponents.prometheus.enabled=false --set values.gateways.istio-ingressgateway.type=NodePort

We don’t need Prometheus for this experiment. Additionally, we’ve chosen Node Port because Load Balancers are not available on on-premise setups.

Creating the experiment

Web Server

Let’s create an Nginx pod to test our HTTPS setup:

$ kubectl run nginx --image=nginx

and create a service for it:

$ kubectl expose pod nginx --port=80 --target-port=80

Check that everything is OK:

$ kubectl get po,svc
NAME        READY   STATUS    RESTARTS   AGE
pod/nginx   1/1     Running   0          46s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.152.183.1     <none>        443/TCP   17h
service/nginx        ClusterIP   10.152.183.116   <none>        80/TCP    19s

Certificate and key

Next, we need to create a certificate and key pair to use for SSL offloading:

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=ssltest/CN=ssltest.com' -keyout ssltest.com.key -out ssltest.com.crt

$ openssl req -out www.ssltest.com.csr -newkey rsa:2048 -nodes -keyout www.ssltest.com.key -subj "/CN=www.ssltest.com/O=ssltest"

$ openssl x509 -req -days 365 -CA ssltest.com.crt -CAkey ssltest.com.key -set_serial 0 -in www.ssltest.com.csr -out www.ssltest.com.crt

Create the secret:

$ kubectl create -n istio-system secret tls ssltest-credential --key=www.ssltest.com.key --cert=www.ssltest.com.crt

Gateway and VirtualService

Create the Gateway:

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-gw
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: ssltest-credential
    hosts:
    - www.ssltest.com
EOF

Create the VirtualService:

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-vs
spec:
  hosts:
  - "www.ssltest.com"
  gateways:
  - nginx-gw
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 80
        host: nginx.default.svc.cluster.local
EOF

Install Nginx on the Virtual Machine

Install Nginx and jq using apt:

$ sudo apt update && sudo apt install nginx jq -y

Get the Node Port which Kubernetes has assigned to the Istio HTTPS port:

$ kubectl -n istio-system get svc istio-ingressgateway -ojson | jq .spec.ports[2].nodePort

As you can see, in my case, it’s 32536, but yours is most certainly different.

Next, replace the /etc/nginx/sites-enabled/default file with one containing this:

server {
  listen          80;
  server_name     www.ssltest.com;

  location / {
    proxy_pass      https://127.0.0.1:32536;
  }
}

Check that everything is in place and reload the config:

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ sudo nginx -s reload

Now put an entry for www.ssltest.com in your /etc/hosts file to point to 192.168.50.4 (the IP address of the virtual machine).

Opening www.ssltest.com in your browser will yield a 502 error, like this:

Debugging Ngnix

Let’s have a look at the Nginx logs:

2020/07/14 09:50:23 [error] 160208#160208: *3 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: 192.168.50.1, server: www.ssltest.com, request: "GET /favicon.ico HTTP/1.1", upstream: "https://127.0.0.1:32536/favicon.ico", host: "www.ssltest.com", referrer: "http://www.ssltest.com/"

Unfortunately, it seems that there was a problem with SSL handshaking. Let’s try a curl from our terminal directly to the Istio Node Port:

$ curl -HHost:www.ssltest.com https://192.168.50.4:32536
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 192.168.50.4:32536

Now we see another error regarding SSL. Next, let’s test it by following the recommendations in the official documentation. As a side note, we’ll also use the -k flag to skip certificate checking, as it’s self-signed.

$ curl -k -HHost:www.ssltest.com --resolve "www.ssltest.com:32536:192.168.50.4" "https://www.ssltest.com:32536"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Voila! At last, this seems to work, but why? The answer is that when using the –resolve flag, curl sends SNI information to the endpoint, which Istio uses to decide what certificate to use for offloading.

All this is great, yet how do we fix our Nginx setup?

The answer lies within these 3 configuration options:

proxy_ssl_name $host;
proxy_set_header Host $host;
proxy_ssl_server_name on;

Let’s add them to our configuration file, it will look like this:

server {
  listen          80;
  server_name     www.ssltest.com;

  location / {
    proxy_pass https://127.0.0.1:32536;
    proxy_ssl_name $host;
    proxy_set_header Host $host;
    proxy_ssl_server_name on;
  }
}

And now once again, reload Nginx:

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ sudo nginx -s reload

Let’s reload our browser window. Although it won’t hit a 502 error anymore, we face a blank page. Next, take a look at the error logs—it seems that there are no new errors, so let’s have a look at the access logs:

192.168.50.1 - - [14/Jul/2020:10:03:05 +0000] "GET / HTTP/1.1" 426 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"

It seems we got a 426 HTTP error, which means that Istio wants us to upgrade our connection, so we do that by setting the proxy_http_version to 1.1:

server {
  listen          80;
  server_name     www.ssltest.com;

  location / {
    proxy_pass https://127.0.0.1:32536;
    proxy_ssl_name $host;
    proxy_set_header Host $host;
    proxy_ssl_server_name on;
    proxy_http_version 1.1;
  }
}

Now go ahead and test / reload Nginx again and refresh your browser page.

Lastly, you can see now that everything works as it should:

If you want to learn more about these topics, check out the following resources: