Allan
Ramsay

Cloud engineering, home automation & photography enthusiast based in New Zealand

Self Configuring Web Proxy With TLS

Self Configuring Web Proxy With TLS

One of the key parts of my (or I would argue, any) home lab is the web proxy. It is usually the literal entry point of the lab, and for mine I had a list of requirements:

  • Self configuring: adding a new service shouldn’t require config file changes and restarts, likewise removed services shouldn’t leave old configuration around.
  • Real TLS: TLS is no longer optional, browsers (rightly) pop up big scary warnings when accessing plain HTTP. We would never deploy a service on the internet without TLS so why should a home lab any different.
  • Segmentation: some services I do want to share out to the internet, but many should only be available internally.
  • Authentication: not everything has good authentication, and adding it at the proxy level is a powerful option.
  • Hot reloads: I don’t want a restart with connection drops whenever a service changes, and along with that container friendly is a must.

I have tried a number of options, including Apache, NginX, HAProxy, Envoy, Caddy and Traefik, with Traefik being the product I think fits best into my home lab.

Traefik Network Setup

Traefik sits in the nice middle ground between classic config file setup like HAProxy and the cluster native xDS protocols that more specialized options like Envoy rely on. The core static configuration is in YAML files, with dynamic configuration being provided by way of various plugins.

Here is a diagram of how the whole setup connects together, with web proxy, DNS and service discovery all working together. I’ll explain how each part works throughout the rest of this article.

A network diagram showing web traffic flows originating from internet and local network going to a Traefik host. Traffic then flows onto a combination of Nomad and Docker services. There is also a Consul service which is used to configure Traefik as well as Dnsmasq.

The static configuration

This is where the core shape of this proxy takes place, where it listens from and on what protocol, the official documentation now refers to this as install (startup) configuration.

There are lots of configuration options available, and the documentation does a great job of explaining them, so I’ll just go over the core features of my setup, starting with entry points:

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
  https:
    address: ":443"
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1/32"
        - "172.16.0.0/12"
        - "10.0.0.0/8"
    transport:
      respondingTimeouts:
        readTimeout: 10m
  external:
    address: ":8443"
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1/32"
        - "172.16.0.0/12"
        - "10.0.0.0/8"
    transport:
      respondingTimeouts:
        readTimeout: 10m

I have three main entry points:

  • http: simply redirects insecure connections to a secure protocol
  • https: the main entry point for traffic within the LAN (more on this later)
  • external: the main entry point for public traffic from the internet

Dynamic / Self Configuration

This is where Traefik shines above the older options, dynamic configuration (now referred to as routing configuration) is a first class citizen, and each change comes with a hot reload. No signals or socket messages required which means we can just throw it into a container and forget about it.

Setting this up just requires some provider configuration:

providers:
  # Configure things that change on the proxy itself in this
  # file, e.g. certificates
  file:
    directory: "/etc/traefik/dynamic"
    watch: true

  # Hooking into the Consul catalog allows for service discovery
  consulCatalog:
    endpoint:
      address: localhost:8500
      datacenter: home
    connectAware: true
    exposedByDefault: false
    watch: true
    prefix: "traefik"
    serviceName: "traefik-ingress"

  # By default any containers running on the same host would get
  # exposed, this is great for single machine setups, not needed
  # for my multi-host lab
  docker:
    exposedByDefault: false

And the associated dynamic file:

# These certs are provided by Lets Encrypt via acme.sh running on
# the machine.
tls:
  stores:
    default:
      defaultCertificate:
        certFile: "/certs/aramsay.co.nz.crt"
        keyFile: "/certs/aramsay.co.nz.key"
  options:
    default:
      minVersion: VersionTLS12

# Middlewares can be defined here too, this "internal" one makes
# sure traffic originates from a private IP address
http:
  middlewares:
    internal:
      ipAllowlist:
        sourceRange:
          - "10.0.0.0/8"
          - "172.16.0.0/12"
          - "192.168.0.0/24"
          - "127.0.0.1/32"

All other configuration comes via that Consul Catalog provider, this includes routes, services and middleware. The great thing about this is proxy configuration that is specific to a service lives with that service’s code/configuration.

Now any service that has supported tags within Consul will automatically be included within Traefik, e.g. in the Nomad job for Home Assistant:

service {
    name         = "home-assistant"
    port         = "http"
    provider     = "consul"
    address_mode = "alloc"
    tags = [
        "traefik.enable=true",
        "traefik.http.routers.home-assistant.rule=Host(`allan.house`)",
        "traefik.http.routers.home-assistant.entrypoints=https",
        "traefik.http.routers.home-assistant.tls=true",
    ]
}

Breaking that down a bit, we are enabling Traefik integration with traefik.enable, then setting up a router with the three required properties:

  • A rule tells Traefik how to match incoming traffic to that router. Generally using Host() to match on hostname but pathname, etc is possible too.
  • One or more entrypoints to tell Traefik where to listen.
  • Setting tls to true without any other options will use the certificate configured in the dynamic configuration file above.

entrypoints=https is key here. There is no way I want Home Assistant directly exposed to the internet (I know I can do this via Home Assistant Cloud but I prefer a VPN for remote access) so setting the entry point just to https ensures that the external listener would just return a 404, regardless of DNS setup.

DNS

I use split horizon DNS in my lab, so you’re reading this on aramsay.co.nz which resolves to a set of IPs in AWS (via Route 53), but within my network I have a separate DNS server (dnsmasq) which resolves the same addresses to internal endpoints. But since I can have a public presence, I can also validate TLS certificates through Lets Encrypt.

To achieve this I have a separate dnsmasq host running, with consul-template generating the configuration files by filtering for services that have the traefik.enable=true flag set. These are the same services that Traefik will pick up too. From there it writes out two sets of entries, one for IPv4 and one for IPv6, taking into account whether Consul Sidecars are present or not. The template is ugly, but it does work:

# Network addresses generated from Traefik (IPv4)
{{- with node "traefik" -}}
  {{- $traefik := .Node.Address -}}
  {{- range services -}}
    {{- if and (.Tags | contains "traefik.enable=true") (.Name | contains "sidecar" | not) -}}
      {{- range .Tags -}}
        {{- if regexMatch "traefik\\.http\\.routers\\..*\\.rule=Host\\(`(.*)`\\).*" . }}
{{ $traefik }} {{ . | regexReplaceAll "traefik\\.http\\.routers\\..*\\.rule=Host\\(`(.+?)`\\).*" "$1" }}
        {{- else if regexMatch "traefik\\.tcp\\.routers\\..*\\.rule=HostSNI\\(`(.*)\\.(.*)`\\).*" . }}
{{ $traefik }} {{ . | regexReplaceAll "traefik\\.tcp\\.routers\\..*\\.rule=HostSNI\\(`(.+?)`\\).*" "$1" }}
        {{- end -}}
      {{- end -}}
    {{- end -}}
  {{- end -}}
{{- end }}

This also means I do not need to care about static IP addresses, etc, if anything on the service updates (e.g. moves to a new host, DHCP lease expires) then that update will be reflected immediately via the Consul metadata and this works for IPv6 too! Any service that is not running internally (e.g. this website) and therefore does not have a local endpoint to resolve will just be passed on to Route 53 as dnsmasq is a forwarding resolver.

Real TLS

I mentioned TLS certificates before, but this was another requirement of mine. I don’t want to deal with self signed certificates, either accepting the browser warnings or dealing with custom CAs. Some of these services (like Home Assistant) are used by my family, so real TLS certificates were a must.

This proxy takes care of that by using the handy acme.sh utility and API keys for the AWS Route 53 DNS service to request and validate TLS certificates every 60 days. These certificates are placed into a shared directly with Traefik, then the dynamic configuration file is touch’d to trigger the hot reload functionality.

Deploying the certificate looks like this:

acme.sh --install-cert -d *.aramsay.co.nz --ecc --fullchain-file /opt/traefik/certs/aramsay.co.nz.crt --key-file /opt/traefik/certs/aramsay.co.nz.key --reloadcmd \"touch ~/traefik/dynamic-configuration.yaml\"

Segmentation

As shown above, the config entrypoints lets me segment services to internal, external or both. Because I’m using port forwarding and running the external entrypoint on an alternate port, traffic from outside of the network never reaches that primary https listener. Split horizon DNS ensures that clients get directed to the correct endpoint too.

Authentication

This one was later to the mix but is a biggie, just about every service comes with its own login/user management. This inevitability means a bunch of things get setup with the same username/password combo (because we’re always just trying things out, and don’t want to admit that “POC” actually stands for “Production On Completion” 😉). I could be more diligent with my password manager, or I could take advantage of my central Pocket ID identity provider and have one passkey for everything.

Obviously the first port of call is to configure everything to use OIDC natively (which fortunately is more common than not, and a bit of a litmus test on the maturity of a project) but for stalwart holdouts of the single username and password (or worse, basic auth), then Tiny Auth has been a really nice solution that integrates with Traefik (using the aforementioned Consul tag system) as a forward auth plugin.

Setup is straight forward, with a middleware config pointing forward auth traffic back through to the Tiny Auth service:

service {
  name     = "tinyauth"
  port     = "http"
  provider = "consul"
  tags = [
    "traefik.enable=true",
    "traefik.http.routers.tinyauth.rule=Host(`tinyauth.aramsay.co.nz`)",
    "traefik.http.routers.tinyauth.entrypoints=https,external",
    "traefik.http.routers.tinyauth.tls=true",
    "traefik.http.middlewares.tinyauth.forwardauth.address=http://tinyauth.service.consul:${NOMAD_HOST_PORT_http}/api/auth/traefik",
    "traefik.http.middlewares.tinyauth.forwardauth.trustForwardHeader=true",
    "traefik.http.middlewares.tinyauth.forwardauth.authResponseHeaders=authorization"
  ]
}

Most of that should be fairly self explanatory, but the authResponseHeaders=authorization is an extra part that allows those browser based services to retain a shared user that is passed through with the request.

Each authenticated service then just needs to add a label/tag like traefik.http.routers.my-service.middlewares=tinyauth@consulcatalog and any requests will first be routed via Tiny Auth/Pocket ID first.

Stopping direct access

This alone doesn’t stop direct access since my LAN is pretty flat, I prefer the concept of zero trust to complex routing rules and numerous VLANs. To make sure that only the web proxy is communicating with these services we need to bring in mTLS. Going on the self configuration side of things again, this is provided by Consul Sidecars, which are Envoy proxy and a lot of TLS magic under the hood. To use these, I have the services listen only on localhost, then expose them through the sidecar where Consul “intents” take over as the ACL. An example Docker Compose file might look like this:

services:
  my-service:
    container_name: my-service
    image: ghcr.io/a-ramsay/my-service
    ports:
      - "127.0.0.1:3000:3000" # Lock service down to localhost only
    labels:
      - "traefik.enable=true"
      - "traefik.consulcatalog.connect=true" # Traefik knows to use mTLS/Consul Connect
      - "traefik.http.routers.my-service.entrypoints=https"
      - "traefik.http.routers.my-service.rule=Host(`my-service.aramsay.co.nz`)"
      - "traefik.http.routers.my-service.tls=true"
      - "traefik.http.routers.my-service.service=my-service"
      - "traefik.http.routers.my-service.middlewares=tinyauth@consulcatalog" # Use auth
    restart: always

  my-service-sidecar:
    image: ghcr.io/a-ramsay/consul-envoy
    restart: unless-stopped
    network_mode: host
    depends_on:
      - my-service
    # This command looks for the service
    command: -sidecar-for my-service -http-addr http://localhost:8500 -grpc-addr localhost:8502 -admin-bind 127.0.0.1:0

  # Helper service I wrote for exposing Docker services and labels to Consul
  # services with tags: https://github.com/a-ramsay/consul-register
  consul-register:
    image: ghcr.io/a-ramsay/consul-register
    restart: unless-stopped
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    network_mode: host
    depends_on:
      - my-service-sidecar

We still configure Traefik the same way with labels on the actual service itself, including the tinyauth@consulcatalog middleware, with the key difference in the ports configuration. Using 127.0.0.1:3000:3000 instead of 3000:3000 tells Docker to only expose this to localhost.

The consul-envoy sidecar container takes over from there, it looks for the service and sets up an mTLS listener on a random port via Consul. This is what gets advertised out, and only other clients within the ACLs set by the Consul Intent system will receive valid certificates.

The final piece of the puzzle here that hasn’t yet been mentioned, but is used on any pure Docker deployment is the consul-register service. You can use this one yourself, it’s available on my GitHub and it watches the Docker socket for events, then updates Consul with services and labels it finds. This is how I can use Docker labels across all hosts within my lab and not just the one Traefik is running on.

In Summary

With this setup I can largely just set and forget the proxy, with it just requiring periodic software updates.

Adding a new service whether it’s installed directly on a physical machine (even Raspberry Pi), a VM running Docker or deployed to my Nomad cluster getting HTTP traffic routed to it is the same process.

  1. Add labels to the service (Docker, Nomad or direct in the Consul configuration)
  2. Consul triggers dnsmasq to add a new DNS entry pointing back to the Traefik VM
  3. Consul triggers Traefik to add a new router and service attached to required entry points
  4. Full HTTPS connection now available everywhere required!

Next steps

There are a couple of other improvements I am considering, the main one being removing the port forwarding from my router through to Traefik and directing that through a Cloudflare tunnel instead. The primary reason here is extra security, every port opened on a router is a risk and with so much more activity in the vulnerability discovery space thanks largely to powerful AI models it has perhaps tipped the scale on the importance of real time and reactive systems like Cloudflare over traditional IDS/IPS tools that my router provides.