Reverse Proxy: Nginx Proxy Manager (Rev: 07/30)

Dockge deployment of the “Nginx Proxy Manager” reverse proxy to create private-network routable URLs with a Let’s Encrypt wildcard certificate and Cloudflare as our DNS provider.

Jul 19, 2024
💡
Dockge deployment of the “Nginx Proxy Manager” reverse proxy to create private-network routable URLs with a Let’s Encrypt wildcard certificate and Cloudflare as our DNS provider.
 
Revision: 20240730-0 (init: 20240714)
 
This post details using Nginx Proxy Manager with Let’s Encrypt to generate HTTPS upgrades for self-hosted services.
 

Preamble

A reverse proxy is an application running on a server that acts as an intermediary between clients and one or more web servers. All client requests go through the reverse proxy, which examines them and forwards them to the appropriate origin server (the primary location where the original versions of a website's content reside).
Reverse proxies provide:
  • Security as they can filter malicious traffic and protect origin servers from direct attacks.
  • Performance by caching frequently accessed content, reducing the load on origin servers.
  • Load balancing by distributing traffic across multiple origin servers, preventing any single server from becoming overloaded.
  • A centralized management interface where policies can be managed.
  • HTTPS certificates to upgrade HTTP connections to HTTPS.
 
When using HTTPS upgrades, it is preferred to use a certificate that can prove a chain of trust.
This ”certificate chain of trust is the hierarchical structure of digital certificates that establishes trust in an SSL/TLS certificate. It works as follows:
  1. An intermediate Certificate Authority (CA) issues the end-entity certificate (used to secure a website or service).
  1. A higher-level root CA certificate issues the intermediate CA certificate.
  1. The root CA certificate is self-signed and is considered a trusted authority.
  1. When a client (such as a web browser) encounters an SSL/TLS certificate, it checks the certificate chain to verify that the end-entity certificate can be traced back to a trusted root CA.
This chain of trust ensures that the end-entity certificate is legitimate and can be trusted, as it has been verified by the trusted root CA.
 
Let's Encrypt (LE) is a free Certificate Authority that automatically obtains and renews websites’ SSL certificates. LE provides the trusted root CA certificates and intermediate certificates that can be used to verify the chain of trust for their SSL/TLS certificates.
When requesting an SSL/TLS certificate from LE, a challenge-response mechanism is used to prove control over the domain:
  • with “HTTP-01 Challenge,” LE sends an HTTP request to a specific URL. LE verifies control over the domain if the response has the provided token.
  • “DNS-01 Challenge” is used when no website is started or that website is not accessible from outside the host. LE then generates a random TXT record to add to the DNS zone. LE verifies control over the domain by confirming that the record matches the expected values.
 
A few possible reverse proxy solutions software exist, as seen on Authelia’s Proxy support page (https://www.authelia.com/integration/proxies/support/). “Authelia is an open-source authentication and authorization server and portal fulfilling the identity and access management (IAM) role of information security in providing multi-factor authentication and single sign-on (SSO) for your applications via a web portal.”
 
This guide will provide instructions on using Nginx Proxy Manager (NPM). NPM is a WebUI that creates and maintains reverse proxies and SSL certificates for those proxies and configures some level of access controls. It integrates with LE, is designed to run in Docker containers, and can handle DNS challenges with many DNS providers, including Cloudflare, the service we will use.

Used values

In the following, we will rely on the following values. Adapt those to match your expected deployment:
  • example.com is our domain (zone), and its DNS provider is Cloudflare.
  • 192.168.222.11 is the static IP of the host running the NPM services (among others).

Prerequisites: Cloudflare

For details on using Cloudflare as your DNS provider, please see the “Cloudflare → DNS Setup Provider” section of the “VPS: Cloudflare Zero Trust access to Web Applications” post.

Add an A record

With Domain Name System (DNS), an A record (also known as a Host Record or Address Record) is a type of resource record that maps a hostname to an IP address.
Here, we want to create a resolver for all entries that do not already exist on example.com. For example, when seeking unknown.example.com, we want the resolver to ask the service running on the private network (here, to 192.168.222.11, i.e., the static IP on our private network where NPM will be installed) for the matching service port to answer (i.e., HTTP on 80, HTTPS on 443). This presents two advantages: 1. Most browsers will automatically assume we are attempting to connect to http://unknown.example.com 2. Most browsers will automatically upgrade the connection from HTTP to HTTPS and request https://unknown.example.com.
With Cloudflare as our domain provider, select example.com from the “Websites” page on the Cloudflare Dashboard. Clicking “Add record” and make it an A ”type,” a * ”name,” pointing to the 192.168.222.11 ”IPv4 address,” disabling the “Proxy status” so it is DNS only. The “TTL” choice can be left to Auto.
Once this is done, any request for any subdomain of our example.com subdomain will resolve to our private IP. As long as we are in the same subnet (or on a VPN that grants us access to this subnet), the DNS will resolve the 192.168.222.11 IP (it is a “name”), and the matching service (HTTP, HTTPS) will be queried.

Obtain a DNS challenge token

A DNS challenge token is a User API Token needed to automate the process of completing a DNS-01 challenge by permitting it to update DNS records using the Cloudflare API while issuing the Let’s Encrypt SSL Certificate.
From the Cloudflare Dashboard, select “Zone” (here example.com). On the “Overview” page, go to the right side, scroll down to the “API” section, and select “Get your API token” followed by “Create token → Create Custom Token”:
  • we will name it with details of what it is for example.com-npm
  • its permissions will be “Zone,” “DNS,” and “Edit”: have permission to edit the DNS Zone(s)
  • we will limit to our domain only: for “Zone Resources,” select “Include,” “Specific Zone,” example.com
Then, “Continue to summary → Create Token”; you will be shown the secret token only at issuance; copy it and store it in a safe place. On the same page, we can see a shell command to “test this token”; feel free to confirm it is valid before continuing.

Nginx Proxy Manager setup

We will deploy NPM as a docker compose service and integrate it as a stack within our Dockge deployment. Please see the “Dockge” post for additional details on setup.

Docker Compose stack in Dockge

For this setup, we will create a wildcard certificate for all hosts on subdomains of example.com
Please refer to the official setup instructions for alternate options, available at https://nginxproxymanager.com/setup/
We will deploy our npm stack such that it uses the default SQLite database, answers on the HTTP and HTTPS ports, and exposes its admin WebUI (port 81). We will add a healthcheck for Docker to be able to restart the service in case of issues and disable IPv6 on our private subnet. The resulting compose.yaml is:
services: npm: image: 'jc21/nginx-proxy-manager:latest' restart: unless-stopped ports: - '80:80' - '81:81' - '443:443' volumes: - ./data:/data - ./letsencrypt:/etc/letsencrypt - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro environment: DISABLE_IPV6: 'true' healthcheck: test: ["CMD", "/usr/bin/check-health"] interval: 10s timeout: 3s labels: - "com.centurylinklabs.watchtower.enable=true"
After the app is running for the first time, the following will happen:
  1. JWT keys will be generated and saved in the data folder
  1. The database will initialize with table structures
  1. A default admin user will be created
From a browser, access http://192.168.222.11:81/ ; the default email and password are set as [email protected] and changeme. Please change both; it is recommended to set the email as the email address you will use to register domains with Let’s Encrypt so it can be auto-populated. The email address is used for expiry notifications and not shared (see https://serverfault.com/a/1132672 for additional details). Here, we will [email protected].
For our setup, we will change the default site to a 404 (not found): in “Settings”, change the “Default Site” to a 404

Generating our first SSL certificate

For our setup, we will generate a wildcard certificate, an SSL/TLS certificate that can secure multiple subdomains of a domain with a single certificate. It includes a wildcard character (*) in the domain name field; this means that site1.example.com or site2.example.com will use the same certificate generated by Let’s Encrypt.
In the “SSL Certificates” tab ”Add SSL Certificate”, where we are told that:
These domains must be already configured to point to this installation
We have already added our A entry to point to our private IP of 192.168.222.11. This a non-routable IP ; those are reserved for private networks and are not globally unique, cannot be used to communicate directly over the Internet, and are used within a local network.
Because of this, we can not use the “HTTP-01 Challenge” to obtain a certificate, and will instead use the “DNS-01 Challenge” instead.
For “Domain names” enter *.example.com (when using NPM, remember to press enter to validate the entry, this is true for any further manually entry of sites):
  • make sure to enter a valid email address to register with Let’s Encrypt; here we will use our [email protected] preferred email address (the same we used for the admin account creation with NPM).
  • select “use DNS challenge”
    • select “Cloudflare”
    • Use the “DNS Challenge” token (note the warning: “This data will be stored as plaintext in the database and in a file!“, so make sure to run this on your infra)
  • Agree to the TOS of Let’s Encrypt
  • “Save”. If everything is configured properly we will now have our first certificate added to our dashboard.
Let's Encrypt certificates expire after 90 days. NPM will attempt to automatically renew these certificates when they have a third of their total lifetime left, which is approximately 30 days before expiration.

Adding our first Proxy Host

⚠️
If you have any DNS rebinding protection service or hardware on your stack, please make sure to add *.example.com to the authorized list before continuing.
DNS Rebind would allow an attacker to manipulate the DNS resolution of a domain to make it appear as if an internal IP address (e.g., 192.168.222.1) is actually an external one (e.g., example.com). This allows them to bypass security controls and access sensitive information.
To mitigate this risk, DNS Rebinding Protection checks the IP address of a domain against a list of known internal IP addresses. If a match is found, the request is blocked or redirected to prevent potential attacks.
A proxy host maps a service (running as HTTP or HTTPS on a given host and a given IP) to a URL. We will add npm.example.com as our first proxy host and link it to HTTP for 192.168.222.11 and port 81.
From the Dashboard, select “Proxy Hosts”, then “Add a Proxy Host”:
  • enter npm.example.com as the “Domain names”, it is possible to multiple names to point to the same mapping (for example, we could also name it nginxproxymanager.example.com or reverseproxy.example.com)
  • Enter “Scheme”: http (how the service is presented), “Forward Hostname / IP”: 192.168.222.11(the IP of host running NPM), and “Forward Port”: 81 (where the WebUI for NPM is present)
  • “Cache Assets” cache frequently requested assets (e.g., images, stylesheets, JavaScript files) from proxied services which can reduce the load on the original servers and improve performance by serving cached copies of these assets. We will enable it.
  • “Block Common Exploits” helps protect proxied services from common web vulnerabilities (like cross-site scripting (XSS)) attacks. Although we are only serving local content to local systems. We will enable it.
  • “Websocket Support” enables support for WebSocket connections between clients and proxied services, allowing for bi-directional communication over TCP/IP, often enabling real-time messaging and live updates in web applications. Although not all services will support it, adding its support is useful in case it becomes part of the tool. We will enable it.
  • If using “Access list”, those must be configured before adding a proxy host, and can present a “Basic Auth Authentication” (username and password), add IP and subnet whilelist (a list of trusted IP addresses that are explicitly allowed to access a system) or blacklist (IP addresses that are explicitly blocked from accessing a system). We have not configured any “Access list” as this service is only accessible on our local network, so we will use “Publicly Accessible”
In the “SSL” tab, we must select the certificate to use (*.example.com). Because we are using a wildcard certificate, there is no need to request a new certificate, our wildcard covers any direct subdomain of example.com (ie we use npm.example.com but not npm.lab.example.com for example; if you were to try, you will get something akin to “Firefox does not trust this site because it uses a certificate that is not valid for npm.lab.example.com. The certificate is only valid for *.example.com.”):
  • “Force SSL” enables or disables forced SSL (HTTPS) for a specific Proxy Host. When enabled, it forces all requests to be made over HTTPS. We will enable it.
  • HSTS Enabled” activates HTTP Strict Transport Security (HSTS) for the proxy host. This setting helps prevent users from being tricked into accessing an insecure version of the website by forcing all requests to be made over HTTPS. We will enable it.
  • “HTTP/2 Support” enables or disables support for HTTP/2 protocol, which allows for multiplexing multiple requests over a single connection, reducing overhead and improving performance.
  • “HSTS Subdomains” refers to the inclusion of all subdomains within a domain in an HSTS (HTTP Strict Transport Security) policy. Because we only have a wildcard for the base example.com, we would get an untrusted certificate error were we to create a test.npm.example.com entry. We will disable it.
In this setup, we will not make use of the “Custom locations” or “Advanced” tabs (the advanced tab is useful for other type of services; for example if setting up a Docker registry, as explained in the next section).
After performing a “Save”, a new entry will be added to our “Proxy Hosts” list. You can click on the entry to get to the running nginx answering this reverse proxy. When doing so, our DNS will query to see where to look up entries for npm.example.com. Unless a direct entry already exists at Cloudflare, the DNS will respond with the *.example.com wildcard entry and tell the browser to ask 192.168.222.11 to resolve this site. The proxy will in turn ask to main nginx server if it is aware of an npm.example.com configuration. Because it is configured, the rules we set above are enforced and 1) the connection is encapsulated in an HTTPS request that goes to HTTP for 192.168.222.11 on port 81 (http://192.168.222.22:81), our NPM Dashboard.

Adding additional private services

Because of the way the reverse proxy functions, it is possible to add other services on IPs reachable by the reverse-proxy host. For example, we have a Docker registry on 192.168.222.14 running on the default port (5000). This service runs on the same subnet as our NPM; we do not need to setup another NPM instance on that other host, instead we can add a registry.example.com ”Proxy Host” entry on the existing NPM instance. When creating it, we use the parameters of the connection (http, 192.168.222.14, 5000).
For Docker Registry in particular, if we were to upload a large container image, we might encounter an HTTP 413 error (”Content Too Large”). To avoid this, we need to pass additional directives to the nginx server directly. In the “Advanced” tab for this “Proxy Host”, we add client_max_body_size 0;. More details on this extra setting and other available can be found at https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size

Further reading

Revision History

  • 20240730-0: Passed timezone and watchtower label to docker compose
  • 20240719-0: initial release