Ubuntu 24.04 VPS Hardening (20240922)

Hosting on a VPS is a great option to run a blogging service, but installing services that might expose ports needs to be done with some precaution (or not at all if the service is only to be used by the server itself).

Martial

Jun 16, 2024

TABLE OF CONTENTS

Remote Access Setup

Changing the default user

Hardening ssh

A note on the default list of authorized Ciphers

Secure DNS

DNS-over-HTTPS using cloudflared

DNS-over-TLS using Quad9

Preventing access and limiting entry

ufw

ufw-docker

firewalld

fail2ban

reboot

Further Hardening

Logwatch

Security Auditing

Rootkit Scanner

Revision History

🔒

Revision: 20240922-0 (init: 20240616)

This post will discuss some of the concepts required to harden your VPS, such as minimizing its attack surface, strengthening access controls, and applying security patches and updates to reduce the risk of unauthorized access.

Remote Access Setup Changing the default user Hardening ssh A note on the default list of authorized Ciphers Secure DNS DNS-over-HTTPS using cloudflared DNS-over-TLS using Quad9 Preventing access and limiting entry ufw ufw-docker firewalld fail2ban reboot Further Hardening Logwatch Security Auditing Rootkit Scanner Revision History

Securing a VPS (Virtual Private Server) protects data, applications, and the overall system from unauthorized access and potential threats. Without adequate security measures, your VPS could become part of a botnet, be used to host phishing websites, send scam emails, and be used for crypto mining or other illegal activities. Depending on the content of your VPS, this might also cause some data theft for your end users, or your VPS might become encrypted in a ransomware attack.

As such, “hardening” your VPS refers to enhancing your server's security by implementing various measures to reduce its vulnerability to attacks and unauthorized access. It involves configuring settings, applying updates, and utilizing security tools to create a more robust and resilient system.

This setup was done on an OVHcloud VPS (a VLE-4), but the instructions below easily adapt to Linode or Digital Ocean. We will not cover the initial host creation of a Ubuntu 24.04 server VM on your cloud provider of choice.

Remote Access Setup

Generally, if you're primarily using your VPS for hosting websites, running web applications, or other server-related tasks, a server OS is highly recommended for its performance, security, and management advantages.

Depending on the cloud provider, your ubuntu account will get your ssh key added for login or you will get a temporary password to change at initial login.

After obtaining your ubuntu user password and host details, initialize a new ~/.ssh/config.d/vps file (you can include all files from the .config.d directory by adding Include config.d/* at the top of the ~/.ssh/config file) and add access details to it:


Host vps
HostName <VPS-DETAILS>
User ubuntu

If you log in using a password, after your initial ssh vps, change it to a secure one (stored in a password manager if needed) using passwd.

The first step is to make sure our host is up-to-date by running sudo apt update && sudo apt upgrade followed by a reboot using sudo reboot -h now

The reboot will help if any kernel upgrade is performed but will also allow us to confirm the login with the new password is functional.

Creating a new SSH key for use with this VPS is easy; use ssh-keygen -t ed25519 -c "vps" -f ~/.ssh/id_ed25519-vps to create one and armor it with a passphrase if preferred (you will want to make sure to add it to your ssh-add or keychain if you do). Then copy the public key to the remote host’s .ssh/authorized_keys:


rsync -avR ~/./.ssh/id_ed25519-vps.pub vps:./.ssh/authorized_keys

Test to confirm that you are not challenged by a password the next time you log into the vps host.

Changing the default user

Because server installs often create default users, those are the first usernames attempted by bots, as such let’s add a new sudo-capable vpsu user:


sudo adduser vpsu
sudo usermod -aG sudo vpsu
sudo rsync --archive --chown=vpsu:vpsu ~/.ssh /home/vpsu

, then test that you can log in as the vpsu user: ssh vpsu@vps

Hardening ssh

Next, let’s harden the ssh configuration by modifying: sudo nano /etc/ssh/sshd_config.

Add/modify the following options (in order within the file):

change the default port to another value of your choice; here, let’s use Port 1327

make sure root login is disabled: PermitRootLogin no

do not authorize password login, only identify files: PasswordAuthentication no

you might also need to disable the cloud-init alterations (which might re-enable password authentication): also set PasswordAuthentication no in sudo nano /etc/ssh/sshd_config.d/50-cloud-init.conf (or other files in that directories, if it is on), see https://askubuntu.com/a/1440509 for details

only allow our new vpsuuser: AllowUsers vpsu (see https://linux.die.net/man/5/sshd_config for more details on this option)

while still in the terminal, restart sshd (so you can make modifications in case of issues): sudo service ssh restart

Update the .ssh/config.d/vps on your host (not the VM) accordingly:


Host vps
HostName <VPS-DETAILS>
User vpsu
Port 1327
IdentityFile ~/.ssh/id_ed25519-vps
IdentitiesOnly yes

, then, on your host, try to ssh ubuntu@vps again. You should not only be denied the option to enter a password, but your ubuntu user should also get Permission denied (publickey)

Finally, while still logged in as vps on the VM, on your host, confirm you can ssh vps

Further reading: https://support.us.ovhcloud.com/hc/en-us/articles/115001669550-How-to-Secure-Your-SSH-Connection-in-Ubuntu-18-04

A note on the default list of authorized Ciphers

The default list is considered to be secure, and modifying this list makes you responsible for confirming your client can support it.

You can find the list of your server ciphers by running on your VPS: sudo sshd -T | grep "\(ciphers\|macs\|kexalgorithms\)". The ciphers line should only contain chacha and aes answers, which are considered secure.

From your client side, you can find the list of the ctos and stoc ( client to server, server to client) supported ciphers by using ssh -vv vps. The content will be in some of the debug2 lines.

To limit that list to a smaller subset, you can edit the /etc/ssh/sshd_config file and add (or alter if one is already present) ciphers line. For example:


ciphers [email protected],aes256-ctr,[email protected]

, followed by a restart of the ssh daemon: sudo service ssh restart

You can confirm the list matches by re-running sudo sshd -T | grep "\(ciphers\|macs\|kexalgorithms\)"

Secure DNS

I use NextDNS and Control-D to block outgoing trackers for most of my systems. For a VPS, I will use either Cloudflare or Quad9’s DNS resolvers.

If you have a domain name whose DNS is hosted at Cloudflare and intend to use Cloudflare’s Zero Trust tunnels, cloudflared is the recommended solution. For additional details, see https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/get-started/create-remote-tunnel/

DNS-over-HTTPS using cloudflared

We will use Cloudflare’s cloudflared's alternate configuration as a ”malware blocking” DNS.

Obtain and install the latest client:


cd /tmp
wget https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb
sudo dpkg -i cloudflared-linux-amd64.deb
rm -f cloudflared-linux-amd64.deb

Run a test in a tmux (or have another ssh into the host). In one terminal, start the client:


cloudflared proxy-dns --port 5053 --upstream https://1.1.1.2/dns-query --upstream https://1.0.0.2/dns-query

Confirm the configuration is working by starting from the other terminal a connection to the DNS resolver on port 5053:


dig @127.0.0.1 -p 5053 cloudflare.com AAAA

You should see the DNS request in the first terminal using the specified https addresses. Kill the cloudflared command and let’s make this configuration a system service; sudo nano /etc/systemd/system/cloudflared-proxy-dns.service and use the following:


[Unit]
Description=DNS over HTTPS (DoH) proxy client
Wants=network-online.target nss-lookup.target
Before=nss-lookup.target

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DynamicUser=yes
ExecStart=/usr/local/bin/cloudflared proxy-dns --upstream https://1.1.1.2/dns-query --upstream https://1.0.0.2/dns-query

[Install]
WantedBy=multi-user.target

Enable the new service:


sudo systemctl enable --now cloudflared-proxy-dns

, and confirm it is running and using the proper upstream resolvers:


sudo service cloudflared-proxy-dns status

Modify /etc/systemd/resolved.conf having it use DNS=127.0.0.1 as the only active entry in the [Resolve] section, then restart the service:


sudo service systemd-resolved restart

Further reading: https://developers.cloudflare.com/1.1.1.1/encryption/dns-over-https/dns-over-https-client/

DNS-over-TLS using Quad9

If you do not use Cloudflare tunnels, you can use Quad9 for its threat-blocking features and upgrade our connection from plain DNS to DNS-over-TLS. For details on available services, see https://www.quad9.net/service/service-addresses-and-features

Edit the /etc/systemd/resolved.conf file and enter the following [Resolve] section:


[Resolve]
DNS=9.9.9.9#dns.quad9.net
DNS=2620:fe::fe#dns.quad9.net
DNS=149.112.112.112#dns.quad9.net
DNS=2620:fe::9#dns.quad9.net
DNSOverTLS=yes

Restart the service for the changes to take effect: sudo service systemd-resolved restart

Test that we are using Quad9:


dig +short txt on.quad9.net.

, should return yes.quad9.net.

Preventing access and limiting entry

ufw (Uncomplicated Firewall) acts as a first line of defense by filtering incoming and outgoing traffic based on predefined rules. fail2ban monitors the system logs for suspicious activity and automatically bans IP addresses showing malicious behavior. Both ufw and fail2ban interact with the underlying iptables firewall to manage rules and actions. The combination of ufw and fail2ban helps protect against various threats, including unauthorized access, brute-force attacks, and other automated attacks.

ufw

Set ufw to deny all incoming connections and allow traffic out:


sudo ufw default deny incoming
sudo ufw default allow outgoing

Allow incoming ssh connections using our alternate port:


sudo ufw allow 1327

Show the existing rules to confirm ssh is added by using sudo ufw show added

Enable ufw on the host:


sudo ufw enable

You can check the existing rules and delete some if you want (it is recommended to do it from the higher number first not to have to check the numbered list again):


sudo ufw status numbered
sudo udw delete <RULE_NUMBER>

You can also check all the iptables entry added:


sudo iptables -S

Further reading: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-with-ufw-on-ubuntu

ufw-docker

As we intend to run docker commands on this VPS, we want to make sure our firewall rules are not bypassed by Docker. This is currently the case; if we run a service that expose a port, that port is not blocked by ufw as Docker modified alternate rules with iptables.

https://github.com/chaifeng/ufw-docker both details the issue and propose a solution to update the host’s iptables rules to fix this issue.

We will install it and use use it by following the instructions provided on the GitHub repo:


# Download the command
sudo wget -O /usr/local/bin/ufw-docker \
  https://github.com/chaifeng/ufw-docker/raw/master/ufw-docker
sudo chmod +x /usr/local/bin/ufw-docker

# Install the iptables
sudo ufw-docker install

# Restart ufw for those to take effect
sudo systemctl restart ufw

Adding a container to the ufw rules is handled differently. From their example:

Expose the 443 port of the container httpd and the protocol is tcp ufw-docker allow httpd 443/tcp

As such: sudo ufw-docker allow containername port/protocol

Taking the example of wg_easy (”The easiest way to run WireGuard VPN + Web-based Admin UI”), to expose the Wireguard port (51820) only through our ufw setup (denying access by default), we would use:


# Add the wg-easy (running container name)'s port 51820 with udp to ufw 
sudo ufw-docker allow wg-easy 51280/udp
# if all goes well, we will see an output prompt ending in: Rule added

# To confirm the rule is active, use ufw
sudo ufw status
# will display a new entry with the docker container's IP; for example:
# 172.20.0.2 51280/udp       ALLOW FWD   Anywhere

firewalld

An alternate solution to ufw-docker as found from their issue board might be https://gist.github.com/deploy595/205ea7985fbf41fe66ab9a082021ed6a

fail2ban

Install and prepare a jail.local file:


sudo apt install fail2ban
cd /etc/fail2ban
sudo cp jail.conf jail.local

Here is a list of possible modifications to sudo nano jail.local:

In the [DEFAULT]section, modify the bantime, findtime and maxretry to match how aggressively you want to block attempts:


# "bantime" is the number of seconds that a host is banned.
bantime = 15m

# A host is banned if it has generated "maxretry" during the last "findtime" seconds.
findtime = 15m

# "maxretry" is the number of failures before a host get banned.
maxretry = 4

In the JAILS section, enable the ones you want. Here we will enable the [sshd] section by adding enabled=true in that section, modify the mode, and change the default port to match our ssh port change for iptables to work properly


[sshd]
enabled=true
mode = aggressive
port = 1327
logpath = %(sshd_log)s
backend = %(sshd_backend)s

Enable, start, and check the service:


sudo systemctl enable fail2ban
sudo systemctl start fail2ban
sudo systemctl status fail2ban

The version installed on your host may give you an error, as reported on GitHub. Please see https://github.com/fail2ban/fail2ban/issues/3755#issuecomment-2150347095 to fix it.

After a successful start, you should see something like:


sudo systemctl status fail2ban
● fail2ban.service - LSB: Start/stop fail2ban
     Loaded: loaded (/etc/init.d/fail2ban; generated)
     Active: active (running) since ...

If you sudo tail -f /var/log/fail2ban.log you should see the status and services loaded by fail2ban

If you have another host on another public IP accessible to you (so you do not block yourself for the configured bantime of 15 minutes), you can attempt to get it banned to test the tool, ssh fakeuser@<VPS-DETAILS> -p 1327 a few times, and you will see a [sshd] Ban <IP> entry in the tail as well as new content when using sudo iptables -S | grep f2b :


-N f2b-sshd
-A INPUT -p tcp -m multiport --dports 1327 -j f2b-sshd
-A f2b-sshd -s <IP>/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN

After bantime, you will see a [sshd] Unban <IP> in the log.

Further reading: https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-22-04

reboot

Confirm all your setup will continue to work after a system update, sudo reboot -h now then, after the VPS reboots, confirm that ufw, fail2ban and iptables are configured and running:


sudo ufw status
sudo service fail2ban status
sudo iptables -S

Further Hardening

Automatic security updates are also recommended using “unattended-upgrades” (with an email sent when a reboot is a necessary or automatic reboot enabled). Ubuntu Pro can also be considered. For further details, please refer to “Unattended Upgrades” and “Sending emails: Postfix using Fastmail.”

If you expect your server to report its status daily, you can consider using logwatch (which works best if the host can send emails).

It is also possible to run some tests on this system's hardening at intervals: security audit and rootkit scanning, using lynis and rkhunter for examples. The details below provide limited details, as the configuration complexity you require depends on your needs. Please refer to each tool's documentation for further instructions. It is also possible to use a tool such as tripwire, a file integrity monitoring tool, to detect and alert you to unauthorized changes in files or directories, helping you maintain system security and compliance.

Logwatch

Logwatch is a log analysis and monitoring tool for Linux systems that parses and summarizes system logs, making it easier to identify potential issues, security threats, and performance trends. The daily email is also useful to confirm your VPS is operational.

Install and prepare for configuration:


sudo apt install logwatch
sudo mkdir /var/cache/logwatch
sudo cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/

Then sudo nano /etc/logwatch/conf/logwatch.conf and alter the file according to your needs (please see the comments below to help you decide) :


## if your VPS can send emails out, set your preferred from and to accordingly
Output = mail
Format = HTML
MailTo = [email protected]
MailFrom = [email protected]
# comment the "mailer" line when using "postfix"
## otherwise go through the configuration file to store results in files that can be consulted manually
Output = file
Format = text
Filename = /tmp/logwatch

## Common
# Adapt after testing if you want more detailed content
Detail = Low
Service = All

Manually test the different levels to find the one you prefer:


sudo logwatch --detail High --range today

Security Auditing

Lynis is an open-source security auditing tool for Unix-based systems that performs a comprehensive security scan directly on the system. It analyzes various aspects of its configuration and identifies potential vulnerabilities. It also tests security defenses and provides actionable recommendations for system hardening and compliance testing.

To install and run a security audit using lynis


sudo apt install lynis
sudo lynis audit system

At the end of this run, you will have many recommendations on hardening the system further. Some, like “Set a password on GRUB boot loader” might not be applicable in a VPS, but you should consider if some of the proposed “Consider hardening SSH configuration” entries should be applied.

Rootkit Scanner

Rkhunter (Rootkit Hunter) is an open-source Unix-based security tool that detects a system's rootkits, backdoors, and local exploits. It works by scanning for known patterns and behaviors associated with malicious software and checking for suspicious system changes and anomalies.

To install and run a rootkit audit using rkhunter


sudo apt install rkhunter
sudo rkhunter --check

Revision History

20240922-0: Added ufw-docker content

20240622-0: Discussion on ssh ciphers and logwatch

20240619-0: Clarifications, adding references, formatting

20240616-0: Initial draft following VPS setup

Dockge (20240908) Post-Steps: Ubuntu 24.04 Desktop (20240730)

Martial

u/geekierone @[email protected]

Ubuntu24

Post-Steps: Ubuntu 24.04 Desktop (20240730)

Martial

May 25, 2024

This is a more opinionated post than usual: I live on the command line on Linux. As such, I perform a few extra steps after a fresh installation to get to a “usable” system.

Guide Ubuntu22 Ubuntu24

Unattended Upgrades on Ubuntu Linux (20240622)

Martial

Mar 12, 2024

"Unattended Upgrades" is a package available on Ubuntu systems that automatically installs updates for security and, optionally, other software packages. This tool is crucial for maintaining system security and stability. It ensures that vulnerabilities and bugs are promptly addressed without requiring manual intervention. This post details the setup instructions for using it for security updates on an Ubuntu Linux 22.04 or 24.04 server and sending emails when the updates are completed.

Guide Linux Ubuntu22 Ubuntu24 Postfix FastMail

Sending emails: Postfix using Fastmail (20240531)

Martial

Mar 12, 2024

Postfix is a free, open-source mail transfer agent (MTA) used to route and deliver email over the Internet. FastMail is a paid service that provides a secure, privacy-focused email provider supporting custom domains. This post details the setup instructions for using Postfix as a Send-Only SMTP server, using Fastmail, on a Ubuntu Linux 22.04 server.

Ubuntu 24.04 VPS Hardening (20240922)

Remote Access Setup

Changing the default user

Hardening ssh

A note on the default list of authorized Ciphers

Secure DNS

DNS-over-HTTPS using cloudflared

DNS-over-TLS using Quad9

Preventing access and limiting entry

ufw

ufw-docker

firewalld

fail2ban

reboot

Further Hardening

Logwatch

Security Auditing

Rootkit Scanner

Revision History

Related Posts

Post-Steps: Ubuntu 24.04 Desktop (20240730)

Unattended Upgrades on Ubuntu Linux (20240622)

Sending emails: Postfix using Fastmail (20240531)