A Focal Fossa in a Digital Ocean
DigitalOcean is an excellent service provider. Focal Fossa is an easy-to-use Linux distribution. This is a quick introduction on how to securely set up a server on Digital Ocean.
This is a very short, quick and dirty guide to setting up a first server. Basically this servers as a memory aid for the auther, and most of the content is inspired by tutorials from DigitalOcean.
Getting a Server
DigitalOcean is definitely an amazing service provider for developers, from the beginner to the expert. I suggest creating an account with them and getting a cheap droplet to start on your journey.
To get started with web development, simply create a Ubuntu 20.04 Focal Fossa LTS droplet. For now, allowing root access with a password is just fine, a key-based authentication can be set up later on.
Once the droplet is created, use ssh, for example Putty on Windows, or OpenSSH on Linux, and log in to the server as root.
A Non-Root User
The root user has very broad privileges and can pretty much do whatever it wants. It is thus not very prudent to log in as the root user all the time. Thus, the first thing to do now, is to create another user using the adduser command (replace symplectos with whatever username you desire, obviously):
adduser symplectos
To allow the newly created user to sometimes carry out administrative tasks as root, the newly created used has to be added to the superuser do group, or sudo, for short. This can be done with the usermod command:
usermod -aG sudo symplectos
Firewall
To protect the newly created server against intrusions, a basic firewall should be set up.
The DigitalOcean droplet comes installed with Uncomplicated Firewall, or UFW, for short.
By default, UFW blocks all incoming and outgoing requests. A very handy feature of UFW is that it recognized standard applications and thus allows for very easy management of common firewall rules. Please note that all the following commands must be run as root or with superuser do.
To see a list of all available applications recognized by UFW, simply type
ufw app list
Since this is a newly installed server, the output should look as follows:
Available applications:
OpenSSH
Obviously ssh traffic should be allowed to pass through the firewall, else it will be impossible to remotely log in to the server at all. To allow an application to go through the firewall, simply run the following command:
ufw allow OpenSSH
To finally enable the firewall, type:
ufw enable
The status of UFW can be checked via the following command:
ufw status
In the current context, the output should look as follows:
Status: active
To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
OpenSSH (v6) ALLOW Anywhere (v6)
SSH Keys
To further harden the security of the server, I suggest disabling remote access for the root user and to only allow key-based authentication for the newly created non-root user.
Create SSH Keys with OpenSSH
The standard OpenSSH installation contains a tool to create new SSH keys. By default, a 2048-bit RSA key pair is created.
ssh-keygend -f ~/.ssh/symplectos-key-rsa -t rsa -b 4096
Elliptic curve algorithms are more modern. The Elliptic Curve Digital Signature Algorithm (ECDSA), with a key size of 521 bits, can be used as follows:
ssh-keygen -f ~/.ssh/symplectos-key-ecdsa -t ecdsa -b 521
I suggest using a passphrase to protect the key.
To use public key authentication, the public key must be copied to the server and installed as an authorized key. A convenient way to do so, is to use the ssh-copy-id command as follows:
ssh-copy-id -i ~/.ssh/symplectos-key-ecdsa symplectos@bell0bytes.eu
Once the public key has been configured on the server, any user in possession of the private key will be allowed to log in remotely.
If the ssh-copy-id is not available, proceed to the Using an Already Existing Key section to proceed.
Creating SSH Keys with Putty
To create ssh keys with Putty, puttygen must first be downloaded from the Putty website.
Once downloaded, start the program and create the keys. The same remarks as above, i.e. ECDSA is probably better than RSA, do apply.
To add the private key to the server, proceed to the next section.
Using an Already Existing Key
There are two options to upload an already existing key, not counting the ssh-copy-id command.
Piping Into SSH with Password-Based Access
Make sure the ~/.ssh folder exists on the remote server, and then append the piped contents of the public key into the ~/.ssh/authorized_key file as follows:
cat ~/.ssh/symplectos.pub | \
ssh symplectos@bell0bytes.eu "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
After entering the password to authenticate the user, the key will be copied and key-based access will be available henceforth.
Manually without Password-Based Access
On the local machine, output the contents of the public key, either via puttygen or, on Linux, by using cat, for example, and copy the output:
cat ~/.ssh/symplectos.pub
On the server, create the ~/.ssh directory:
mkdir ~/.ssh
To append the public key to the authorized_keys file, open that file with a text editor and paste the above copied content into the file:
echo "ssh-rsa EXAMPLE...== symplectos@bell0bytes.eu" \
>> ~/.ssh/authorized_keys
Note, change ssh-rsa to ssh-ecdsa if using the ECDSA algorithm.
Last but not least permissions for the folder and the authorized_keys file must be set to 700 and 600 respectively. Also, make sure the non-root user owns the directory and the files. To do so, run the following command with superuser do:
chmod -R go= ~/.ssh
chown -R $USER:$USER ~/.ssh
Configuring SSHD
To restrict ssh access to only those users within a specific group, create that group and add the desired users:
sudo groupadd sshusers
sudo usermod -a -G sshusers symplectos
To configure the sshd service, a configuration file can be created in the /etc/ssh/sshd_config.d/ folder.
The following example disabled all password-based access. For an explanation of each parameter, check the official manpage
sudo vim /etc/sshd_config.d/bell0bytes.conf
# set log level to verbose to have a clear audit track of which key was used to login
LogLevel VERBOSE
# don't let users set environment variables
PermitUserEnvironment no
# only user the newer, more secure protocol
Protocol 2
# disable X11 forwarding
X11Forwarding no
# disable port forwarding
AllowTcpForwarding no
AllowStreamLocalForwarding no
GatewayPorts no
PermitTunnel no
# don't allow login if the account has an empty password
PermitEmptyPasswords no
# don't allow password auth
PasswordAuthentication no
# disallow root login
PermitRootLogin no
# ignore .rhosts and .shosts
IgnoreRhosts yes
HostbasedAuthentication no
# verify hostname matches IP
UseDNS no
# disable compression after login
Compression no
# send keep alive messages
TCPKeepAlive no
# disallow agent forwarding
AllowAgentForwarding no
# allow only certain user groups
AllowGroups sshusers
# maximum number of client alive messages sent without response
ClientAliveCountMax 0
# timeout in seconds before a response request
ClientAliveInterval 300
# local addresses sshd should listen on
# ...
# time in seconds before login times out
LoginGraceTime 30
# maximum allowed attempts to login
MaxAuthTries 3
# maximum number of open sessions
MaxSessions 2
# maximum number of login sessions
MaxStartups 3
After restarting the sshd service
sudo systemctl restart sshd
it should be possible for the non-root user to log in to the server with key-based access.
Intrusion Detection with PSAD
PSAD, also known as Port Scan Attack Detector, is a collection of lightweight system daemons that run on Linux systems and analyze the log messages of iptables to detect port scans and other suspicious traffic. PSAD is used to change an Intrusion Detection System into an Intrusion Prevention System.
A serverfault user, FINESEC, explained this quite well:
Fail2BAN scans log files of various applications such as apache, ssh or ftp and automatically bans IPs that show the malicious signs such as automated login attempts. PSAD on the other hand scans iptables and ip6tables log messages (typically /var/log/messages) to detect and optionally block scans and other types of suspect traffic such as DDoS or OS fingerprinting attempts. It's ok to use both programs at the same time because they operate on different level.
Note: See the next section for Fail2Ban instructions.
Installating PSAD
To install psad, simply run
sudo apt install psad
Configuring PSAD
Configuration is done in the /etc/psad/psad.conf file. Check the official docs for a list of all possible parameters.
HOSTNAME
This parameter specifies the name of the server, i.e.
HOSTNAME bell0server;
HOME_NET
This parameter can be used to define a home network, i.e. a safe network, whose IP addresses should not be considered as attackers. Networks should be specified in CIDR notation and multiple networks can be given as a comma separated list:
HOME_NET 192.168.178.0/24, 2001:7e8:c87b:d800::/64;
This setting might be fine for a homelab, but for a droplet on Digital Ocean, I am not completely comfortable with setting the entire IP range as a HOME_NET.
ALERTING_METHODS
This parameter specifies how PSAD is supposed to send alerts. Since I will eventually set up a monitoring tool, I usually set this parameter to noemail:
ALERTING_METHODS noemail;
ENABLE_PSADWATCHD
If set to Y, PSAD will watch init services.
ENABLE_AUTO_IDS
If this parameter is set to Y, PSAD will automatically block malicious IPs by adding rules into iptables.
EXPECT_TCP_OPTIONS
By setting this parameter to Y, TCP information will be logged and analyzed as well, which helps to detect certain scan or attack signatures.
IPT_SYSLOG_FILE
This parameter specifies the location of the syslog file, which should be set to /var/log/syslog on Ubuntu.
AUTO_BLOCK_TIMEOUT
This parameter specifies for how long IPs are banned. The default is an hour, i.e. 3600s.
Configuring UFW
To make UFW play nice with psad, tell UFW to log all traffic in such a way that psad can analyze it. This can be done by adding the following line to the two files before.rules and before6.rules in the /etc/ufw directory, at the very end of the files, just before the COMMIT:
# log all traffic so psad can analyze it
-A INPUT -j LOG --log-tcp-options --log-prefix "[IPTABLES] "
-A FORWARD -j LOG --log-tcp-options --log-prefix "[IPTABLES] "
Reload UFW and PSAD
Reload UFW and PSAD by typing:
sudo systemctl restart ufw
sudo psad -R
sudo psad --sig-update
sudo psad -H
Analyize and Check
To analyze traffic, simply run:
sudo psad --fw-analyze
[+] Parsing INPUT chain rules.
[+] Parsing INPUT chain rules.
[+] Firewall config looks good.
[+] Completed check of firewall ruleset.
[+] Results in /var/log/psad/fw_check
[+] Exiting.
And to check the status of psad, run:
sudo psad --Status
The output might look like this:
[+] Top 50 signature matches:
"P2P Napster Client Data communication attempt" (tcp), Count: 4, Unique sources: 2, Sid: 564
"MISC Microsoft SQL Server communication attempt" (tcp), Count: 2, Unique sources: 1, Sid: 100205
"POLICY HP JetDirect LCD commnication attempt" (tcp), Count: 2, Unique sources: 1, Sid: 568
"SCAN UPnP communication attempt" (udp), Count: 1, Unique sources: 1, Sid: 100074
[+] Top 25 attackers:
117.146.123.112 DL: 2, Packets: 1, Sig count: 1
118.44.200.87 DL: 2, Packets: 3, Sig count: 3
49.128.174.248 DL: 2, Packets: 2, Sig count: 2
79.124.62.90 DL: 2, Packets: 12, Sig count: 2
89.248.165.164 DL: 2, Packets: 1, Sig count: 1
195.54.161.151 DL: 1, Packets: 6, Sig count: 0
45.146.165.25 DL: 1, Packets: 7, Sig count: 0
Unblocking
To unblock all the IP addresses blocked by PSAD, ru the following command:
psad -F
To unban a specific IP address, run:
psad --fw-rm-block-ip 192.168.178.1
Enabling Fail2Ban
PSAD monitors network activity to detect and prevent potential intrusions. Fail2Ban can be used to protect additional applications or services that might run on a server, such as ssh, apache or nginx.
Fail2Ban is a free and open source Intrusion Prevention Software tool written in Python. The main purpose of Fail2ban is thus to scan the log files of various services, such as SSH, FTP, SMTP, Apache or NGINX, and to block those IP addresses that perform too many suspicious actions, such as failed login attempts or script injection attempts.
If key-based access is enabled, it is still a good idea to activate fail2ban, especially since it can later on be used to safeguard apache and nginx webservers as well.
To install fail2ban, simply run
sudo apt-get install fail2ban
Configuring fail2ban is done using a local jail file (to be sure that a package upgrade won't overwrite local changes). To get started, copy the already existing conf file:
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
The first thing to configure is to tell fail2ban to use UFW instead of iptables, and to add any local, or safe, IP addresses to be ignored, meaning that access from these addresses will never be banned. Usually this should be set to the loopback addresses as well as any internal addresses:
ignoreip = 127.0.0.1/8 ::1 192.168.178.0/24 2001:7e8:c87b:d800::/64
banaction = ufw
Once done, enable and start fail2ban:
sudo systemctl enable fail2ban
sudo systemctl start fail2ban
By default, fail2ban protects sshd by parsing its log file, which is located at /var/log/auth.log. To check the filter fail2ban is using to parse the lock, open the /etc/fail2ban/filters.d/sshd.conf file. Note that to each jail, defining the actions to be taken on an infraction, corresponds a filter file, with the same name, specifying how to parse a specific log file.
To check the status of fail2ban and all its jails, the fail2ban-client can be used as follows:
sudo fail2ban-client status
sudo fail2ban-client sshd
Status
|- Number of jail: 1
`- Jail list: sshd
Status for the jail: sshd
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/auth.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
Manually Banning and Unbanning IPs
To ban or unban IP addresses, the unbanip and the banip commands be used, respectively:
sudo fail2ban-client set sshd unbanip 123.456.789.012
sudo fail2ban-client set sshd banip 123.456.789.012
Fail2Ban in Action
As a note: In about a year, Fail2ban blocked 2753 IP addresses trying to access my home server:
Status for the jail: sshd
|- Filter
| |- Currently failed: 0
| |- Total failed: 127
| `- File list: /var/log/auth.log
`- Actions
|- Currently banned: 2753
|- Total banned: 2753
