Linux Technology

How to Set Up a Secure SSH Server: A Beginner's Guide

Updated on
9 min read

Introduction

Secure Shell (SSH) is essential for encrypted remote access to Unix-like systems. This beginner’s guide provides clear, step-by-step instructions on how to install, configure, and secure an OpenSSH server for safe remote administration. Whether you’re managing a production host, a virtual private server (VPS), or a home lab setup, this article offers practical commands, configuration examples, and security tips to mitigate common threats like brute-force attacks and misconfigurations. Ideal for IT beginners, DevOps practitioners, and hobbyists, this guide ensures you can confidently secure your SSH server.

SSH Basics: How SSH Works

At its core, SSH operates using a client-server model, where the SSH client establishes a connection to the SSH daemon (sshd) server. Utilizing SSH version 2 (the secure protocol), it facilitates secure key exchange, encryption, integrity protection, and authentication. For a detailed understanding, refer to RFC 4251 — The Secure Shell (SSH) Protocol Architecture.

Authentication Methods

1. Password Authentication: The client transmits a password over an encrypted channel. While easy to set up, it’s vulnerable to brute-force attacks.

2. Public-Key Authentication: Here, the client proves ownership of a private key that corresponds to a publicly available key mentioned in the server’s ~/.ssh/authorized_keys. This method is highly recommended as the password does not traverse the network.

Modern best practices advocate for strong key algorithms (ED25519 or RSA 3072/4096) using passphrases and hardware-backed keys (like YubiKey).

Preparation: Before You Start

  1. Choose Your Host and OS
    Opt for a supported Linux distribution (Debian/Ubuntu, RHEL/CentOS, or Fedora). If you’re building a home lab, check out the hardware guidance in this article: Building a Home Lab: Hardware Requirements (Beginners).

  2. Update Packages

    • Debian/Ubuntu
    sudo apt update && sudo apt upgrade -y
    • RHEL/CentOS/Fedora
    sudo yum update -y  # or: sudo dnf upgrade --refresh -y

  3. Plan Administrative Access
    Create a non-root administrative user with sudo and confirm the ability to switch accounts.

    sudo adduser adminuser  
sudo usermod -aG sudo adminuser  # Debian/Ubuntu  
# or for RHEL-based systems  
sudo usermod -aG wheel adminuser

  4. Out-of-Band Console and Backups
    If working with a remote VPS, ensure access to an out-of-band web console or rescue mode to avoid lockout. Back up your current SSH configuration:

    sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak-$(date +%F)

    Always document changes for easy rollback.

Install and Verify OpenSSH Server

To install the OpenSSH server package and enable the service, follow these steps:

  • Debian/Ubuntu 
    sudo apt install -y openssh-server  
sudo systemctl enable --now ssh
    • RHEL/CentOS/Fedora
    sudo yum install -y openssh-server  # or: sudo dnf install -y openssh-server  
sudo systemctl enable --now sshd

To verify the service status and listening ports, use:

sudo systemctl status sshd  # or: sudo systemctl status ssh  
ss -tlnp | grep sshd  
sudo journalctl -u sshd -e

Note: the default SSH port is 22. Changing the port can minimize noisy scans but does not replace robust authentication.

Set Up Key-Based Authentication

Public-key authentication is more secure than passwords. Follow these steps for key generation and deployment:

  1. Generate a Key Pair on Your Client
    • Recommended: ED25519
    ssh-keygen -t ed25519 -a 100 -C "[email protected]"  
# -a 100 increases KDF rounds to slow password-guessing on encrypted private keys
    • If compatibility necessitates RSA, ensure a strong size:
    ssh-keygen -t rsa -b 4096 -o -a 100 -C "[email protected]"

Comparison of Key Types

Key TypeStrength / NotesUse Case
ED25519Small, fast, secure by defaultRecommended for most users
RSA 3072 / 4096Widely supported, larger keysUse if ED25519 unsupported
ECDSADeprecated in some implementationsAvoid unless necessary

  1. Copy the Public Key to the Server

    • Easiest method: ssh-copy-id
    ssh-copy-id [email protected]
    • Manual method:
    # On server (as the target user)  
mkdir -p ~/.ssh  
chmod 700 ~/.ssh  
# On client  
cat ~/.ssh/id_ed25519.pub | ssh adminuser@server 'cat >> ~/.ssh/authorized_keys'  
# Set permissions  
ssh adminuser@server 'chmod 600 ~/.ssh/authorized_keys && chown -R $(whoami):$(whoami) ~/.ssh'

  2. Protect Private Keys

    • Utilize passphrases on private keys and secure file permissions (600) to protect them. Use ssh-agent and ssh-add for convenience and caching:
    eval "$(ssh-agent -s)"  
ssh-add ~/.ssh/id_ed25519
    • Consider hardware-backed keys (YubiKey, smartcards) for additional assurance.

  3. Securely Backup Keys

    • Maintain an encrypted backup of your private key (e.g., password manager or encrypted archive). If a private key is lost, revoke access by removing the corresponding public key from the servers.

Basic sshd_config Hardening

The primary server configuration file is /etc/ssh/sshd_config. Always test changes prior to restarting the service.

  1. Test Config Syntax

    sudo sshd -t  # prints errors if syntax is incorrect

  2. Recommended Settings
    Add or edit the following lines in /etc/ssh/sshd_config:

    PermitRootLogin no  
PasswordAuthentication no  
PubkeyAuthentication yes  
PermitEmptyPasswords no  
ChallengeResponseAuthentication no  
UsePAM yes  
AllowUsers adminuser anotheruser  
MaxAuthTries 3  
LoginGraceTime 30  
ClientAliveInterval 300  
ClientAliveCountMax 2  
LogLevel VERBOSE  
Banner /etc/issue.net

  3. Cryptography and Algorithms
    OpenSSH provides secure defaults. For comprehensive hardening, refer to the OpenSSH manual pages and the CIS benchmarks.

  4. Apply and Reload Safely
    Maintain one open session while testing changes. If issues arise, you can revert from that session:

    sudo sshd -t  # test  
sudo systemctl reload sshd  # reload to apply changes without dropping existing connections

    If reload fails, use the open session to rectify the configuration.

Network-Level Protections and Anti-Bruteforce Measures

  1. Firewalls: Limit exposure to SSH access

    • UFW (Ubuntu/Debian)
    sudo ufw allow OpenSSH  # allows port 22  
sudo ufw allow from 203.0.113.0/24 to any port 22  
sudo ufw enable
    • firewalld (RHEL/Fedora/CentOS)
    sudo firewall-cmd --permanent --add-service=ssh  
sudo firewall-cmd --reload

  2. Fail2ban to prevent repeated attempts

    • Install and start Fail2ban:
    sudo apt install -y fail2ban  # Debian/Ubuntu  
sudo yum install -y fail2ban  # RHEL/CentOS (EPEL may be required)  
sudo systemctl enable --now fail2ban
    • Modify jail configuration: /etc/fail2ban/jail.d/sshd.local
    [sshd]  
enabled = true  
port = ssh  
maxretry = 3  
bantime = 3600
    • Check Fail2ban status
    sudo fail2ban-client status sshd

  3. Port Knocking / Change Default Port
    Changing the SSH port can reduce random scanning but is not a substitution for strong authentication. Port knocking tools may add further complexity; consider them only if you understand the trade-offs.

  4. Network Segmentation
    For sensitive hosts, bind SSH to internal interfaces or access it exclusively via a VPN/jump host.

User, Key, and Lifecycle Management

  • Carefully manage ~/.ssh/authorized_keys: each line corresponds to one public key entry. Maintain strict permissions (700 for ~/.ssh and 600 for ~/.ssh/authorized_keys).
  • Regularly rotate keys and immediately revoke access for users who leave by removing their public key entry from authorized_keys.
  • In larger environments, centralize user/key management using LDAP/SSSD or automation tools. Refer to this guide on LDAP Integration: Linux Systems (Beginners Guide).
  • Employ configuration management tools like Ansible for consistent distribution of users and keys: see our Configuration Management: Ansible (Beginners Guide).

Monitoring, Logging, and Auditing

Where to Find Logs:

  • Systemd systems: journalctl -u sshd -f
  • Traditional logs: /var/log/auth.log (Debian/Ubuntu) and /var/log/secure (RHEL/CentOS).

Useful Commands:

# Live follow  
sudo journalctl -u sshd -f  
# Search auth log  
sudo grep 'sshd' /var/log/auth.log | tail -n 200

Take note of:

  • Repeated failed authentication attempts from the same IP.
  • Unexpected new public keys added to authorized_keys.
  • Logins by unusual accounts or at odd times.

Integrate logs into a centralized server or SIEM (rsyslog/syslog-ng -> ELK/Cloud provider logs) for history retention and alerts.

Backup, Recovery, and Safe Maintenance

  • Always version your SSH configuration before making changes: 
    sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak-$(date +%F)
  • Keep at least one active session while applying changes as a safety net.
  • Familiarize yourself with your provider’s console/rescue workflow for recovery if remote access fails. Test recovery procedures in non-critical environments.
  • Prepare a documented rollback plan and have an emergency account that you periodically verify.

Automation and Keeping Secure at Scale

Automation curtails human error:

  • Use Ansible or similar tools to manage /etc/ssh/sshd_config, disable root login, and distribute authorized_keys methods across servers. See our Configuration Management: Ansible (Beginners Guide).
  • Automate key rotation and onboarding workflows leveraging CI/CD for configuration changes—test in staging prior to production rollout.
  • Validate changes through a canary host before large deployments.

Troubleshooting Common Problems

1. Verbose Client Output:

ssh -v [email protected]  
ssh -vvv [email protected]  # increases verbosity

2. Common Server-Side Issues:

  • Incorrect permissions: ~/.ssh should be 700; authorized_keys should be 600; ensure files are owned by the user.
  • SELinux contexts (on RHEL/Fedora): Execute restorecon -Rv ~/.ssh to reset contexts.
  • Malformed authorized_keys: Ensure each key is a single line beginning with the key type (e.g., ssh-ed25519).

3. Recovering from Lockout:

  • Utilize your provider’s web console or rescue mode; revert to a backed-up sshd_config if necessary.
  • Avoid disabling password auth until at least one key-based admin user is confirmed to be able to log in.

Conclusion and 10-Point Security Checklist

To summarize, SSH security focuses on minimizing attack surfaces (managing who can connect and how) and ensuring robust authentication and monitoring. Approach security in phases: prepare, enable keys, harden configurations, implement network protections, and automate where feasible.

10-Point Security Checklist

  1. Update the OS and packages.
  2. Create a non-root admin user with sudo access.
  3. Install and enable the OpenSSH server.
  4. Generate and install ED25519 (or RSA 4096) key pairs.
  5. Disable PermitRootLogin and PasswordAuthentication (after verifying keys).
  6. Restrict users with AllowUsers / AllowGroups configurations.
  7. Configure firewall rules and limit access geographically where feasible.
  8. Install Fail2ban and tailor ban thresholds to your needs.
  9. Route SSH logs to a centralized server and conduct regular monitoring.
  10. Back up sshd_config and routinely test recovery procedures.

Test these steps in a virtual machine before proceeding and refer back to the checklist frequently. Subscribe to updates for more beginner-friendly guides on secure system administration.

References and Further Reading

Internal Resources

TBO Editorial

About the Author

TBO Editorial writes about the latest updates about products and services related to Technology, Business, Finance & Lifestyle. Do get in touch if you want to share any useful article with our community.