If you manage more than one server, you know the drill. One box lives on a non-standard port, another needs a specific key, and a third sits behind a bastion host. Pretty soon your shell history is a graveyard of commands like ssh -i ~/.ssh/prod_ed25519 -p 2222 [email protected], and one typo in an IP address sends you to the wrong machine, or nowhere at all.

The OpenSSH client has a fix for this, and it has been there for decades: the SSH config file, ~/.ssh/config. You write the details once, and from then on you type ssh web while the client fills in the user, the port, the key, and the jump host. It also works with scp, sftp, and rsync, because they all read the same file. If you have not set up keys yet, do that first: see how to set up SSH keys on Linux.

Prerequisites

You need the OpenSSH client, which is already installed on practically every Linux desktop and server, and at least one server you can reach. You also need to know, for each server, its address, your username on it, and which key (if any) it expects. Everything in this guide runs on your own machine. Nothing here changes the server.

Step 1: Create the file with the right permissions

The client config file does not exist by default. Create it, then lock it down so only you can read and write it:

BASH
mkdir -p ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/config
chmod 600 ~/.ssh/config

The permissions matter. OpenSSH ignores a user config file that other users can write to, and tells you why with a Bad owner or permissions error. The failure mode is easy to miss: your aliases simply stop working, and you are back to typing full commands with no obvious explanation. Keeping the file at 600 removes the whole class of problem.

This is the client file, ~/.ssh/config, not /etc/ssh/sshd_config, which configures the server daemon. Editing it changes nothing about how your machine accepts incoming connections. Confusing the two is the most common beginner mistake with SSH configuration: ~/.ssh/config is about how you connect out.

Step 2: Write your first host entry

Open the file in your editor and add one entry per server. Each entry starts with a Host line, which is the nickname you will type, followed by indented options:

TEXT
Host web
    HostName 203.0.113.10
    User deploy
    Port 2222
    IdentityFile ~/.ssh/id_ed25519

That is the whole trick. ssh web now means exactly what the long command used to mean: connect to 203.0.113.10 as deploy, on port 2222, offering the key ~/.ssh/id_ed25519. You can verify the client understood it without connecting to anything:

BASH
ssh -G web

The -G flag prints the effective configuration for that nickname: hostname, user, port, identity files, and every default the client adds. I ran it against the entry above and the output confirmed hostname 203.0.113.10, user deploy, and port 2222, exactly as written. If -G shows the wrong values, your file has a typo or an ordering problem, and you will find out here instead of at login time. Make -G your first stop whenever an alias misbehaves.

Step 3: Add more hosts, wildcards, and defaults

There is no limit on entries, and the file stays readable because each host is just a short stanza. Add a second server the same way:

TEXT
Host db
    HostName 203.0.113.11
    User postgres
    IdentityFile ~/.ssh/id_ed25519

Patterns make the file scale. Host *.internal matches every nickname ending in .internal, and Host * matches everything, so it is the right place for defaults that apply to all your servers:

TEXT
Host *.internal
    User admin
    ProxyJump bastion

Host *
    ServerAliveInterval 60
    AddKeysToAgent yes

Two rules govern how entries combine. First, the client reads the file top to bottom and the first value it finds for each option wins. That means specific entries must come before general ones: put Host web above Host *, or the wildcard will claim the option first and your specific value will never apply. Second, options from different matching stanzas merge, so db.internal picks up User admin and ProxyJump bastion from the wildcard stanza and ServerAliveInterval 60 from the catch-all. I confirmed both with ssh -G db.internal: it reported user admin and proxyjump bastion, while an unknown host still picked up serveraliveinterval 60 from the Host * block. The first-match rule is the one to internalize; it causes most "my config is ignored" confusion.

Step 4: Reach servers behind a bastion with ProxyJump

The stanza above already shows the pattern that saves the most typing in real infrastructure: the jump host. If your database server only accepts SSH from a bastion, ProxyJump bastion connects to the bastion first and tunnels through it, with a single command. The bastion itself needs its own entry:

TEXT
Host bastion
    HostName 203.0.113.5
    User ops
    IdentityFile ~/.ssh/id_ed25519

Two more options belong in almost every multi-server file. ServerAliveInterval 60 sends a keepalive packet every minute, which stops idle connections from being silently killed by NAT routers and firewalls, the classic cause of a frozen terminal after lunch. AddKeysToAgent yes hands your key to ssh-agent on first use, so you type your key passphrase once per session instead of once per connection. If you also want a tunnel, LocalForward maps a port through the connection: LocalForward 5433 localhost:5432 under Host db lets a local database client reach the remote Postgres on localhost:5433.

Step 5: Verify the whole setup end to end

Before you delete your old shell aliases, test every nickname once:

BASH
ssh -G web
ssh -G db.internal
ssh web
ssh db.internal

The -G checks catch typos and ordering mistakes. The real connections confirm the key, the port, and the jump path all work together. If the server prompts for a password where it used to accept your key, the entry is pointing at the wrong IdentityFile, or the server's authorized_keys does not contain that key. Fix it in the config file, not by going back to long commands. Once the checks pass, confirm your habits still work: scp file.txt web:/tmp/ and rsync -av ./site/ web:/var/www/ should both resolve the nickname, because both tools read the same config file.

Five mistakes that break the file

Most SSH config problems come from a short list:

  1. Wrong file. ~/.ssh/config is the client file. /etc/ssh/sshd_config is the server. Editing the server file will never fix a client alias.
  2. Loose permissions. The file must not be writable by anyone but you. Keep it 600 and the directory 700.
  3. Specific entries below Host *. First match wins, so put exact hostnames at the top and wildcards at the bottom. The moment you add a Host * block, everything under it that repeats an option is dead text.
  4. Indentation and typos in option names. Options are case-insensitive but must be spelled correctly; Hostname works, Host Name does not, and a stray space in IdentityFile silently changes the path.
  5. A key path that does not exist. IdentityFile pointing at a missing file does not produce an error at parse time; the client simply offers nothing and falls back to password auth. ssh -G shows you exactly which files it will offer, so check there first.

For the server side, the audit checklist in the SSH hardening playbook covers key-only login and the ports that should never be open. And if your sessions keep dropping, tmux survives dropped SSH sessions so a dead connection costs you nothing.

The payoff

A good config file is boring in the best way. Your commands shrink from a line of flags to a single word, your keys and ports stop living in your shell history, and every tool that speaks SSH inherits the same setup for free. The file is plain text, it travels with your dotfiles, and it works the same on every machine with OpenSSH installed. Spend the half hour once, and every server you ever touch gets easier.

Sources