← Nabla

Notes · operations

Someone has to own the servers.

Many people I’ve worked with saw the system administrator, or the DevOps engineer, as someone who changes DNS records and restores a backup when asked. That’s a small part of the job. The rest starts with the first server, long before a company can justify hiring someone full time to do it. This is the list I hold the role to, whoever ends up doing the work.

  1. 01

    Access and secrets

    VPN, SSH, accounts and the permissions behind them. Everyone gets a personal account with only the rights they need. With shared accounts and passwords nobody can tell who did what, and one person can’t be locked out without locking out everyone.

    When someone leaves, their access goes the same day and every secret they knew is changed. Keep a list of who has access to what, so this takes minutes.

    Database passwords, API keys and certificates end up in .env files, pipeline settings and vendor dashboards. Each one needs an owner, one place where it’s kept, such as a password manager or a secrets vault, and a date when it’s next changed. It’s changed at once if it turns up somewhere it shouldn’t, such as a commit.

    Log who logs in and who runs sudo, and keep those logs somewhere other than the server itself, so you can tell afterwards who did what.

    For a model An agent with shell access is one more account. Give it its own credentials and only the rights the task needs, so you can see what it did and revoke it on its own.

  2. 02

    Patching and end of support

    Operating systems and runtimes get security updates until the vendor stops supporting them. Track the end-of-support date of everything that runs, and plan each upgrade before its date arrives.

    Check that updates really get installed. A server can go years without a single package change, and if its update sources have moved or been shut down, it can’t receive updates at all. Nobody notices until someone looks at the date of the last installed package.

    Turn on automatic security updates for the operating system, for example unattended-upgrades on Debian, so patching doesn’t depend on anyone remembering. Operating system updates don’t cover a runtime installed from a third-party repository, the application’s Composer or npm packages, or container base images. Each of those needs its own updates. Try every larger upgrade on staging first.

  3. 03

    Monitoring and alerting

    Uptime checks, central logs and alerts on errors. When something breaks, the admin should hear about it before the users do. If users keep finding outages first, the monitoring doesn’t work, whatever agent is installed. It should also catch the slow problems: a filling disk, an expiring certificate, updates that stopped installing.

    Watch capacity and cost too. Disk, memory and the hosting bill grow slowly, so look at the trend every month and act before a limit or a budget is reached.

  4. 04

    Backup and recovery

    A schedule, a copy kept somewhere else, and regular restore tests. Nightly snapshots on the same infrastructure don’t help if that infrastructure is lost. A restore test shows whether the backups work and how long a restore takes. Write down when the last one was done.

    Agree with management how much data the business can afford to lose and how long it can be down. Those two numbers set how often to back up and how fast a restore has to be.

  5. 05

    Rebuilding and environment parity

    Every server should be rebuildable from configuration kept in version control: Ansible, Terraform, a Dockerfile, or at least a script. Nobody dares to upgrade a server that was set up by hand over years, and if it’s lost, someone has to reconstruct it from memory.

    The same configuration then builds development, staging and production, so they run the same operating system, runtime versions and web server. Copying data and files between environments helps only if the software underneath matches. Otherwise bugs that depend on versions show up first in production.

  6. 06

    Deployment and CI/CD

    Pipelines, configuration and environments. Changes reach production through the pipeline. If someone has to fix production by hand, for example by clearing a cache the deploy forgot, the pipeline is missing a step, and adding it is part of the fix.

    Developers write the application and often the pipeline too. The admin sets up the servers to receive it and makes sure nobody deploys around it. See also deploying through CI/CD.

  7. 07

    Network, DNS and TLS

    DNS, the reverse proxy, the firewall, the VPN and certificates. Most people already think of this as the job. Keep the configuration in version control, so a change can be reviewed and rolled back.

  8. 08

    Documentation and vendors

    An inventory of every server: its operating system, runtimes and versions, what runs on it and who owns it. Runbooks for the common jobs.

    A list of hosting, licences, SaaS accounts and support contracts, with what each one costs, when it renews and who to call when it fails. Put the renewal dates where someone will see them in time. If the vendor contacts live in one person’s head, the service isn’t 24/7/365, because that person sleeps and goes on holiday.

  9. 09

    Personal data

    Know where personal data is kept, backups and offsite copies included, which country it’s in and how long it stays there. GDPR covers the copies as well as the live database.

  10. 10

    Incident response

    Fast, accurate triage is where it starts. After an incident, write down what happened and why, and open an issue for whatever will stop it happening again, without waiting to be asked.

Whose job is it?

Where nobody wrote down who owns what, requests get bounced with “not my area”. Some of those are fair. Application code, its configuration values and its data model belong to the developers. The platform belongs to the admin: servers, operating systems, runtimes, network, backups and monitoring.

Write that boundary down. Until someone does, it’s a management gap, and whoever runs the servers owns the platform in practice, whatever their job description says.

A boundary still leaves both sides looking across it. Developers shouldn’t be blind to the servers their code runs on, and the system administrator shouldn’t be blind to what the developers build and how it gets deployed. The role was renamed DevOps for this reason.

Holding the access brings one more duty. Whoever alone knows the state of the servers has to put the risks in front of management, and repeat it until someone decides what to do. Put it in the tracker, so there’s a record of who was told and when. See one issue is one issue.

Being quick with requests is good, and many admins are. An admin can still close every request within the hour and leave the servers years behind on updates. Most of the list above is work nobody requests: patching, restore tests, the inventory and the warnings.

Who can afford one?

The work is needed almost as soon as the first server is running, and the list above applies to one server as much as to a thousand. A full-time sysadmin or DevOps engineer, though, is rarely cost effective until you’re running around a thousand servers. In most small and mid-sized companies someone still has to do the work. Name that person, and have them plan it, understand it and document it.

Serverless partially fixes the problem. The cloud provider takes over the servers, the operating systems and most of the patching, although you still have to move to a new runtime version each time the provider retires the old one. It also opens a whole new can of worms.

The recipe is the same as for servers. Plan every service before it’s turned on, understand what it does and what it costs, and document it. Most of the list above still applies, with the cloud account in place of the server.

Questions to ask your admin

Nabla, Samo Fortuna s.p.