The job title keeps changing. Systems administrator, Linux engineer, DevOps engineer, site reliability engineer, platform engineer. The work underneath has a stable core. Someone has to understand what the machines are doing, keep them running, change them safely and know what to do when they break.
Here's what that takes in an enterprise in 2024, roughly from the ground up.
1. How Linux actually works
Not the commands. What's underneath them.
- Processes and the kernel. What happens when you run a program: fork, exec, signals, process states, what "uninterruptible sleep" means, why a zombie process isn't using any memory.
- Memory. Virtual memory, the page cache, why "free" memory is supposed to be low, what the OOM killer is doing and how it picks a victim.
- Storage. Block devices, partitions, LVM, filesystems like ext4 and XFS, mount options,
fstab, and what happens when a disk fills up at 2 a.m. - systemd. Units, targets, dependencies, timers, cgroups and the journal. Love it or hate it, it runs nearly every enterprise distribution. Learn to write a unit file from memory and to read
journalctlfluently. - Boot. Firmware, the bootloader, the kernel, the initramfs, then systemd. When a machine won't boot, you have to know which stage failed.
How to build it: break things on purpose in a lab. Fill a disk, kill a process mid-write, delete a library, corrupt fstab, and practice recovering. You learn more from one bad Saturday in a lab than from a month of reading.
2. Networking
Most "the server is broken" tickets are network problems wearing a disguise.
- The fundamentals: IPv4 and IPv6 addressing and routing, TCP and UDP, DNS, DHCP, MTU, and what a three-way handshake looks like in a packet capture.
- The tools:
ip,ss,dig,tcpdump,curl -v,mtrandnft. Being able to say exactly where a connection fails is what makes you valuable. - Enterprise extras: VLANs, bonding, load balancers, proxies, and TLS from the certificate to the cipher suite.
3. Security as a habit
- Least privilege. Scoped sudo, service accounts, sandboxed systemd units. I wrote about this in the least privilege principle.
- SELinux or AppArmor, left on. Learn to read a denial and write a policy fix instead of disabling it. Turning SELinux off is how a lot of breaches get easier.
- Patching on a schedule, with a fast lane for anything internet-facing.
- SSH done right: keys, no root login, no passwords, a bastion or VPN.
- Logging shipped somewhere central, so you can answer "who did what, when."
4. Automation
If you do it twice, script it. If the script matters, put it under version control. If it changes servers, use a tool built for that.
- Shell and Python. Bash for glue and Python for anything with logic, data or an API.
- Configuration management. Ansible is the most common choice in enterprises today. Puppet and Salt are still out there. The point is that servers are built from code, not by hand, so they're consistent and reproducible.
- Infrastructure as code for the cloud and virtualization layer. Terraform, or its open source fork OpenTofu.
- Git for everything above, and the pull-request habit of having changes reviewed before they touch production.
5. Virtualization, containers and cloud
- Virtualization: KVM and libvirt, Proxmox, and VMware, which most enterprises still run and many are reconsidering. See VMware vs Proxmox.
- Containers: how Docker and Podman actually work, which is mostly Linux namespaces and cgroups. Knowing that turns containers from magic into Linux you already understand.
- Kubernetes: if your organization runs it, you'll be expected to debug it. That's mostly Linux and networking skills applied to a different control plane.
- Cloud: at least one major provider, with real fluency in its identity and networking models. That's where most cloud security mistakes happen.
6. Observability and troubleshooting
Monitoring tells you something is wrong. Observability helps you find out why.
- Metrics with Prometheus or similar, logs centralized and searchable, and alerts that fire on symptoms users would notice, not on every CPU spike.
- A method for incidents. Start from what the user sees, form a hypothesis, test it, change one thing at a time and write down what you did. Calm and methodical beats clever and frantic.
- Reading the numbers correctly. Load average, iowait, available memory. My guide to top, htop and glances covers the common misreadings.
7. The skills nobody certifies
These are the ones that get people promoted:
- Writing. Runbooks someone else can follow at 3 a.m., change requests a manager can approve, and postmortems that explain what happened without blaming anyone.
- Change discipline. Know the rollback before you start. Test in staging. Don't change production on a Friday afternoon unless it's on fire.
- Saying "I don't know yet." Then finding out. Confident guesses during an outage cost more than honest uncertainty.
- Teaching. Leaving a team better than you found it, by documenting, automating and explaining.
Certifications
Certifications don't make an engineer, but they're a useful structure for learning and a signal to employers. The ones worth considering:
- RHCSA and RHCE from Red Hat, hands-on and highly respected in RHEL shops.
- LFCS from the Linux Foundation, plus CKA for Kubernetes administration.
- Cloud certifications from AWS, Azure or Google, if your organization lives there.
I've helped write some of the Linux Foundation's exams, so I'm biased toward one feature they share with Red Hat's: they're performance-based. You get a live system and a list of tasks, not multiple choice. That's the right way to test this job, because the job is doing it, not recognizing the right answer in a list.
The one thing that doesn't change
Tools turn over every few years. Upstart came and went. So did Chef's moment in the sun and a dozen container orchestrators before Kubernetes. The engineers who last are the ones who learned how the system works underneath, because that's what lets them pick up the next tool in a week. Learn the fundamentals deeply and the rest comes easily.