Top tools for junior Linux admins

This topic was created by diodesign .

Silver badge

Top tools and tips for junior Linux admins

Our sysadmin blogger Trevor Pott wants your suggestions for utilities, tools, tips, tricks, IRC channels, resources and anything else to ease new Linux administrators into the job of keeping servers running smoothly. Fire away, please!

C.

0
1
Happy

regex

http://www.zytrax.com/tech/web/regex.htm

become familiar with regex and save yourself some time and frustration. The link above is an excellent gentle intro to this seemingly hard subject.

2
0
Happy

Arch Linux is an intimidating distro for the newcomer, but everyone should try it. The reason I mention it here is the documentation is phenominal for linux users in general: wiki.archlinux.org

Centos is a good distro to learn for server use, and most usefully Redhat's documentation is online - it is a the same lineage - with Kickstart being a good place to learn scripted deployment (versus imaging).

On the BSD side I've found FreeBSD beautifully simple to administer; package (port) management takes a little more learning though.

1
0
Anonymous Coward

+1 for using a meta-distribution like Arch or Gentoo

I learned more about Linux just by installing Gentoo once than by spending years working with existing Linux systems.

1
0
Silver badge

Good conferences ...

as a compliment to web sites - meet other guys with beards and spectacles. You pick up more than just the problem that you have today and get to make friends. Try this one in Newcastle at the end of March:

http://www.flossuk.org/Events/Spring2013

0
0
Happy

Rule #1 : Read the man pages

Rule #2 : Read the man pages.

This is SO key, but many newbies miss this. You should have man pages installed by default and this is your best resource.

man <command>

e.g. man lvm

man -k lvm does a keyword search and gives you a list of pages relating to LVM.

once inside the man page, use slash (/) to search for FILES or SEE ALSO (literally type /SEE ALSO) and it will hop you down to some useful bits.

If a man page is in a certain section, then you need to specify that.

man 1p zcat gives you the zcat command from section 1p.

HTH.

4
1

man2html

Definitely, the manpages are excellent, except for coreutils for which the complete documentation is here:

http://www.gnu.org/software/coreutils/manual/html_node/index.html

If you're writing C code, then the manpages of the Linux Programmer's Manual (the "manpages-dev" package in Ubuntu) are extremely useful, and often better as a first reference than the glibc manual. They're useful even when you're writing Python or Perl, since so many functions in higher-level languages are just wrappers for the C functions.

I use man2html to allow me to read man pages in a browser. It gives you some creature comforts like scroll bars, a variable-width font, and links you can click on. And if my local man repository doesn't have the command I need, I use http://manpages.ubuntu.com/ . You don't have to be hard core to be a good Linux admin.

0
0
Headmaster

Re: Rule #1 : Read the man pages

I have to put a big caveat in here, that SOME man pages are excellent. But there are so many full of endless cruft, and it's a fairly unintuitive format - obviously written by old-school engineers.

When I was a real newb, I was endlessly frustrated with the endless advice to "read the man page" when I needed a quick one-line solution to an issue. WHICH man page? (Sorry, man -k can find even more cruft.) Then multipage listings of obscure switches that are never used in real life and lengthy explanations of underlying architecture can make it almost impossible to get to the info you need. Not to mention the idiosyncratic format and use of conventions that a naive user find difficult to decipher.

The famous example of sudoers 5 is a great case in point. Ok, after traversing many many pages, you get to some examples of how the thing might work, but wow, it's a mission.

So, sure, I think any documentation is better than none (a particular curse for opensource), but the accessibility of many of those pages could be vastly improved.

1
1
kbb

tcpdump

For troubleshooting networking issues there's nothing like a good tcpdump on a server imported into wireshark on a desktop for ease of viewing.

0
0
Linux

Some of the stuff I find handy...

* When ordering a Dell PowerEdge server, don't forget to buy the Enterprise DRAC and get it wired up. Provides a Web interface for the status of the machine, virtual media (yes, another way to install the OS without needing PXE boot) and, most importantly, a VNC terminal session to the main hardware (right from power up, BIOS, grub and the OS!).

* Install OMSA on PowerEdges, just for their "reboot recovery" option (if the kernel hangs, you can set it to wait so many seconds and then cold reboot the server).

* I like using rdist to keep customised system files synchronised across Linux servers and desktops. Think of it as a "networked make" utility and it's extremely handy once your server count gets into double figures.

* I use rsync to sync the system disk partition to another identical partition once a week via cron. In fact, it's a bit trickier than that (you have to change grub.conf and the copied /etc/fstab) so I've got a script to do it, but basically if I mess up the original system partition, I can boot into the second partition as an immediate rescue. Most useful if you run the sync script just before doing "yum update" (it allows you to roll back the update by rebooting into the pre-update partition).

* nagios for monitoring of course - install nrpe on the clients (yep, we rdist our custom config for nrpe) and then add the clients to the nagios server setup (we wrote a script for this part). With Dells that have OMSA installed, there's nagios modules to check OMSA info (e.g. temperature) and even some disk controller statuses.

* LDAP for authentication - quite tricky to setup, but useful when it's working properly. Don't forget to leave root as a local user in case the LDAP server goes titsup!

* amanda 3.X for tape (or even disk) backups. Amanda 2.X was dumb about autoloaders so we had to write a lot of custom wrapper scripts to check tape barcodes, load from autoloader, put back etc, but 3.X is actually good with autoloaders at long last. Now has (mostly) working Windows Amanda clients so we've even started backing up Windows clients too. Server-side encryption and compression are bonuses too (replace gzip with pigz in the Amanda server config if your tape server has many CPU cores).

* For VMs, we mostly use Proxmox VE in a clustered setup with KVM-based VMs (a mix of Windows and Linux clients). It's probably not as slick as VMware, but it's a *lot* cheaper! To be honest, I tried VMware a few years ago and couldn't believe it prompted you to rebuild kernel modules interactively as part of the boot sequence if you updated your kernel RPM and rebooted your VMware server. What if you didn't have console access at the time?! They've probably fixed this awfulness by now, but I couldn't believe how naff that was at the time.

0
0

Re: Some of the stuff I find handy...

"* When ordering a Dell PowerEdge server, don't forget to buy the Enterprise DRAC and get it wired up. Provides a Web interface for the status of the machine, virtual media (yes, another way to install the OS without needing PXE boot) and, most importantly, a VNC terminal session to the main hardware (right from power up, BIOS, grub and the OS!)."

Actually, since we are talking Linux here, the iDRAC express (the standard one) will allow you to set it up so that it gives you BIOS, Grub and the OS via the serial console. A pig to set up, but once done allows you full access to the hardware without having to pay the extra for the Enterprise licence....

Won't work for Windows though...

0
0
Devil

cmake

www.cmake.org believe it or not... this make file generator is great for automating the configuration of linux. It has the ability to script, generate files from templates, a full suite of file manipulation commands, and to top it off integration with the tools ctest and cdash. Setup tests that confirm your desired configuration and then have those tests posted on a web based dashboard.

cmake- it ain't just for making make files .

[I acknowledge that there are better configuration management systems out there. BUT for those forced to homebrew their own for whatever reasons (such as no external internet connection because of security) this tool rocks. Not to mention you can find thousands of people who already know cmake.]

0
0

Base distro

Your best bet, at least to begin with, is to choose a linux family, for instance debian-based distributions include debian, ubuntu and a variety of other offshoots, RHEL includes Redhat, centos, oracle linux etc. Stick with one family of linuxes for the first few months and you'll learn enough to be able to work around the nuances of the others, rather than lamenting your apparent lack of google-fu prowess when a configuration file in one family has moved to some obscure location in another.

1
0
Anonymous Coward

Re: Base distro

I'd tend to agree, the begineer should think of the different Linux distribution "families" as different OSes. Learn CentOS/Fedora or Ubuntu/Debian etc. etc. and then, once you're good on that, use the knowledge to learn another Linux if you so desire.

0
0

Arch Wiki

The arch wiki (and to a lesser degree the gentoo wiki) are some of the best tools to look for info - regardless of what distro your using IMO

https://wiki.archlinux.org/

If your a sysadmin who has to deal with 100+ Redhat based servers I suggest spacewalk - it is by far the best tool of its type.

Here's a nice one liner - shows top 10 highest connections through iptables (using for a bridging firewall..)

tcpdump -tnn -c 20000 -i br0 | awk -F "." '{print $1"."$2"."$3"."$4}' | sort | uniq -c | sort -nr | awk ' $1 > 100 '

1
1

Bash Scripting!

I've only been doing my sysadmin job for a year - prior to that I had no Unix/Linux experience at all!

One of my favourite help pages has been Steve Parkers bash scripting tutorial:

http://steve-parker.org/sh/sh.shtml

0
0
Flame

Not so much a tool as a philosophy

If you are going to do it more than three times, script it.

OK, maybe cron.

If you are getting good at bash scripting, and want to use cron to run them at scheduled times for some reason they run fine when you run them manually, but fail when run by cron change the first line in the script to:

#!/bin/bash -l

The -l argument specifies a login shell. If your usual login sets a lot of environment variables this can quickly sold a multitude of strange problems.

(Very useful for scripting oracle schema exports/imports.

3
0
Mushroom

Re: Not so much a tool as a philosophy

McGeneration does not really know what fancy words such as "philosophy" mean. They are attracted by shiny presentations, though. They can "see the quality of anything with a single glance".

They want Immediate Gratification and that is why anything Unixoid is above them.

0
3

This post has been deleted by a moderator

wub

Find and read the log files.

Many good ideas are already posted, but I didn't see logs. When a problem arises, it is probably logged somewhere. If you are still learning your way around, the log files will give you great ideas about what to study next. If you havn't got an issue at the moment, you may see something interesting to follow up on.

I just had a problem with a networked scientific instrument connected to a Windows machine (what can you do? the instrument makers write the software). The problem had to do with bootp, and the logging option was off. I turned logging on, restarted bootp, and got the clues I needed to fix the problem with a simple change to the hosts file. Without the log information, it probably would have been a while before I thought about checking that.

2
0
Flame

Plus

A proper egrep (much better than grep ) command will extract relevant log messages in a second out of hundreds of thousand of other messages. But yeah, you need to learn "cryptic" regular expressions. Too much for the ADHD kiddos.

1
0
Go

virt-sysprep

http://libguestfs.org/virt-sysprep.1.html

virt-sysprep (included in RHEL 6.3 and later) can reset network configuration in cloned VMs (and much much more).

0
0
h3
Bronze badge

All you really need is ksh93. (It has builtins for almost everything or can).

(And to understand the design of why UNIX was implemented like it was).

It is becoming harder to do though (Mainly due to the actions of one Redhat creator of systemd/pulseaudio/avahi).

If I was starting now I would probably try to learn on Solaris 9 in a VM. (Or a BSD) what you learn is fairly transferable. Both of them have manpages that are fairly decent and not blatantly wrong.

The way Linux is going is similar to XP there is hundreds of ways to do stuff and was a new way each week almost. They have started using loads of binary only stuff instead of text also which complicates things.

0
1
Mushroom

Hehe, I wondered how long before this subject would crop up.

0
0
h3
Bronze badge

I wouldn't recommend if you are trying to learn using stuff like virt-sysprep

You should be able to clone boxes by dd'ing the disk and running a script.

0
0

the first tool that i install on any linux machine is midnight commander (if not already there)

from there on the rest is more easy

1
0
Anonymous Coward

Linux from scratch

At an early stage of my Linux apprenticeship I had the chance to discover their web site ( http://www.linuxfromscratch.org/ ) and I had the patience to give it a try. It takes some time to do it but at the end, besides the satisfaction of having built your own Linux distribution you will come enriched with invaluable knowledge on how an OS is working internally. This experience is distro neutral so after that you will feel at ease in front of any flavor of them. Also another important gain is that you will never feel intimidated by the sequence ./configure, make and make install, ever. Add to this reading on CentOS website about when is appropriate to compile from source and when it is not ( http://wiki.centos.org/PackageManagement/SourceInstalls ) and you've just made a long way towards safeguarding you mental sanity as a *nix sysadmin.

In the end, for those who feel ready to begin their journey in the land of Tux I will recommend as a starting point the IBM Developers website ( http://www.ibm.com/developerworks/linux/ ). They have a good collection of tutorials there.

1
0

This post has been deleted by a moderator

Linux

Re: command line text searches

grep -il error `find . -name "*.log"`

would be more intutive and needs one command less. I think you could even get rid of find with modern grep variants that work recursively.

1
0
Vic
Silver badge

Re: command line text searches

> would be more intutive and needs one command less.

How does it deal with filenames containing spaces, though?

Vic.

0
0

how about mc

Yes, I know, there is all the shell programming, but mc gets me through a bunch of task without too much trouble ... as a casual sysadmin.

Ok - I confess, I used nc before :-)

1
0
Thumb Up

Re: how about mc

FCL is still out there :)

looks more to NC than MC

but without them i am like "blind"

0
0
Silver badge

Simple Text Editors

I'd vote for nano as a really useful tool. It's a bit less obscure for a quick tweak to a config file than vi or emacs.

The awk and bash HOWTOs have been very helpful in teaching me script programming, I even managed to type a one-liner without looking anything up and had it work first time the other day.

In fact, the tldp.org website is a great help in understanding all sorts of Linux things.

1
0
Bronze badge

POSIX

Start with the POSIX command set since they are available everywhere (including Windows with SFU). All the other commands should be considered gravy and learned as secondary but nicer ways to work which may or may not be available on a random system you end up working on. Remember that the Linux admin will also be the guy they call for random Unix jobs :)

0
1

I like to use 'watch' quite a lot to give me a live output of commands like 'du -sh'

It's not very pure and it's been a while since i used it but when i was first faced with a linux server i found webmin to be a life saver until i learned how to do simple thing like add a user or restart a service.

I like 'iptraf' too for a quick look at whats going on with a nic.

'startx is also quite an amusingly obscure way to start the desktop, always makes me smile when a new person see's me do that when they have spent the past 45 minutes fumbling around trying to do something.

My advice to new comers is always to use linux as your main os for a while, no dual booting or having a VM on your windows laptop. The problem with that is that as soon as you hit a problem that will be faster to get around by going back to windows, you'll go back to windows and learn nothing.

Dive in, be prepared to do some reading to be able to get yourself up to the same level that you are with windows. Once your there then you will find that you can do an awful lot more with linux.

I found Debian to be a wonderful and easy to use linux, i recommend it to anyone and personally i find it much easier to use that the redhat's out there.

1
0
Linux

So many tools

One definately worth looking at is nagios. Know where and when there is a problem.

Need to know more about a command or related commands? Use "man" and "apropos".

Definately learn shell scripting (usually thats bash nowadays). Thats for repeating tasks and cron jobs.

Netcat and openssl are invaluable tools for debugging servers (unencrypted/encrypted respectivly).

Log files, log files, log files. Yes, they are an invaluable tool for finding errors.

While we are at that, grep. Grep is your friend.

Lots more, but those are just the first important things that came to my mind.

0
0
Holmes

my list

If you really want to master Linus system administration:

0. Take a LSB course

1. Linux from scratch

2. Choose family

3. Master the package manager (pkg/apt or rpm/yum,zypper,..)

4. Learn basic vi

5. Learn basic regex

6. Learn to script, start with basic bash later you can matter python

7. Learn autoyast (side family)/kickstart (red hat family)/preseed (Debian family) for fully automated OK installation

8. Learn puppet/chef

By then you've got a few years of experience.

1
1
Anonymous Coward

Re: my list

I don't know why you got the down-vote because I don't think you deserve it. Although I might change the order for some items on your list, basically this is the path to follow for those who aspire to master Linux system administration.

0
0

Re: my list

probably a nano/emacs addict objecting to having VI on the list...

2
0

Re: my list

If you're going to learn puppet or chef then you should invest your time to learn ruby instead of python. It's mandatory anyway for chef, and puppet is written with it. Puppet templates can contain embedded ruby, and its also useful when writing your own (facter) facts. It's incredibly easy and quick to use for general purpose scripting tasks too.

0
0
Silver badge

First[0], install Slackware on a couple text boxen, and one of the BSDs on another.

Network the boxen together (freely available Celeron boxes with built in 10baseT will work nicely for this exercise, or 10base2 "thin net" if you don't have a spare hub). Use the Slackware boxen as "users", and the BSD box as "server". Create several user logins on each box. Along with whatever GUI you use on the main screen, hang a dumb terminal off a serial port on each box, and send it a login prompt. Handy when the GUI goes tits-up.

Then find a copy of the Coherent Lexicon, from Mark Williams Company (try used book stores in University towns). Read it, cover to cover. Yes, including the rather excellent Taylor UUCP tutorial[1]. Try the examples on the Slackware box as a user. Become root, if needed. When you break a Slackware box, learn to repair it in situ, without re-installing the OS.

When you're done with that, get a copy of UNIX Power Tools (O'Reilly). Do the same as the above.

If/when you get stuck, read the man pages; "apropos" and "whatis" can help. Do not ask questions on user forums (Usenet, etc) until you have done a thorough search online ... somebody, somewhere, has already asked & had the exact same problem answered.

Concurrent with this, enroll in "un*x sysadmin 101" (or equivalent) at a local JC/Poly.

Sound old-school? Absofuckinglutely. But you can't run before you can crawl.

[0] If you don't already know vi & sh inside out, start with those. See "run" and "crawl", above.

[1] I still use UUCP on internal links for a Usenet system I consult for. I also use it to move email between departments at a couple of Fortune 150s. I'll leave the "why?" as an exercise for the reader.

1
1
Anonymous Coward

Re: First[0], install Slackware on a couple text boxen, and one of the BSDs on another.

What you say is absolutely true but it will freak out all these kids who might look outside Windows. It's bad enough that they will have to type at a command prompt.

0
0
Silver badge

@AC 18:14 (was: Re: First[0], install Slackware on a couple text boxen, and one of the BSDs ...)

Thing is, AC, that it doesn't really matter if it freaks 'em out. It's the truth. Simply installing *buntu, or OS X Server, or a Windows Server, doesn't make you a sysadmin. Even though, technically, you are the admin of that particular box.

If a scholar wishes to become an actual, real-life Systems Administrator, s/he has to understand & use the CLI (and what run levels are, the difference between scripting, compiling and assembling ... etc., etc., etc.). This is true of every major server system that I'm aware of. Yes, including those from Cupertino & Redmond.

The bottom line is that if you can't handle the CLI as a day-to-day desktop, you're not cut-out to be an actual SysAdmin. Yet. But it's always possible to learn. There are no short-cuts, no "magic fix it pills". It takes time, hard work, sweat & tears ... and the occasional embarrassing gaffe, some of which are firing offenses. Just like any other profession worthy of calling one's own.

0
0
Silver badge

dd

Good post for noobs about to dd

http://www.linuxquestions.org/questions/linux-newbie-8/learn-the-dd-command-362506/

0
0
Pint

A couple more

A very good piece of server monitoring software can be found in Xymon (Hobbit as was). Great for quick glance monitoring state of server farms. Knocks spots off the paid for IBM Tivoli Monitoring.

The other is Webmin which is a great admin tool - you can farm out some administrative tasks easily to people with no special privileges, reducing the danger of people finding ways to circumvent sudo. (Ours was used to allow help desk people to set up printers across platforms without needing to know the quirks of each of the different command lines)

1
0

Become very good friends with ssh and byobu or screen very early, even if running a local vm. Once it's installed and the network is running, switch to ssh. In the past 3 years the only time I've used a console connection on a linux/unix server was on a lights-out console to see a kernel oops in proxmox had caused everything to stop.

Don't be afraid to be creative and break things. Once you've broken it spend time trying to fix it before getting someone else to sort it out. You remember it faster for the next time it breaks.

<search engine of choice> is your friend, nobody expects you to remember everything.

Less is better than More

1
0

Tmux (or screen).

Get comfortable with vi, it's everywhere.

strace for those hard to track down problems.

lsof can tell you a lot about what a process is doing.

dstat is handy for getting an overview of system performance.

As others have pointed out - read the man pages. Don't just blindly copy and paste stuff from tutorials.

Just build and break a bunch of stuff (in VMs or wherever).

0
0
Facepalm

catb et al

One of the hardest things for new comers to wrap their minds around is that they don't know how to ask the right questions. Someone even wrote a very famous and effective howto about it:

http://www.catb.org/~esr/faqs/smart-questions.html

John H Woods is right about dd and if you are talking to new sysadmin I would throw in ddrescue too:

http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html

In my opinion best way for new users to get to know Linux is by using VirtualBox in windows until they have learned enough to be comfortable with partitioning their drives.

My thoughts on LFS for new users are mixed. Yes you learn a lot, but you don't know what a lot of it means. Once you are done with it you can go on to BLFS but most people I talk to are ready for a package manager at that point.

Source based distros are my favorite way on my PC so when my friends are ready to dive in I show them Arch and Funtoo.

0
0
Bronze badge
Thumb Up

book --Unix Power Tools

This is the "Everything Within" book for Unix and Linux beginner admins. Although 20 years since the first edition, and more than ten years since the last, this book still cuts the mustard. Eminently readable, and crammed with beginner tips, but most importantly an introduction to the 'nix mindset. Just remember that for the details, time has not stood still, and check the man pages for the tools you actually have installed/are using.

Unix Power Tools, Third Edition

By: Jerry Peek; Shelley Powers; Tim O'Reilly; Mike Loukides

Publisher: O'Reilly Media, Inc.

Pub. Date: October 28, 2002

Print ISBN-13: 978-0-596-00330-2

Pages in Print Edition: 1160

2
0
Silver badge

For the given example

Fiddling with /etc/hosts, /etc/hostname, /etc/network/interfaces and so on is fine for a few machines. If you have lots of (virtual) machines, try dhcpd.

0
0

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Forums