Add Comment
Top tools for junior Linux admins
Our sysadmin blogger Trevor Pott wants your suggestions for utilities, tools, tips, tricks, IRC channels, resources and anything else to ease new Linux administrators into the job of keeping servers running smoothly. Fire away, please! C.
This topic was created by diodesign.
Give them a copy of http://www.amazon.com/UNIX-Linux-System-Administration-Handbook/dp/0131480057 to get them acquainted with the OS and of http://www.amazon.com/Practice-System-Network-Administration-Second/dp/0321492668 to undestand what being a sysadmin is (this latest book is platform independent, I wish more Windows admins read it).
As a comment to your article http://www.theregister.co.uk/2013/02/21/linux_isnt_that_hard_really/ stop cloning linux vm's. Learn how to use kickstart for redhat distributions, mirror internally the OS repositories (mrepo https://github.com/dagwieers/mrepo is quite easy to setup) and spinning a new vm will take you 3 minutes with the latest updates. Yes, really.
Your rants on the ifcfg-xxxx files for configuring nics on redhat based distributions are quiet funny. You try shoehorning your windows admin practices on other types of systems and you run into problems. Stop doing that! Kickstarting the OS with a clean, unattended installation will solve all your problems about that particular point. It really is a non issue for most people except the ones cloning vms.
The network config files of redhat are very easy, are very well documented (https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-Network_Interfaces.html). The documentation for redhat systems is simply excellent: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/ , as a linux admin using redhat based distros get familiar with it. Stop using forums to find your answers and start reading the fine manuals, they are there for a reason.
Learn how to use the vi editor. Yes, you have to. No, this is not optional. Sooner or later you will need to do something on a system with that one as only editor available, so you might as well just get used to it. You do not need to be a vi guru, just be able to edit a file.
Learn how to write your own shell scripts. Read other people's scripts. Understand them.
Learn a high level scripting language. Lots of admins know Perl, Python is fine as well.
Learn how to troubleshoot your system. Where are the log files, how to bump up debugging for the various subsystems, how to turn services on and off, how to find info on those services and subsystems without an internet connection (yes, it happens that you are cut off the net and you have a problem).
If you have to work with readhat distros, learn to like selinux. It is your friend and might save your ass on those internet facing servers when they get owned by a faulty web app deployed by hit and run ruby on rails developers (or php, or python or ....)
Once your really understand how a linux computer works, then you can start scaling out with cfengine/puppet/chef/ansible, whatever.
All the above things will make your junior admins better Windows admins as well. They will understand how protocols work and they will concentrate on finding the solutions to their problems instead of just clicking around without really knowing what they are doing in their shiny tools.
If your junior admins are not willing to do this, they should be forced by management. If they still do refuse to do it, get new juniors admins. If the company wants to run linux servers, admins need to support it. It is that simple.
/usr/sbin/sys-unconfig
Back in the day when I was cloning E10K domains running Solaris 2.6/7/8 and JET wasn't around I had to use a combo of
1) ufsdump over NFS to the SSP
2) transfer the ufsdmp to the new SSP/E10K
3) jumpstart interactive the newly prep'd E10K
4) ufsrestore via NFS the ufsdump
5) THEN to get round all things network and /dev - do ...
5a) run/usr/sbin/sys-unconfig
5b) rm /etc/path_to_inst
6) boot the domain OK> boot -arv
and hey presto the domain booted 'as a fresh install' and rebuilt it's device tree
In modern times - when cloning ESX hosted VM's I still use the '/usr/sbin/sys-unconfig' ...
It does exactly what is says on the tin.
Re: /usr/sbin/sys-unconfig
### never delete any of your cribs ###
- just dug out the procedure; from the back of an olde backup
Here it is in all of it's splendour
##############################
Pre-work.
Get the ufsdumps on a server on a local network. Ie one of the ssp's.
The images need to be on the 192.168.42.x network or you will have to plumb up
another interface later on the public.
1) Create the /etc/ethers entries for global and local mac adresses for the domain.
2) Add the entries for the domains ip in /etc/hosts
3) Setup the client as an jumpstart client using add_install_client.
4) Bringup -A off the domain and if 2.6 limit-ecache-size.
5) show-nets to select the correct interface and add a nvalias to sspnet.
6) boot sspnet -s
7) format and select the root disk. Should be a nice c0t0d0s2 :)
8) slice it up as you want and label it.
9) newfs the slices you made.
10) nfs mount the ufsdump directory onto /mnt
11) mount the root slice onto /a
12) cd /a; ufsrestore -xf /mnt/root-ufsdump
13) Stop veritas from coming up.
# touch /a/etc/vx/reconfig.d/state.d/install-db
# rm -f /a/etc/vx/reconfig.d/state.d/root-done
14) Comment out veritas and disk suite entries for rootdisk from /a/etc/system
# TERM=sun-cmd; export TERM; vi /a/etc/system
* rootdev:/pseudo/md@0:0,40,blk for disksuite
* set md:mddb_bootlist1="sd:7:16 sd:15:16 sd:23:16 sd:31:16" for disksuite
* rootdev:/pseudo/vxio@0:0 for veritas
* set vxio:vol_rootdev_is_volume=1 for veritas
15) Copy the good path_to_inst to the old root dir
# cp -p /etc/path_to_inst /a/etc/path_to_inst
16) Remove the old devices tree and dsk/rdsk entries.
# rm -rf /a/devices/*
# rm -rf /a/dev/dsk/*
# rm -rf /a/dev/rdsk/*
17) Recreate those trees for Solaris 2.6.
# drvconfig -r /a/devices -p /a/etc/path_to_inst
# cd /devices
# find . -print | cpio -pduVm /a/devices
# disks -r /a
# devlinks -r /a
18) Recreate those trees for Solaris 7 / 8 / 9.
# rm -f /a/dev/cfg/c*
# devfsadm -c disk -r /a -p /a/etc/path_to_inst
19) Sort out the /a/etc/vfstab and change to the underlying devices.
Hash out vxvm volumes or disk suite volumes.
20) Setup the boot block
# installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t0d0s0
21) Sort out the network or the reboot will take lots of time due to nis+ and dns.
Just edit the /a/etc/hosts and /a/etc/hostname.* files to reflect the new values.
22) Sort out the defaultrouter + don't route
# echo "default ip addr" > /a/etc/defaultrouter
# touch /a/etc/notrouter
23) Sort out the /a/etc/ssphostname. Put the admin network name of the main ssp.
The default is main_ssp with a corresponding entry in the /a/etc/hosts file.
Use the 192.168.42.x value for the ssp address.
24) Stop the VCS cluster form starting.
# mv /a/etc/rc2.d/S92gab /a/etc/rc2.d/s92gab
# mv /a/etc/rc2.d/S70llt /a/etc/rc2.d/s70llt
# mv /a/etc/rc3.d/S99vcs /a/etc/rc3.d/s99vcs
25) Edit /a/etc/syslog.conf and remove the chainsaw entry
26) Restore the other filesystems.
# cd /
# umount /a
# mount /dev/dsk/c0t0d0s6 /a
# cd /a; ufsrestore -xvf /mnt/var-ufsdump
27) Umount all filesystems and reboot the box with the reconfigure option.
# umount /a
# umount /mnt
# luxadm set_boot_dev /dev/dsk/c0t0d0s0
28) Setup the eeprom
# eeprom "local-mac-address?=true"
29) Sort out the dns entries (XXX = mpc/jgc/mit)
# rcp -p sda:/etc/resolv.conf.XXX /etc/resolv.conf
30) Reconfigure verbose reboot
# reboot -- -rv
31) The first reboot will fail the interfaces, this is because the reconfigure has not
sorted out the qfe in /dev/ before the kernel tries to mount it. Sorted on next reboot
or manual configure.
32) Install veritas the standard way.
... but ifconfig still only shows the loopback adapter. This is because the sysconfig networking scripts (located at /etc/sysconfig/networking-scripts/ifcfg-eth*) haven't been updated with the new MAC address.
Not necessarily.
Probably slightly cowboy, but you can use the udev files to associate MAC addresses with eth0/1/2 etc as you say, then in the ifcfg-eth0 configuration files not use a MAC address - ie once udev has sorted them out you just have ifcfg-eth0 not tied to a MAC address.
Someone will tell me that the world will end if I do this, but I've been happily running systems like this for years.
Don't forget 'sar' and 'ksar' and mpstat
It's there and it's free ... and works on all flavours
With Ksar you can produce .pdf reports of your systems performance [great for producing charts to then explain to your 'off-shore' developers that it's not the tin underperforming it's their single threaded java app'.
Re: Don't forget 'sar' and 'ksar' and mpstat
+1 for kSar
Its been very useful to me on many occasions - such as users saying 'my machine was very slow last night' and you can show them a graph of memory being used up or huge network loads.
Beer, because its Friday afternoon.
Learn Uzing Brainz Zells
So you are a Linux beginner and the first thing you want to do is to "clone virtual machines" ? Yeah, sounds like the best recipe for a M$-sponsored Badmouthing Exercise. I guess that is the whole point of the article. Windows is much better in the bribing business, as there is more money involved than in the Linux business.
So you want to really learn something new ? Remember how you did it in school: Go to the library or bookshop and get yourself a tutorial book. One that explains concepts as opposed to explaining dumb-down GUIs.
Unixoid systems (and thereby Linux) are all about
A) understanding key concepts such as files, STDIN, STDOUT, root vs normal user, permission bits, processes and the like, device files, file systems, regular expressions and the like
B) a small set of powerful commands/tools such as rm, ln, mv, vi, ps, ls, kill, grep, perl, cat. xargs, cut, split, sed. Learn those key commands/tools properly and how to combine them. They are NOT AT ALL cryptic to those who have actually taken time to learn them. They will allow you to automate lots of repetitive stuff and be much more productive than any blinking, shining GUI.
If you really need GUIs, then please stick to Windows, the OS of the MBA Generation.
My own faves
OK, as I came into Linux from Unix anyway, I already had a number of fave commands which worked well going over, including find (especially find . -exec which is damn useful for things like large scale permission fixes), grep and such.
One tool I was introduced to, however, was webmin (www.webmin.com) which is a web based front end which I've used on various flavours of Linux including openSUSE and RedHat/Fedora. It cuts a lot of time and effort out of the everyday configuration tasks on a system. Certainly with SuSE, I can combine procedures using Webmin and YaST, but Webmin takes a lot of the need for tools like YaST and its equivalents on other systems out of the equation.
I'm also something of a fan these days of Nagios (www.nagios.org), amonitoring tool which can be used to monitor quite a large range of devices, from switches up to servers, Linux, Unix or Windows. I tend to use Nagios Core and build my system from there.
ZSH
I switched after 18 years of CSH, TCSH and BASH, and it was worth it, even if just for the double-asterisk.
Other top old-timer tricks - to copy a whole directory between machines without rsync:
tar czf - srcdir | ssh remotehost tar zvxf -
I still get murmurs of approval for that one. And my other favourite for batch renaming files:
1. ls -1 | awk '{ print "\""$1"\"" }' > file1
3. cp file1 file2
4. vi file2 to edit the filenames with regex as you need to; then
5. paste file1 file2 > file3
6. vi file3 to insert "mv " at the start of each line.
7. source file3 to run it.
Re: ZSH
zsh is by far my most preferred shell too. Would love to get it across all our servers, but alas, it's only on my own machines (and work laptop), all other boxes come and stay with bash out-of-the-box :(
Re: ZSH
Piping through ssh is fun, but isn't that tar example a bit contrived? Why not simply "scp -rC"?
Avahi
Oh and another one: avahi (which implements zeroconf)
Really. for small to mid size networks, avahi makes managing IP addresses a thing of the past. Just give the machine a name then "aptitude install avahi-utils", then you can "ssh newhost.local" without ever needing to know the IP address. I saw someone above mention dhcpd over static IP address - that's fixing exactly half the problem. Avahi fixes the other half.
Look it up...
"There's nothing new under the sun." Almost every problem I have had has been had by someone else first. That's why my favorite tools are http://www.superuser.com and its related sites.
Munin
One of the first tools I install on an unfamiliar system is munin. It's an excellent overview of what the box is up to.
Usually it'll run for an hour or more, and the high points on the graph are the bottlenecks they've been chasing for the last n weeks...
Vic.
And packaging, of course
Packages are essential to the long-term stability of a system. Knowing what you've got and how it got to be there are essential.
It's vital not to be distracted by the ".deb vs .rpm" "debate"[1]. They're equivalent. With one *possible* exception[2], anyone telling you that one or the other is better simply doesn't know how to drive the one he's slagging off.
When I was getting to know Linux, I broke a number of installations by relying on the advice of people who told me just to "make install" eveything. It works - for a little while. Then you find out you've over-written something that another application relied on...
Vic.
[1] I use the term quite wrongly, of course.
[2] I'm not sure it's possible to find out how a package's files have changed since installation in the apt/dpkg world; the equivalent of "rpm -V". This might just be ahole in my knowledge, but I didn't get any answers when I asked the LUG...
Where are all my files? <whimper>
For the time when you hit return on
Do Not Do This@Anytime home$ rm -rf *
and then shriek "Oh, good gracious me, what a silly billy" (or words to that effect) there's <a href="http://carlo17.home.xs4all.nl/howto/undelete_ext3.html>ext3grep</a>.
You are still deep in a world of pain and toil, but all is not lost. I did this a couple of years ago (I thought I was in a subdirectory, was in fact in my home directory) and eventually I lost hardly anything, although rm had been running for about five seconds before I reached the shriek stage. Now I have ext3grep installed on a live USB stick, but I do lots of backups...
<boffin icon, because you're not giving this job to a noob>
Re: Where are all my files? <whimper>
I almost suffered badly from a simpler mistake, but logged in as an ordinary user thankfully not much happened. I was lucky that time, so don't try this folks:
chmod -R <somesetting> .*
I wanted to change settings on all hidden files/directories in my home folder, but I had not anticipated that '..' is also a match to '.*' and so the recursive application went UP a directory then down everyone else's home!
So think VERY CAREFULLY about wildcard/regex matches before doing something like this, and maybe test on a begin action before something almost irreversible like rm/chmod/chown.
neat tools
One of the very first items I install on a Debian based machine is 'sysv-rc-conf' which gives a nice overview of all services and their runlevels. There's also 'rcconf' on Deb based boxes which is similar to ntsysv on RH based boxes, but you can't select the individual runlevel to run at, just simply enable/disable. Then of course there's chkconfig on RH boxes to set individual levels for services...
Surprised no one has brought up OMD yet (omdistro.org) for monitoring, which incorporates nagios, check_mk, pnp4nagios and few other tools, which makes monitoring a breeze.
htop is also a neat tool, I tend to replace top with htop on my machines by aliasing htop to top.
If you connect to many boxes via ssh, look into setting up the ssh config file, followed by an alias in your ${SHELL}rc file, e.g. alias box1='ssh box1' where box1 again is set in /home/$LUSER/.ssh/config with the settings required.
If you have to work with netcat a lot, consider swapping netcat for cryptcat (http://sourceforge.net/projects/cryptcat/)
deborphan is a useful tool for deb based boxes, finding orphaned deb installations which are no longer required.
of course learning awk and sed etc. is a given, though consider there are also other useful tools to use instead, such as tr in various cases rather than sed.
Get to know Kerberos. At some point in your SysAdm life you'll come across a large network with multiple users and systems, and you don't want to have every person having local accounts on each boxes accessed by ssh keys only...
Ideally learn to harden Linux machines from early on, makes life later on much easier too.
Also get yourself familiar with a version control system such as cvs,svn,git,mercury, and keep your config files in there, especially useful if you have a given config setting across various boxes, be it httpd confs, kerberos confs (nsswitch.conf, pam/system-auth, etc....), and any other used across various boxes.
Linux is not Windows, but it doesn't mean it's 100% safe/secure. Look into chkrootkit and rkhunter.
There are loads more of course. A forum I frequent and is quite useful is
linuxquestions.org
Enjoy,
A
Touching an individual machine means you are losing
My advice would be that if you are touching an individual machine, then you are losing.
For servers that means Puppet, Nagios, single signon, a brutal approach to hardware failure, funnelling everything through a ticketing system, referring to that as the documentation for changes in your configuration repo, documentation written for use by trained people rather than blow-by-blow. Because you end up with a low headcount, then that means evolution of hardware, not once-in-a-blue-moon refresh projects.
For desktops that means either SOA for BYOD, but not some expensive middle ground. It means automating the common helpdesk tasks. It means using the vendor's tools rather than third-party tools, because that lowers your training costs because users get good hits from Google. It means online training.
For networking it means DHCP for IPv4 and Dynamic DNS. It means IPv6 is standard for intranet use (ie, no interior NAT). It means not fiddling with ethernet autonegotiation. It means anycast DNS forwarders. It means cookie cutter cupboard, building and core designs. It means treating VM servers as first class items in the network. It means 802.1x for wireless rather than web landing pages.
I'm learning all this stuff for fun in my spare time.
I've found TLDP really helpful. I'm going to try a LFS build as soon as I'm happier with vi - I actually learned to ed (for fun), which came in really handy for learning to grep. I'm working my way through distros that force me to learn before I'm happy (then I move on) - started with Linux Mint, moved onto a Debian minimal install, now I'm making Gentoo my personal distro.
As a complete beginner, man pages would just blur in front of my eyes. Now I read them like novels, gasping with delight when I find a useful new switch!
grep, regex, TLDP "guide"books, wikibooks, LPI Essentials Guide (from linupfront) and generally googling for "advanced [keyword] tutorial" like bash, zsh, grep, regex, globbing were all helpful. I tried getting books out of the library, but they were all stuff I knew, except for Networking for Dummies, which was a big help for a complete noob like me learning TCP/IP principles (even though it's Windows based, the standards are all the same) and it's good for when things with ipconfig comes up.
That's me, anyway...
tell them to migrate to windows where you can drag n drop files and just press a resync button if needed
sysresccd and backtrack
I know it's actually a few OSes and some tools on a CD, but man, those two have most of the tools I use. I haven't looked at Kali yet.
It's kind of funny asking for tools that an nix admin find useful. I would say "all of them". It might seem obvious to the more experienced. In order to do serious work, you'll need to use many tools. So it would be better to ask for something more specific like: "Best network traffic analysis tools". Then you might get some useful answers.
