Re: I want to play with VMs
I guess that you want to learn by playing, and you don't have a specific purpose in mind. I'm very familiar with Linux and KVM, and work with reasonably large clusters.
Virtualisation itself is a commodity nowadays. There are minor differences between the different hypervisors, but all in all, they require compute, network, and storage and they spit out VMs. For your purposes, any compute unit with virtualisation extensions should be fine, and pick a hypervisor that you can easily script against (ie. there's an API). For most purposes, standard networking setups suffice - you will want to learn about VLANs and bridges in Linux; at some point you might want to look into the Software-Defined-Networking software, but this is usually only relevant for public cloud deployments. Storage is the most interesting, as virtualisation adds more layers into the mix and introduces mixed usage patterns - for a very simple setup, store VMs on local disks or on an NFS mount, and provide NFS as extra storage for VMs. Later, you might want to invest time exploring Ceph/Rados, Sheepdog, GlusterFS, or similar. In my opinion, iSCSI is more trouble than it's worth - too many vendors do things too differently (and storage folks don't like you messing with their kit), so you usually end up running your own iSCSI targets if you want the most flexibility, which introduces more points of failure and a lot of work to do it right.
The real interesting parts of virtualisation and where most effort is being invested nowadays is around the supporting management infrastructure. If you have hundreds or thousands of virtual machines running across many independent clusters, you need a way to manage them. You need to know which VMs are running on which servers; you need to have sensible VM allocation (and reallocation) algorithms; you need good monitoring; you need DHCP and DNS servers; you need some kind of CMDB storing metadata; you need user access control; you probably want to use image-based deployments, so you'll want a management solution for that which the VM-deployment-scripts can make use of; and you'll want to support PXE installations (at least for the physical servers, if not for the VMs); you'll want full lifecycle management of the infrastructure and VMs, from initial resource allocation and creation through to migration and recovery from failures and finally to decommissioning.
On top of that comes the management of applications and services on individual servers, and the orchestration of deployments using the IaaS-APIs and configuration tools (such as CFEngine, Chef, Puppet, Ansible, SaltStack, etc.). At the very least, you'll want a reproducable way to configure your physical servers and keep them up to date and consistent.
For KVM, you might want to look at http://www.linux-kvm.org/page/Management_Tools for some tools other people and companies have written.
If you really want to understand with a bottom-up approach, I recommend starting by setting up redundant DNS & DHCP servers with support for PXE, and using virt-manager to create VMs on remote servers using a single VLAN and bridged networking using local disks or NFS backed storage. Once you've managed that, start automating things yourself, but evaluate different tools as you go along. For KVM, there is no one standard, and every software solution manages things their own way. There are lots of moving parts, and any full solution will be necessarily complex (or simple but inflexible). Oh, and don't forget to design monitoring into your solution: Make sure you have the possibility to extract metrics at every step (no need to collect them all from the start - you'll figure out what's important as you go along).
When you're done with this, you may then decide that virtualisation itself is not that important for your situation, and the same management and orchestration toolbox can be applied to general application and resource management where you'll get much better performance/better ROI by deploying smarter software stacks running on bare metal, in containers like LXC, or using external IaaS providers.
Good luck!
Biting the hand that feeds IT
