Virtualisation: headroom, shared storage and licensing

Virtualisation is a bet that your workloads do not all need their peak at the same moment. When the bet is right, you buy far less hardware than one server per workload would need. When it is wrong, you find out on the busiest day of the quarter, and the failure is shared: every workload on the affected host goes together. This guide is about placing that bet deliberately, which means doing the capacity arithmetic, understanding the shared dependencies you have just created, and licensing the result correctly.

What you are actually buying

A hypervisor is software that divides one physical machine into several virtual ones, each with its own operating system. The physical machine is the host; the virtual machines are guests. Hypervisors that run directly on the hardware, which is the arrangement in server estates, include open-source options such as KVM and commercial ones such as Microsoft Hyper-V and VMware ESXi. This site does not rank them, and the choice is mostly decided by your skills, your licensing position and the tooling you need around the hypervisor, rather than by the hypervisor’s own features, which are broadly comparable for ordinary workloads.

What the move to virtual machines changes is the unit of purchase. Instead of buying a server per workload, you buy capacity in hosts, and you carve it up in software. That gives you consolidation, faster provisioning, portability between hosts, and the ability to maintain a host without taking its workloads down. It costs you a new layer to operate, a shared failure domain, and a set of capacity and licensing questions that did not exist when each workload had its own box.

Sizing: headroom is the product

The mistake that recurs is sizing for the average. The right question is how much capacity is needed when a host has failed and its workloads have restarted elsewhere.

In a cluster of N identical hosts that must survive the loss of one, the workloads must fit on N minus 1 hosts. That caps steady-state commitment at (N minus 1) / N of total capacity, before the overhead of the hypervisor itself.

Most platforms can enforce this with an admission-control setting that refuses to start a guest if the cluster could no longer survive a host failure. It is common to disable it when capacity gets tight. That does not create capacity; it removes the warning that capacity is gone.

CPU and memory are not equally forgiving

CPU can be overcommitted, meaning the total number of virtual CPUs exceeds the physical cores, because guests rarely use all their CPUs at once. There is no universal safe ratio, and anyone who quotes one without knowing your workloads is guessing. Latency-sensitive services tolerate a low ratio; idle utility servers tolerate a high one. The honest approach is to measure contention directly rather than relying on a ratio.

The signal is time spent waiting for a physical core. Platforms name it differently: CPU ready on ESXi, steal time on KVM, and wait time per dispatch on Hyper-V. On a Linux guest, steal time is visible in the standard tools:

# The "st" column is the percentage of time this guest wanted CPU but the host gave it to others
vmstat 1 5

# Or in top, the %st figure on the CPU summary line
top -b -n 1 | head -5

Sustained single-digit percentages of steal on a latency-sensitive guest deserve investigation. Read the figure as a symptom that the host is oversubscribed, not as a precise threshold.

Memory is less forgiving. Overcommitting it means the host must reclaim memory from guests by ballooning or by swapping to disk, and a guest whose memory is swapped by the host can slow down by orders of magnitude. Plan memory as committed, not overcommitted, for anything that matters.

Storage: the shared dependency

The first thing virtualisation tends to create, without anyone choosing it, is a single shared failure domain: one storage system behind many hosts. If the array is unavailable, every guest stops, however many hosts you have. Shared storage is also the main performance constraint. It must be sized for latency and I/O rate, not only capacity: ten terabytes of slow disk and ten terabytes of flash are the same capacity and very different platforms.

There are two broad approaches. External shared storage, a SAN or a network file server, is a separate system you buy, size and operate, with its own controllers, firmware and failure modes. Distributed or hyperconverged storage pools the local disks of the hosts themselves in software, so storage scales with compute and loses the separate box, at the price of using host resources for storage and making host maintenance a storage event too. Neither is better in general. What matters is that you can name, for your design, what happens when a single storage controller, a single switch or a single host disappears, and that you have tested it.

Thin provisioning, where guests are promised more disk than physically exists, adds a quiet failure. If the underlying space runs out, guests can pause or crash all at once. Monitor free space on the pool with the same seriousness as CPU, and set alert thresholds well before the point at which a surge could fill the remainder.

High availability, live migration and fault tolerance

These three are routinely confused, and they do different jobs.

• High availability restarts a guest on another host after its host has crashed. The guest goes down and comes back, usually within minutes. It protects against host failure and is not instantaneous.
• Live migration moves a running guest between hosts without stopping it. It is for planned maintenance, not for failures, because it needs the source host to be alive. It generally needs compatible processors or a masked CPU feature set, and a fast network between hosts.
• Fault tolerance, where offered, keeps a lock-stepped shadow of a guest so that failure causes no interruption. It carries limits on guest size and doubles the resources consumed, so it is reserved for a few critical guests.

None of them is a backup. All of them faithfully preserve a deleted file or an encrypted disk. The distinction is covered in backup, high availability and disaster recovery.

Placement rules matter as much as the features. If two nodes of a clustered application can land on the same host, the cluster protects against nothing. Anti-affinity rules keep them apart, and they should be checked after every large change.

Licensing: do it before the design, not after

Virtualisation changes what you need to license, and mistakes are expensive and discovered by audit. Three patterns recur. Verify each against the vendor’s current published terms, because they change, and ask for the answer in writing.

1. Operating system guests. Windows Server licensing is tied to the physical cores of the host. Standard edition covers a limited number of guests per fully licensed host, while Datacenter edition covers unlimited guests, so the break-even depends on how many Windows guests you will run. Windows and Linux hosting works the arithmetic.
2. Database and application licences. Some products license per core of the guest, with minimums per virtual machine. Some vendors treat every host in a cluster where the guest could run as in scope. A design that lets a licensed guest move across a large cluster can multiply the bill. Restrict such guests to a small set of hosts where the vendor’s terms require it.
3. Hypervisor and management tooling. Hypervisor licensing models have changed more than once across the industry, in some cases substantially. Treat a renewal as a decision point and not an administrative task, and model what the next three years cost under the current terms.

Sprawl, snapshots and drift

Virtualisation makes creating a server so cheap that nobody deletes one. Sprawl has real costs: licences, patching, backup storage and attack surface. Give every guest an owner and a review date, and reclaim the ones that nobody claims.

Snapshots are another trap. A snapshot preserves a point in time, which is useful for minutes or hours around a change. Left in place for weeks, it grows, degrades disk performance, and can take a very long time to merge back. Treat a snapshot as a short-lived safety net for a change and never as a backup.

How this goes wrong

• Sized for average load. The cluster could not restart workloads after a host failure, because admission control had been switched off to fit one more guest.
• The array is the cluster. A single storage controller update took every guest down together. The design had many hosts and one shared dependency.
• A host overcommitted on memory. The first sign was swapping, which looked like an application problem for two weeks.
• A licence moved with the guest. Live migration let a licensed database run on every host in a large cluster.
• Snapshots as backups. A months-old snapshot chain failed to merge in the middle of an incident.
• Management plane forgotten. The host’s management network, its time source and its own configuration were not backed up or monitored.

What to do next

Work out, for your own estate, how much you can commit and still survive a host loss, and compare it with what is committed today. Then compare the options in the architecture comparison matrix, and if a hosted private cloud is on the table read private cloud for what it adds beyond a virtualised cluster, and what it does not.