Every troubleshooting session on this platform eventually comes down to one question: which component owns this problem? You can’t answer that without the org chart. So here it is.
Layer 0: the Xen hypervisor
At the bottom is Xen — a bare-metal, type-1 hypervisor. It boots first (you’ll see it in the GRUB entry: xen.gz loads, then the Linux kernel), and it owns exactly three things: CPU scheduling, memory partitioning, and interrupt routing. That’s it. Xen deliberately contains no device drivers, no filesystem, no network stack. It is a few megabytes of code whose job is to slice hardware among domains.
A domain is Xen’s word for a running VM. Which brings us to the special one.
dom0: the control domain
dom0 is the first domain Xen boots, and it’s privileged. It’s a Linux system — in the 8.x generation, a CentOS 7-based distribution maintained by the XCP-ng team with backported security fixes — and it has direct access to the physical hardware. All the actual device drivers (NICs, HBAs, RAID controllers) live in dom0’s kernel, not in Xen.
When you SSH to an XCP-ng host, you are in dom0. This surprises VMware people: it’s a real Linux shell, with yum, systemd, and normal logs under /var/log. Two rules keep you out of trouble:
- dom0 is an appliance. Don’t install random packages, don’t “fix” its config files outside documented procedures. Updates assume a known state.
- dom0 has fixed, small memory — a few GiB by default regardless of host RAM. Every guest’s I/O flows through dom0 (more on that below), so dom0 memory pressure is a platform problem, not a VM problem. There are legitimate reasons to raise it on busy hosts, and it depends on you’re pool size.
Guest VMs are domUs — unprivileged domains. In the 8.x era they run in HVM mode with paravirtualized drivers (PV-on-HVM): hardware-assisted virtualization for CPU/memory, but disk and network I/O goes through the fast PV path — a shared-memory ring between the guest’s PV driver (netfront/blkfront) and dom0’s backend (netback/blkback or tapdisk). That’s why guest tools matter so much: without PV drivers, the guest falls back to emulated devices via QEMU, and performance falls off a cliff.
XAPI: the toolstack and the pool
XAPI (the XenAPI daemon, historically xapi) is the management brain, running in dom0 on every host. It exposes the API that everything — xe CLI, Xen Orchestra, your Terraform provider — talks to, over HTTPS on port 443.
XAPI’s defining concept is the pool. Hosts joined into a pool share one configuration database (VMs, SRs, networks, everything), replicated from one elected host: the pool master. All API writes go through the master; other members proxy to it. This is why “the master is down” is a distinctive failure mode on this platform — the VMs keep running fine, but management operations stall until the master returns or you promote a new one.
The practical takeaways:
- Point tooling at the pool master (or at XO, which handles it for you).
- The pool database lives at
/var/lib/xcp/state.dbon the master. XAPI syncs it to members; pool metadata backup is a real operational task. xeis the CLI front-end to XAPI. Anything the UI can do,xecan do — usually with more precision.
The supporting cast in dom0
XAPI delegates. These are the daemons you’ll meet in the logs:
- xenopsd — the VM lifecycle manager. Start, stop, migrate, hotplug: XAPI decides what, xenopsd does how, talking to Xen. Migration failures usually leave their real story in xenopsd’s logging.
- SM (Storage Manager) — a set of Python drivers implementing each SR type (LVM, NFS, iSCSI…). Every VDI attach, snapshot, and coalesce runs through SM, and it writes the single most valuable troubleshooting file on the platform:
/var/log/SMlog. - tapdisk / blktap — the datapath for VHD-based storage. Each active virtual disk gets a tapdisk process in dom0 doing the actual VHD block translation.
- squeezed — dynamic memory control (ballooning) when VMs have a memory range instead of a fixed size.
- xcp-rrdd — collects performance metrics (the RRDs behind XO’s charts).
- networkd (xcp-networkd) — manages bridges/bonds/VLANs. XCP-ng uses Open vSwitch by default; each pool network is an OVS bridge on every host.
- message-switch, xcp-idl — internal message plumbing between the above. You mostly notice it when reading logs.
Where the logs live
The fastest way to look competent on this platform is to know which file to open:
| Symptom | First file to read (in dom0) |
|---|---|
| API operation failed, task errored | /var/log/xensource.log |
| Storage: attach, snapshot, coalesce, SR issues | /var/log/SMlog |
| VM won’t start / migration failure details | /var/log/xensource.log + xenopsd entries |
| Host-level kernel or driver issues | /var/log/kern.log, dmesg |
| Xen hypervisor itself | xl dmesg |
A full field guide to reading these is on the roadmap; for now, remember xensource.log for control-plane, SMlog for storage.
How Xen Orchestra fits
Xen Orchestra is not installed on hosts. It’s a Node.js application running in its own VM (or anywhere, really) that connects to each pool master’s XAPI over HTTPS and builds a unified view across all your pools. It adds everything XAPI doesn’t have: the web UI, scheduled backup jobs, delta backup logic, replication, RBAC, self-service, audit. Kill the XO VM and your hosts and VMs are unaffected — you’ve lost the management dashboard and backup scheduler, not the infrastructure. That separation is one of the platform’s best design decisions, and XO deployment options are covered in detail in their own post.
The mental model, compressed
- Xen slices hardware. Tiny, no drivers.
- dom0 is a privileged Linux VM holding the drivers and all the management daemons. Treat it as an appliance.
- XAPI is the API and the pool brain; the master holds the database.
- xenopsd runs VMs, SM/tapdisk run storage, and each writes its own log.
- XO sits outside, talking to pool masters, adding backup and orchestration.
When something breaks, walk the chart: is it a Xen problem (rare), a dom0 problem (drivers, memory, disk full), a XAPI problem (API/database/master), an SM problem (storage — it’s usually storage), or an XO problem (UI/backup layer)? Nine times out of ten, naming the owner is half the fix.