Friday, May 27, 2022

Nested Virtualization KVM/QEMU and Qubes-OS

 Virtualization and Qubes-OS, I find very useful in exploring complex packages or new versions of Linux. If I try a complex package and then find it's too complex, or it's dependencies are too complex or it really doesn't give me what I want, I just throw away the Virtual Machine (VM). If I like the VM, the Qube, I can copy it to other machines running Qubes-OS.

Qubes-OS is still my favorite, since you can spin up and/or throw away a VM in 30 seconds. I recently had a contract where I was using RedHat, and I converted my main Linux laptop to RedHat and have been using KVM/QEMU for virtualization. However, it's slow to create a new VM.


 I finally have nested virtualization working with Qubes-OS running as a VM under Redhat KVM/QEMU. I just want to put some notes down. I know this invalidates some of the security of Qubes-OS, at least according the authors, but for what I want it for, I think this is OK.

This Redhat page is helpful:

First in Redhat 8.5, you need to enable the Input Output Memory Management Unit, IOMMU, support. My /etc/default/grub looks like this:


GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel-swap rd.luks.uuid=luks-50fdf3da-12a5-450e-863b-6b981be120bc rhgb quiet intel_iommu=on iommu=pt modprobe.blacklist=nouveau"

The important parts are enabling IOMMU, "intel_iommu", and enabling IOMMU passthrough "iommu=pt". I blacklist nouveau since I use the NVidia drivers so I have CUDA support. If the NVidia drivers detect nouveau or that nouveau was used for boot, the NVidia driver will refuse to run.

Next I created a /etc/modprobe.d/kvm.conf file containing:


# Setting modprobe kvm_intel/kvm_amd nested = 1
# only enables Nested Virtualization until the next reboot or
# module reload. Uncomment the option applicable
# to your system below to enable the feature permanently.
# User changes in this file are preserved across upgrades.
# For Intel
options kvm_intel nested=1
options kvm_intel enable_shadow_vmcs=1
options kvm_intel enable_apicv=1
options kvm_intel ept=1
# For AMD
#options kvm_amd nested=1

When creating the VM, I found that you need to "Copy host CPU configuration". For Qubes-OS 4.1, I chose a "Generic 2018" Linux kernel model. Qubes-OS uses a stripped down Fedora 32 Domain 0, dom0. I use the QXL VGA driver and Spice for the display server. This is the best combination of display driver support I have found so far for KVM/QEMU VMs.

Finally, you need to set Qubes-OS sys-net and sys-usb to Para Virtualized, PV according to this link:


The important part of this page is down at the bottom:


  1. Change the virtualization mode of sys-net and sys-usb to “PV”
  2. Add qubes.enable_insecure_pv_passthrough to GRUB_CMDLINE_LINUX in /etc/default/grub
  3. Run sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
  4. Reboot

 I am currently using the non-UEFI boot, the non-OVMF BIOS, so my grub mkconfig is "sudo grub2-mkconfig -o /boot/grub2/grub.cfg".


And with that bit of configuration I am able to run a nested virtual Qubes-OS.