Virtualization and Qubes-OS, I find very useful in exploring complex packages or new versions of Linux. If I try a complex package and then find it's too complex, or it's dependencies are too complex or it really doesn't give me what I want, I just throw away the Virtual Machine (VM). If I like the VM, the Qube, I can copy it to other machines running Qubes-OS.
Qubes-OS is still my favorite, since you can spin up and/or throw away a VM in 30 seconds. I recently had a contract where I was using RedHat, and I converted my main Linux laptop to RedHat and have been using KVM/QEMU for virtualization. However, it's slow to create a new VM.
I finally have nested virtualization working with Qubes-OS running as a VM under Redhat KVM/QEMU. I just want to put some notes down. I know this invalidates some of the security of Qubes-OS, at least according the authors, but for what I want it for, I think this is OK.
This Redhat page is helpful:
https://docs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/
First in Redhat 8.5, you need to enable the Input Output Memory Management Unit, IOMMU, support. My /etc/default/grub looks like this:
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel-swap rd.luks.uuid=luks-50fdf3da-12a5-450e-863b-6b981be120bc rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_iommu=on iommu=pt modprobe.blacklist=nouveau"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
The important parts are enabling IOMMU, "intel_iommu", and enabling IOMMU passthrough "iommu=pt". I blacklist nouveau since I use the NVidia drivers so I have CUDA support. If the NVidia drivers detect nouveau or that nouveau was used for boot, the NVidia driver will refuse to run.
Next I created a /etc/modprobe.d/kvm.conf file containing:
# only enables Nested Virtualization until the next reboot or
# module reload. Uncomment the option applicable
# to your system below to enable the feature permanently.
#
# User changes in this file are preserved across upgrades.
#
# For Intel
options kvm_intel nested=1
options kvm_intel enable_shadow_vmcs=1
options kvm_intel enable_apicv=1
options kvm_intel ept=1
#
# For AMD
#options kvm_amd nested=1
When creating the VM, I found that you need to "Copy host CPU configuration". For Qubes-OS 4.1, I chose a "Generic 2018" Linux kernel model. Qubes-OS uses a stripped down Fedora 32 Domain 0, dom0. I use the QXL VGA driver and Spice for the display server. This is the best combination of display driver support I have found so far for KVM/QEMU VMs.
Finally, you need to set Qubes-OS sys-net and sys-usb to Para Virtualized, PV according to this link:
https://www.qubes-os.org/doc/installation-troubleshooting/
The important part of this page is down at the bottom:
- Change the virtualization mode of sys-net and sys-usb to “PV”
- Add
qubes.enable_insecure_pv_passthroughtoGRUB_CMDLINE_LINUXin/etc/default/grub - Run
sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg - Reboot
I am currently using the non-UEFI boot, the non-OVMF BIOS, so my grub mkconfig is "sudo grub2-mkconfig -o /boot/grub2/grub.cfg".
And with that bit of configuration I am able to run a nested virtual Qubes-OS.
