Nested Virtualization KVM/QEMU and Qubes-OS

 Virtualization and Qubes-OS, I find very useful in exploring complex packages or new versions of Linux. If I try a complex package and then find it's too complex, or it's dependencies are too complex or it really doesn't give me what I want, I just throw away the Virtual Machine (VM). If I like the VM, the Qube, I can copy it to other machines running Qubes-OS.

Qubes-OS is still my favorite, since you can spin up and/or throw away a VM in 30 seconds. I recently had a contract where I was using RedHat, and I converted my main Linux laptop to RedHat and have been using KVM/QEMU for virtualization. However, it's slow to create a new VM.

 

 I finally have nested virtualization working with Qubes-OS running as a VM under Redhat KVM/QEMU. I just want to put some notes down. I know this invalidates some of the security of Qubes-OS, at least according the authors, but for what I want it for, I think this is OK.

This Redhat page is helpful:

https://docs.fedoraproject.org/en-US/quick-docs/using-nested-virtualization-in-kvm/

First in Redhat 8.5, you need to enable the Input Output Memory Management Unit, IOMMU, support. My /etc/default/grub looks like this:

 

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel-swap rd.luks.uuid=luks-50fdf3da-12a5-450e-863b-6b981be120bc rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_iommu=on iommu=pt modprobe.blacklist=nouveau"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

The important parts are enabling IOMMU, "intel_iommu", and enabling IOMMU passthrough "iommu=pt". I blacklist nouveau since I use the NVidia drivers so I have CUDA support. If the NVidia drivers detect nouveau or that nouveau was used for boot, the NVidia driver will refuse to run.

Next I created a /etc/modprobe.d/kvm.conf file containing:

 

# Setting modprobe kvm_intel/kvm_amd nested = 1
# only enables Nested Virtualization until the next reboot or
# module reload. Uncomment the option applicable
# to your system below to enable the feature permanently.
#
# User changes in this file are preserved across upgrades.
#
# For Intel
options kvm_intel nested=1
options kvm_intel enable_shadow_vmcs=1
options kvm_intel enable_apicv=1
options kvm_intel ept=1
#
# For AMD
#options kvm_amd nested=1

When creating the VM, I found that you need to "Copy host CPU configuration". For Qubes-OS 4.1, I chose a "Generic 2018" Linux kernel model. Qubes-OS uses a stripped down Fedora 32 Domain 0, dom0. I use the QXL VGA driver and Spice for the display server. This is the best combination of display driver support I have found so far for KVM/QEMU VMs.

Finally, you need to set Qubes-OS sys-net and sys-usb to Para Virtualized, PV according to this link:

 

https://www.qubes-os.org/doc/installation-troubleshooting/

 

The important part of this page is down at the bottom:

 

1. Change the virtualization mode of sys-net and sys-usb to “PV”
2. Add qubes.enable_insecure_pv_passthrough to GRUB_CMDLINE_LINUX in /etc/default/grub
3. Run sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
4. Reboot

 I am currently using the non-UEFI boot, the non-OVMF BIOS, so my grub mkconfig is "sudo grub2-mkconfig -o /boot/grub2/grub.cfg".

 

And with that bit of configuration I am able to run a nested virtual Qubes-OS.

What's Ed Doing with his Life?

 

You might be wondering what I have been doing with my life, and why I am not chasing contracts hard.

In general, I have been making progress on my house, my cars, and my computers. I haven't been chasing contracts as much as I should but I try to keep people knowing that I'm available. My preference would be to find something here in Ohio, or at least with people we know.

It's been about 15 years, since I really spent any significant time at my house and it shows. There is a possibility, I will be back at L3, either Florida or Utah, and it may be years again before I get another chunk of time to work on some of this.

The outside is improving, where I have spent most of my time. Adam Katter who does my lawn, is good at showing up and cutting the lawn when it needs to be cut. But his crew only cuts where they think its safe to cut. The area that they will cut has been shrinking for the last 15 years and I really wanted to get back to them mowing most of the lawn. With about three days work, I resurrected one of my two riding lawn mowers and savagely cut down lots of areas they would not mow. At the very end, the secondary deck belt snapped, but on Monday Katter's crew showed up and cut all of the area I mowed down. A victory and the lawn looks better.

Today was replacing the deck belt and un-doing some more of the jury-rigging someone had done to the deck. I couldn't find the right belt, but found one that was close. On my free John Deere, there is a relationship between the tension on the primary deck belt and the secondary deck belt. You need both belts to be the correct size in order for the tension to be good on both belts. I fashioned an offset plate for tensioning the primary deck belt until Amazon can bring me the correct belt.

I wanted to get the mower going to mow the spot where the old RV was sitting. The hoses and wires for the old RV were trapped under a collection of trash cans that Uncle had retried when they developed cracks or holes. With a sawsall, I was able to cut up the trash cans and make them and their contents fit in the newer trash can. Then I was able to disconnect the hoses and electricity to the RV, get it started and get it out of its hole. And then savagely cut down what was growing around it. Another victory..

I'm not sure what I want to do with the old RV, but I will at least replace the fan controller, pressure wash it with the recently repaired pressure washer, and probaby put a new exhaust on it.

Next big problem is I have a brush pile that is almost a story high. I got the Dodge Cummin's pickup running last weekend and plan to bring my cousin's chipper over and spend a day turning the brush pile into a smaller pile. I will put new wood on my trailer bed in order to do this.

Twice this winter, the van needed a bit of freon. I got under it today and it appears to be the seam where the two compressor halves come together. This means I will need to order another compressor.

My computers have been suffering without a real internet connection. It has been impractical to do many updates with metered cell phone hotspots. Now that I have Starlink, I have been going through and updating them. I have another laptop to backup and update tonight. I have another Qubes NUC that is not taking updates and will probably need to be backed up and re-installed. I think its just too far out of date.

 Now onto the laptop...

  

RISC-V Ubuntu Virtual Machine

I'm pretty enthused with the open RISC-V architecture and have been meaning to get a RISC-V virtual machine up and going so I can learn the assembly language. Like many pieces of open source software, it didn't go that easy.

 

There is Debian, Fedora and Ubuntu versions of Linux for the RISC-V. I randomly chose Ubuntu.

 

I had trouble with U-Boot not being able to find the Device Tree Binary, the .dtb file, and found a post, which indicated I should get a specific version of U-Boot and compile it.

 

From:

 

https://discourse.ubuntu.com/t/ubuntu-server-on-risc-v-documentation-needs-updating/23927/4

 

The user xypron, supplied a script to build U-Boot.

 

#!/bin/sh

set -e

if test ! -f opensbi/build/platform/generic/firmware/fw_payload.bin; then
wget https://cdimage.ubuntu.com/releases/20.04/release/ubuntu-20.04.3-preinstalled-server-riscv64+unmatched.img.xz
xz -dk ubuntu-20.04.3-preinstalled-server-riscv64+unmatched.img.xz
export CROSS_COMPILE=riscv64-linux-gnu-
git clone https://source.denx.de/u-boot/u-boot.git
cd u-boot/
git reset --hard v2021.10-rc3
make qemu-riscv64_smode_defconfig
make -j$(nproc)
cd ..
git clone https://github.com/riscv/opensbi.git
cd opensbi/
make PLATFORM=generic FW_PAYLOAD_PATH=../u-boot/u-boot.bin
cd ..
fi
qemu-system-riscv64 -machine virt -m 1G -nographic \
-bios opensbi/build/platform/generic/firmware/fw_payload.bin \
-smp cores=2 -gdb tcp::1234 \
-device virtio-net-device,netdev=net0 \
-netdev user,id=net0,tftp=tftp \
-drive if=none,file=ubuntu-20.04.3-preinstalled-server-riscv64+unmatched.img,format=raw,id=mydisk \
-device ich9-ahci,id=ahci -device ide-hd,drive=mydisk,bus=ahci.0 \
-device virtio-rng-pci

 

I had already downloaded the ubuntu-20.04.4 image, so I removed the wget and changed the 20.04.3 references to 20.04.4.

 

This script built a version of U-Boot and OpenSBI that allowed the Ubuntu RISC-V image to boot.

 

Another piece that was helpful to me was to resize the disk image.

 

From the helpful Ubuntu Wiki:

 

https://wiki.ubuntu.com/RISC-V

Optionally, if you want larger disk, you can expand the disk (filesystem will be automatically resized too).

qemu-img resize -f raw focal-preinstalled-server-riscv64.img +5G

New Favorite Trackball - Elecom M-XT3UR

Years ago, I was dealing with carpal tunnel syndrome pain, pain that would keep me awake at night. I built my own braces, built a sling where I could sleep with my hands elevated at night. I was trying a bunch of things and wanted to avoid surgery. Some of my friends had benefited from surgery, some had not. I stumbled onto split keyboards and trackballs. With split keyboards and trackballs, I found the pain diminishing, to the point today where I rarely think about pain. For me, split keyboards and trackballs let me type all day.

For keyboards, my favorite is the Microsoft Wired Natural Ergonomic Keyboard 4000.

For trackballs, my favorite has been the Wired Logitech Trackman Wheel, T-BB18. These have been out of production for years, and I have been replacing them with the Wireless Logitech Trackman Wheel, M370. The M370 is pretty good, except for the batteries. The batteries last a long time, but when they start going low, double clicking, dragging, and other operations get a bit flakey. This drives me nuts when I am trying to get something important done and I can't get double clicks to work cleanly. I really want a wired trackball. I don't want to worry about batteries.

Many recommended the Wired Elecom M-XT3UR as a replacement, and I can support that recommendation. Its a bit smaller than the M370 and I think I would like a trackball that is larger than the M370, but after a weekend I feel at home and find myself wanting a spare to take into work. I want to replace my M370s with the Elecom trackballs.

Thumbs up to the Elecom M-XT3UR.

LetsEncrypt

This weekend, I decided to try the EFF’s LetsEncrypt certificates. It was easy for Apache, not as well worked out for Postfix/Dovecot.

My self signed certificates on a couple of my Linux boxes have been torturing me when using Apple Mail. Apple is doing the right thing trying to warn you about self signed certificates.

I tried to go straight to the EFF email solution STARTTLS. After fumbling around with that for many hours, I decided I would get Apache working and then tackle Postfix/Dovecot. I don’t really want Apache on these boxes nor do I want them exposed directly to the internet, but it seemed like a good way to start understanding LetsEncrypt.

I cleared out all the Apache stuff that I had played with previously on a Debian system, reinstalled Apache2, and used these instructions to get the LetsEncrypt certificate installed:

https://www.digitalocean.com/community/tutorials/how-to-secure-apache-with-let-s-encrypt-on-ubuntu-16-04

I looked at a number of different pages about LetsEncrypt and this set of instructions seemed straight forward.

I poked a hole in my firewall for apache, which was necessary for LetsEncrypt and setup a public DNS entry for that Debian box.

The Apache certificates installed painlessly.

Next I tackled the Postfix/Dovecot certificates. I ended up using this set of instructions:

https://www.upcloud.com/support/secure-postfix-using-lets-encrypt/

This took more fumbling around but it works and now my Apple mail is happy with reading mail from my main Debian box.

I decided to leave the Debian box with the default web page up overnight in case I discovered something that would require re-doing the certificates.  Oh wow! The internet is a dangerous place. I thought nobody would notice my Debian box. The logs this morning were just full of random IP addresses trying all sorts of non-existant URLs, most ending in “.php”. I was glad I had cleared out all my old experiments and reinstalled Apache. I had what I wanted, so I turned off all external access again.

For Apache, LetsEncrypt is easy. I’m am going to play more with STARTTLS and see if I can find a way to make easy to do. Maybe I’m just misunderstanding something.

Plug for SportsTalkDM.com

This is a shameless plug for my cousin’s boy Daniel.

Daniel has been quoting sports scores to me since he was 6 or 7 years old. I think he got his iPod when he was about 8 and he would sit in his room with his little miniature NFL helmets in front of him and make videos of himself commenting on upcoming football matches. A few weeks back, the now 14 year old Daniel asked me, what I would use to make a website.  Since then he has created a website, SportsTalkDM.com, Sports Talk with Daniel, and a Patreon page, Patreon.com/sportstalkdm and posted a bunch of content. I think his NFL article is pretty good for a 14 year old. Like most first time web authors, he is watching for every page view. I haven’t helped him to this point, but I probably will this weekend, since adding the Javascript for Google AdSense has got him stumped. I would appreciate if anyone could click on his page. He doesn’t have feedback or anyway to comment on his main site, but you can comment and like on his Patreon page. I think this is pretty good and I want to try to encourage him.

DO-178B/C Friendly Debugging Macros

Sorting through some PDFs and a PDF I had saved if from a friend, Mike Potts, mjp66@sbcglobal.net, that is worth talking about.

In DO178B/C code, you want each line of code covered by a requirement or requirements, and ideally nothing else. The requirements say what you are going to do and only what you are going to do. Every DO178 project I have been in has had to deal with dead or deactivated code. There are many cases where you would like to keep some debugging code in there that you could re-activate when things go wrong to see information. Mike has a set of C macros that leave no code behind when compiled for production. Not even stray semi-colons.

I have used this philosophy many times over the years and keep a set of notes he gave me long ago about how to use this:


Hello, 

Ed asked about debug printf.

Here is how it works, you enable Serial-Input-Output (SERIO) Debugging Prints with #define SERIO_DEBUG.

To issue a debugged printf, you call it with two parentheses and NO semicolon.

Yes it looks weird, and takes some getting use to, but watch.

SERIO_DEBUG_MSG(("NMEA Semap, BUSY %11d\n", hi_pri_clock_read()/TICKS_PER_MS_HIGH))

A bunch of these macros are stored in the debugging module's header file:

#if (defined(SERIO_DEBUG))
#define SERIO_DEBUG_MSG(macro_string) printf macro_string;
#else
#define SERIO_DEBUG_MSG(macro_string)
#endif

If SERIO_DEBUG is defined, then the SERIO_DEBUG_MSG and the first set of parentheses turn into a printf and a semicolon.

If SERIO_DEBUG is not defined, the while macro disappears (and does not even leave a semicolon behind).

This is kinda nice, not even a semicolon left behind to disturb certification. I like pre-processor tricks.
If you want to carry this idea forward, you can use something similar to:

#ifdef LINK_ERROR #include <debug.h>
#define GPS_ERROR_MSG(macro_string) {debug_msg macro_string; debugbreak();}
#else
#define GPS_ERROR_MSG(macro_string) debug_msg macro_string;
#endif

This error printf happens to be writen as non-maskable, but can optionally issue a debugger break statement that is provided by a specific tool set.

We were consistent with this naming convention.

Enables were always *_DEBUG, and the associated debugging prints were always *_DEBUG_MSG.
This allowed us the option of sprinkling some extra computation code (within #ifdef *_DEBUG) near the printfs and still 'clearly' be debugging code.

These was also a master debug_on macro to globally enable the whole group.

-Mike

Mike is a really sharp guy and always goes that the extra distance to make things right. One of his recent project's was to put together a really excellent setup guide for the Ubiquity EdgeRouter X, a fantastic little router. His setup guide was mentioned on Security Now, Episode 641, before Christmas and can be found at https://github.com/mjp66/Ubiquiti