Debugging Windows HVMs

Debugging Windows code can be tricky in a virtualized environment. The guide below assumes Xen hypervisor and Windows 7 VMs.

User-mode debugging is usually straightforward if it can be done on one machine. Just duplicate your normal debugging environment in the VM.

Things get complicated if you need to perform kernel debugging or troubleshoot problems that only manifest on system boot, user logoff or similar. For that you need two Windows VMs: the host and the target. The host will contain WinDbg installation, your source code and private symbols. The target will run the code being debugged. Both will be linked by virtual serial ports.

First, you need to prepare separate copies of both target and host VM configuration files with some changes. Copy the files from /var/lib/qubes/appvms/vmname/vmname.conf to some convenient location, let's call them host.conf and target.conf.
In both copied files add the following line at the end: serial = 'pty'. This will make Xen connect VM's serial ports to dom0's ptys.
From now on you need to start both VMs like this: qvm-start --custom-config=/your/edited/host.conf host
To connect both VM serial ports together you will either need socat or a custom utility described later.
To determine which dom0 pty corresponds to VM's serial port you need to read xenstore, example script below:
```
#!/bin/sh

id1=$(xl domid "$1-dm")
tty1=$(xenstore-read /local/domain/${id1}/device/console/3/tty)
echo $tty1
```
Pass it a running VM name and it will output the corresponding pty name.
To connect both ptys you can use socat like that:
```
#!/bin/sh

id1=$(xl domid "$1-dm")
id2=$(xl domid "$2-dm")
tty1=$(xenstore-read /local/domain/${id1}/device/console/3/tty)
tty2=$(xenstore-read /local/domain/${id2}/device/console/3/tty)
socat $tty1,raw $tty2,raw
```
...but there is a catch. Xen seems to process the traffic that goes through serial ports and changes all 0x0a bytes into 0x0d, 0x0a pairs (newline conversion). I didn't find a way to turn that off (setting ptys to raw mode didn't change anything) and it's not mentioned anywhere on the Internet, so maybe it's something on my system. If the above script works for you then you don't need anything more in dom0.
On the target system, run bcdedit /set debug on on the console to turn on kernel debugging. It defaults to the first serial port.
On the host system, install WinDbg and start the kernel debug (Ctrl-K), choose com1 as the debug port.
Reboot the target VM.
Run the above shell script in dom0.

If everything is fine you should see the proper kernel debugging output in WinDbg. However, if you see something like that:

Opened \\.\com1
Waiting to reconnect...
Connected to Windows 7 7601 x64 target at (Wed Mar 19 20:35:43.262 2014 (UTC + 1:00)), ptr64 TRUE
Kernel Debugger connection established.
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
... Retry sending the same data packet for 64 times.
The transport connection between host kernel debugger and target Windows seems lost.
please try resync with target, recycle the host debugger, or reboot the target Windows.
Unable to read KTHREAD address fffff8000281ccc0
**************************************************************************
Unable to read debugger data block header
**************************************************************************
Unable to read KTHREAD address fffff8000281ccc0
Unable to read PsLoadedModuleList
Unable to read KTHREAD address fffff8000281ccc0
**************************************************************************
Unable to read debugger data block header
**************************************************************************

...then you're most likely a victim of the CRLF issue mentioned above. To get around it I wrote a small utility that basically does what socat would do and additionally corrects those replaced bytes in the stream. It's not pretty but it works:

#include <errno.h>
#include <stdio.h>
#include <fcntl.h>
#include <termios.h>

int fd1, fd2;
char mark = ' ';

void out(unsigned char c)
{
    static int count = 0;
    static unsigned char buf[17] = {0};

    // relay to ouptput port
    write(fd2, &c, 1);
    fprintf(stderr, "%c", mark);

    /* dump all data going over the line
    if (count == 0)
        fprintf(stderr, "%c", mark);
    fprintf(stderr, "%02x ", c);
    if (c >= 0x20 && c < 0x80)
        buf[count] = c;
    else
        buf[count] = '.';
    count++;
    if (count == 0x10)
    {
        count = 0;
        fprintf(stderr, " %s\n", buf);
    }
    */
}

int main(int argc, char* argv[])
{
    unsigned char c = 0;
    struct termios tio;
    ssize_t size;

    if (argc < 3)
    {
        fprintf(stderr, "Usage: %s pty1 pty2 [mark character]\n", argv[0]);
        return EINVAL;
    }

    fd1 = open(argv[1], O_RDONLY | O_NOCTTY);
    if (fd1 <= 0)
    {
        perror("open fd1");
        return errno;
    }
    fd2 = open(argv[2], O_WRONLY | O_NOCTTY);
    if (fd2 <= 0)
    {
        perror("open fd2");
        return errno;
    }
/*
    // This doesn't make any difference which supports the theory
    // that it's Xen who corrupts the byte stream.
    cfmakeraw(&tio);
    if (tcsetattr(fd1, TCSANOW, &tio) < 0)
    {
        perror("tcsetattr 1");
        return errno;
    }
    if (tcsetattr(fd2, TCSANOW, &tio) < 0)
    {
        perror("tcsetattr 2");
        return errno;
    }
*/
    if (argc == 4)
        mark = argv[3][0];

    while (1)
    {
        size = read(fd1, &c, 1);
        if (size <= 0)
            break;

parse:
        if (c == 0x0d)
        {
            size = read(fd1, &c, 1);
            if (size <= 0)
            {
                out(0x0d);
                break;
            }
            if (c == 0x0a)
            {
                out(0x0a);
            }
            else
            {
                out(0x0d);
                goto parse;
            }
        }
        else
            out(c);
    }

    close(fd1);
    close(fd2);
    return 0;
}

This utility is a unidirectional relay so you need to run two instances to get duplex communication, like:

#!/bin/sh

id1=$(xl domid "$1-dm")
id2=$(xl domid "$2-dm")
tty1=$(xenstore-read /local/domain/${id1}/device/console/3/tty)
tty2=$(xenstore-read /local/domain/${id2}/device/console/3/tty)
./ptycrlf ${tty1} ${tty2} - &
./ptycrlf ${tty2} ${tty1} + &

With this everything should be good:

Opened \\.\com1
Waiting to reconnect...
Connected to Windows 7 7601 x64 target at (Wed Mar 19 20:56:31.371 2014 (UTC + 1:00)), ptr64 TRUE
Kernel Debugger connection established.
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Kernel Version 7601 MP (1 procs) Free x64
Built by: 7601.18247.amd64fre.win7sp1_gdr.130828-1532
Machine Name:
Kernel base = 0xfffff800`0261a000 PsLoadedModuleList = 0xfffff800`0285d6d0
System Uptime: not available

Debugging HVMs in the Qubes R4.0

There are two main issues to be adopted to get all things to work in the R4.0.

Add a virtual serial port

Qemu in the stub domain with virtual serial port added in a recommended way (<serial type="pty"/>) fails to start (Could not open '/dev/hvc1': No such device). It seems like a lack of multiple xen consoles support/configuration. The only way that I have found is to attach serial port explicitly to the available console.

Unpack stub domain in dom0:

$ mkdir stubroot
$ cp /usr/lib/xen/boot/stubdom-linux-rootfs stubroot/stubdom-linux-rootfs.gz
$ cd stubroot
$ gunzip stubdom-linux-rootfs.gz
$ cpio -i -d -H new --no-absolute-filenames < stubdom-linux-rootfs
$ rm stubdom-linux-rootfs

Edit Init script to remove last loop and to add "-serial /dev/hvc0" to the qemu command line.
Apply changes:

$ find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../stubdom-linux-rootfs
$ sudo mv ../stubdom-linux-rootfs /usr/lib/xen/boot

Connect two consoles

Run the following script:

debugname1=win7new
debugname2=win7dbg
id1=$(xl domid "$debugname1-dm")
id2=$(xl domid "$debugname2-dm")

tty1=$(xenstore-read /local/domain/${id1}/console/tty)
tty2=$(xenstore-read /local/domain/${id1}/console/tty)

socat tty1,raw $tty2,raw

Happy debugging!

9.7 KiB Raw Blame History

Debugging Windows HVMs

Debugging HVMs in the Qubes R4.0

Add a virtual serial port

Connect two consoles

9.7 KiB

Raw Blame History