QEMU Developer Guide

Alex Bennée

April 2020

Introduction

  • Alex Bennée
    • alex.bennee@linaro.org
    • stsquad on #qemu
  • Virtualization Developer @ Linaro
  • Projects:
    • QEMU, KVM, ARM

What is QEMU?

From: www.qemu.org

"QEMU is a generic and open source machine emulator and virtualizer."

Two Types of Virtualization

  • Hardware Assisted Virtualization (KVM*)
  • Cross Architecture Emulation (TCG)

Hardware Assisted Virtualization

High Performance, Cloud, Server Consolidation

KVM Virtualisation

Full System Emulation

Android Emulator, Embedded Development, New Architectures

tcg-system-emulation.png

Linux User Emulation

Cross-development tools, Legacy binaries

linux-user-emulation.png

Technologies

Language and Libraries

  • C (C99 + GNU extensions)
  • glue(glue(glue(…)))
  • GLib
    • but own lists QLIST_*

QEMU Object Model (QOM)

  • Namespaces
    • Device
    • Properties
  • HMP: info qom-tree
  • QDev devices are QOM objects
    • all based off DeviceState
    • still some legacy in code base
  • see hw/misc/unimp.c

Command Line

  • Legacy
--drive if=ide,media=cdrom,...
  • Architected
-netdev user,id=unet,hostfwd=tcp::8022-:22 -device virtio-net-device,netdev=unet
  • Convenience
--cdrom filename

Dynamic Configuration

  • HMP (for Humans)
${QEMU} -serial mon:stdio ...
C-a c
(qemu)
  • QMP (QEMU Machine Protocol)
    • QAPI schema
    • JSON protocol
    • discovery/introspection

Tracing

  • per dir trace-events
  • trace_foo_baa(arg1, arg2)
  • multiple backends
    • simple (binary trace file)
    • log (printf fmt stderr)
    • dtrace/ftrace
    • lttng user-space tracing
  • see docs/devel/tracing.txt

Synchronisation Primitives

  • iothread_mutex (a.k.a BQL)
  • Read Copy Update
  • Sequence Locks
  • QEMU Hash Table (QHT)
  • atomics (with/without mem barriers)

Code

Prerequisites

  • sh/make/glibc/glib/fdt/python3
  • gcc or clang
  • apt build-dep qemu

Configure Overview

  • use out-of-tree builds
  • –target-list is your friend
  • plain POSIX shell (not autotools)
  • ./tests/tcg/configure.sh

Plain

mkdir -p builds/all && cd builds/all
../../configure

Quick ARM builds

mkdir -p builds/arm.all && cd builds/arm.all
../../configure --disable-tools --disable-docs \
  --target-list=aarch64-softmmu,aarch64-linux-user,arm-softmmu,arm-linux-user

Debug Builds

mkdir -p builds/all.debug && cd builds/all.debug
../../configure --enable-debug --extra-cflags="-O0 -g3"

Sanitiser Builds

mkdir -p builds/all.clang-sanitizer && cd builds/all.clang-sanitizer
../../configure --cc=clang-7 --cxx=clang++-7 \
  --enable-sanitizers \
  --target-list-exclude=xtensa-softmmu,xtensa-linux-user \
  --extra-cflags="-O0 -g3"

Building Overview

  • make -j(nproc)
  • make -j(nproc) aarch64-linux-user/all
  • make help

Running Tests

  • make check
  • make check-tcg
  • make check-acceptance

TCG tests

  • Need cross-compiler
  • Individual targets
    • make build-tcg-tests-aarch64-linux-user
    • make run-tcg-tests-aarch64-linux-user

Docker Subsystem

  • make docker
  • make docker-test-build@debian-armhf-cross J=9 V=1

Translation Overview

Documentation

System Emulation Process Model

Current threading model

vCPU Run Loop (TCG)

simplified run loop

target/*

  • target/FOO/translate.c
    • gen_intermediate_code
      • translator_loop (common)
        • .init_disas_context
        • .tb_start
          • .translate_insn
          • .translate_insn
        • .tb_end

DisasContext

  • local state for translation
  • DisasContextBase
    • tb, pc, jump, ss state
  • DisasContext
    • arch specific flags
    • including state of TB Flags

Translation Block Flags

  • State relevant to translation
  • Part of next TB lookup
    • same PA -> multiple TBs
  • Calculated or cached
    • cache must be correct

Translate Instruction

Decode Opcode

uint32_t insn = arm_ldl_code(env, s->base.pc_next, s->sctlr_b);
...
int rd = extract32(insn, 0, 5);
int rn = extract32(insn, 5, 5);
uint64_t imm = extract32(insn, 10, 12);
bool setflags = extract32(insn, 29, 1);
bool is_64bit = extract32(insn, 31, 1);

See also decodetree

Fetch Registers

TCGv_i64 tcg_rn = cpu_reg(s, rn);
TCGv_i64 tcg_rd = setflags ? cpu_reg(s, rd) : cpu_reg_sp(s, rd);
TCGv_i64 tcg_result;

Fetch Registers 2

cpu_X[i] = tcg_global_mem_new_i64(cpu_env,
                                  offsetof(CPUARMState, xregs[i]),
                                  regnames[i]);
...
...
TCGv_i64 cpu_reg(DisasContext *s, int reg)
{
    if (reg == 31) {
        return new_tmp_a64_zero(s);
    } else {
        return cpu_X[reg];
    }
}

Fetch Registers 3

static TCGv_i64 read_fp_dreg(DisasContext *s, int reg)
{
    TCGv_i64 v = tcg_temp_new_i64();

    tcg_gen_ld_i64(v, cpu_env, fp_reg_offset(s, reg, MO_64));
    return v;
}

Do Operation

  • use TCG primitives

or

  • use TCG helper

TCG Primitives

tcg_result = tcg_temp_new_i64();
tcg_gen_subi_i64(tcg_result, tcg_rn, imm);
if (is_64bit) {
    tcg_gen_mov_i64(tcg_rd, tcg_result);
} else {
    tcg_gen_ext32u_i64(tcg_rd, tcg_result);
}
tcg_temp_free_i64(tcg_result);

see tcg/README and include/tcg/tcg.h

TCG Helpers

/* in helper.h */
DEF_HELPER_2(my_sub, i64, i64, i64)
/* in foo_helper.c */
uint64_t HELPER(my_sub)(uint64_t a, uint64_t b)
{
    return a - b;
}
/* in translate.c */
TCGv_i64 tcg_imm = tcg_const_i64(imm);
gen_helper_my_sub(tcg_rd, tcg_rn, tcg_imm);
tcg_temp_free_i64(tcg_imm);

TCG Load/Store

  • tcg_gen_[ld|st]_i[32|64]
    • works on qemu's address space
    • typically env + offset
  • tcg_gen_qemu_[ld|st]_i[32|64]
    • works on guests address space
      • TCGAddr (calculated guest address)
      • idx (MMU index)
      • MemOp (size of operation)

Debugging

${QEMU} -d in_asm,op,op_opt,out_asm [-s -S|-g] ${TESTCASE}
gdb-multiarch ${TESTCASE} -ex "target remote localhost:1234"

Useful Debugging Options

  • -d help
  • -D output.log
  • -dfilter 0x10000-0x20000,0x40000+0x100
  • use rr

Setup using libvirt

  • use libvirt/virt-manager
/usr/bin/qemu-system-ppc64le -name guest=debian-buster-ppc64le,debug-threads=on \
  -S \
  -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-12-debian-buster-ppc64l/master-key.aes \
  -machine pseries-2.12,accel=tcg,usb=off,dump-guest-core=off \
  -m 4096 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 \
  -uuid a554f3b4-f59a-4879-a245-b5bf905aed0e \
  -no-user-config -nodefaults \
  -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-12-debian-buster-ppc64l/monitor.sock,server,nowait \
  -mon chardev=charmonitor,id=monitor,mode=control \
  -rtc base=utc \
  -no-shutdown -boot strict=on \
  -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.0,addr=0x3 \
  -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 \
  -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 \
  -drive file=/dev/zvol/hackpool-0/debian-buster-ppc64el,format=raw,if=none,id=drive-virtio-disk0,cache=none,aio=native \
  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
  -drive if=none,id=drive-scsi0-0-0-0,readonly=on \
  -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 \
  -netdev tap,fd=28,id=hostnet0 \
  -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d9:0e:28,bus=pci.0,addr=0x1 \
  -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 \
  -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-12-debian-buster-ppc64l/org.qemu.guest_agent.0,server,nowait \
  -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
  -chardev spicevmc,id=charchannel1,name=vdagent \
  -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 \
  -device usb-kbd,id=input0,bus=usb.0,port=1 -device usb-mouse,id=input1,bus=usb.0,port=2 \
  -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on \
  -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x9 \
  -device intel-hda,id=sound0,bus=pci.0,addr=0x2 \
  -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 \
  -object rng-random,id=objrng0,filename=/dev/urandom \
  -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x8 \
  -msg timestamp=on

Simplified Command Line

./ppc64-softmmu/qemu-system-ppc64 \
  -machine pseries-2.12 -m 4096 -smp 8 \
  -nodefaults \
  -serial mon:stdio \
  -no-shutdown -boot strict=on \
  -device virtio-scsi-pci,id=scsi0 
  -drive file=/dev/zvol/hackpool-0/debian-buster-ppc64el,format=raw,if=none,id=drive-virtio-disk0 \
  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
  -drive if=none,id=drive-scsi0-0-0-0,readonly=on \
  -nic user,model=virtio-net-pci,hostfwd=tcp::2222-:22 \
  -display none

Virtualisation Overview

vCPU RUN Loop (KVM)

  • qemu_kvm_cpu_thread_fun()
    • kvm_cpu_exec()
      • ioctl(KVM_RUN…)

Questions?