QEMU notes

Table of Contents

1 Tasks [1/3]   tasks

1.1 DONE clean up binfmtmisc setup

  • CLOSING NOTE [2019-02-13 Wed 14:07]
    In my last testing/next merge

We have the Persistent (F) flag now.

1.2 TODO Update/cleanup tcg/README

* stsquad ponders it being time to convert tcg/README to markdown       [18:38]
<pm215> restructured text, please
<pm215> (should be in docs/devel/, really)

  • [ ] move to docs, convert to RST
  • [ ] document generic helpers

<2017-10-31 Tue 18:39>

1.3 DONE TLB Optimisations

Make a little sub-routine for TLB lookup for better cache locality as suggested by cota_

Paper: Optimizing Memory Emulation in Full System Emulators More detailed paper: Optimizing Memory Emulation in Full System Emulators

Emilio running with this on list

1.4 TODO GDB/Kernel/OS Interaction

Also known as the single-stepping through the kernel without going nuts. See discussion at weekly priorities meeting

1.5 Next Cycle

  • some minor -M profile
  • more v8.x stuff

1.6 BKK19 Meeting

  • nvdimm - in progress should be 4.1
  • cpu hot plug
    • why not instantiate and sleep them (not much use)
      • dynamic creation complicates things like GIC allocation
      • naive implementation instantiate max cpus and sleep non used
    • kata containers upstream wants ACPI not the current ARM workaround
      • see RFC on the kata list
      • powerpc also has hotplug (but doesn't support acpi)
      • Marc is wary about implementing something in virt that doesn't follow the architecture

1.7 TODO GSoC interview

1.7.1 Follow-up questions

  • given more time, would you choose a different data structure?
  • how would you change your program if the switch only has 32 ports?" (scaling down)
  • how does your solution perform when the number of ports is very large?" (scaling up)
  • modern network switches have multiple cores, how would you exploit those hardware resources?" (multi-threading)
  • also that leads to cache-efficiency of trees vs arrays, concurrency…

1.8 TODO fix up risu with empty –trace

<pm215> ajb-linaro: just noticed that if you pass risu –trace a [15:00] filename it can't write to (eg in a nonexistent directory) it silently claims to have succeeded

<2019-06-04 Tue 15:20>

2 Maintainer Tasks [0%]

2.1 fpu/next [0/0]

2.3 gitdm/next

3 Postponed tasks [2/9]

3.1 TODO tcg/perf tweaks

Should really resurrect these

3.2 TODO Re-spin debug/logging patches

3.3 BLOCKED pstate simplification

The initial effort proved to be more complicated than seemed right.

3.4 TODO Add a architectural breakpoint test

While gdbstub insertion is probably safe using architectural breakpoints is likely to break things.

3.5 BLOCKED Profile current Android emulator (VIRT-118)

3.6 MTTCG Clean-up tasks

3.6.1 TODO Add large pages to unit test cases

We need to ensure the cputlb code works ok with large pages

3.7 TODO fix the default coroutine for sanitizer

Now we can use the latest sanitizer support.

<bonzini> stsquad: what is glib's racy gthread support? :)
<bonzini> stsquad: yeah, i meant that we should get to the point where
    the equivalent of KVM_RUN takes BQL as little as possible
<bonzini> stsquad: then for the rest we can look at TCG and KVM in
    parallel
> bonzini: re:gthread I'm seeing tsan warnings whenever gthread spawns
    a new thread - as thread->name is set in one and read in another
    outside of the lock
<bonzini> stsquad: ok, let me look at it
<bonzini> stsquad: but it could be a tsan bug
<bonzini> stsquad: pthread_create is release, and starting the thread
    is acquire
> e.g:
> http://ix.io/1BrO
> bonzini: ^ it might be, the wording makes me suspicious
> bonzini: because pthread_create is called first - and then the
    resulting thread structure is modified after the thread is spawned
> c.f:
> http://ix.io/1BrP
<bonzini> stsquad: it's a tsan bug
<bonzini> stsquad: G_LOCK(g_thread_new) provides ordering
<bonzini> stsquad: that's the "acquire" in g_thread_proxy
<bonzini> stsquad: but why are you using gthread at all, rather than
    qemu-thread?
> bonzini: ahh - it won't continue after the lock until the originator
    unlocks
<bonzini> stsquad: yup, that's what the G_LOCK/G_UNLOCK pair does
> bonzini: these are test case failures
<bonzini> stsquad: oh, tests use gthread
> bonzini: in this case ./tests/test-char
<bonzini> stsquad: what's the tsan splat like?
> http://ix.io/1BrS
> bonzini: ^
> actually (without the latest glib .so)
> http://ix.io/1BrT
> hmm and sadly coroutines fail later on:
> http://ix.io/1BrU
> but that is tsan failing to find the longjmp buf
stsquad tries with the llvm build of libtsan
<bonzini> strange it doesn't report the mutex in the splat
<bonzini> stsquad: okay that longjmp thing might be easier to fix
<bonzini> stsquad: we use swapcontext the first time, and then
    setjmp/longjmp
<bonzini> stsquad: perhaps we could have a different backend that does
    pthread_sigmask (block all signals)+swapcontext+pthread_sigmask
    (restore previous signal mask)
> bonzini: why a whole new backend?
<bonzini> stsquad: yeah, i guess it could be #ifdef __SANITIZER__
> bonzini: it's good that I don't need the gthread corutine anymore so
    if we can fix it up
<bonzini> stsquad: in fact, the common parts of
    coroutine-sigaltstack.c and coroutine-ucontext.c should be moved
    to common coroutine-posix.{c,h}
> bonzini: so what is the problem exactly - that we shouldn't mix
    swapcontext/setjmp_longjmp or just signals arriving as we
    swapcontext?
<bonzini> stsquad: both - i suppose we shouldn't mix
    swapcontext/setjmp_longjmp, and sigprocmask is needed to avoid
    signals arriving during swapcontext
> bonzini: why should masking be done only for __SANITZER__ then?
<bonzini> stsquad: for !sanitizer we can keep using longjmp, it's
    faster
<bonzini> stsquad: not sure why sanitizer doesn't like
    swapcontext+longjmp, but i guess it depends on how tsan implements
    swapcontext
<bonzini> stsquad: it may break with swapcontext too
<bonzini> stsquad: there are functions to mark the backend
    (http://llvm.org/viewvc/llvm-project?view=revision&revision=273260)
<bonzini> stsquad: but it seems to be for asan only
<bonzini> stsquad: anyhow, we can try not using longjmp and see what
    happens
> bonzini: ok I'll have a look tomorrow
> bonzini: it's a bit late to start off into the weeds now ;-)
<bonzini> yeah :)
>

3.8 DONE ACTION Fixup RISU tests in LAVA

  • CLOSING NOTE [2017-10-13 Fri 16:47]
    Finished a while ago.

Have started failing (risu tarball got killed?)

3.8.1 DONE Fix fmulxsquash tests

  • CLOSING NOTE [2017-10-13 Fri 16:46]
    bad data

3.9 ACTIVE Run RISU tests for pm215

<pm215> ajb-linaro: do you have a spare half hour to sort out the
    necessary risu testing for VIRT-377 (frecpe bug) ?
> pm215: spare is a loaded word, but sure
<pm215> that's in my "would be kinda nice to fix for 2.2 and it's
    easy" list
ajb-linaro checks his image library
<pm215> (the fix is just s/1023/2045/)
<pm215> we probably need better test images or we'd have caught it the
    first time around
> pm215: ok - I'll generate a bigger set
<pm215> VIRT-381 was my other maybe-for-2.2 but that's just an UNDEF
    case so I don't really care
<chazy> let’s focus on getting the Android debugging and KVM debugging
    stuff out the door in November, and take these things up in early
    December?
<chazy> (of course if ajb-linaro needs a 30 minute break from the
    Android build system here and then, that’s fine too :) )
> well the main thing is being able to get onto a Lab machine
<pm215> the thing about VIRT-377 is that I really want a fix by next
    Tuesday, or it's not worth bothering because it's missed the boat
    for 2.2

3.9.1 QEMU SVE implementation from company C

Initial implementation - Chris Ashton <cja@cray.com>

3.9.2 Q: what version of QEMU are you based on?

Working on master.. as of a few months ago. linux-user

3.9.3 Q: does the work include 8.1/8.2/8.3 pre-requisites?

atomic support from 8.1, complex numbers

3.9.4 Q: expanded existing SoftFloat 2a or independent FP16 et all?

ad-hoc

3.9.5 Q: what was the instruction validation strategy?

Tested with mini-applications that have vectorised loops

  • lulash (coral2)
  • HPC benchmarks

some hand-written, some compiler Micro-kernels

3.9.6 Q: any licensing problems, happy for code to be GPLv2?

4 Notes

4.1 Invocation

4.1.1 Introspection of machine and cpu types

Boot with various start-up modes:

./aarch64-softmmu/qemu-system-aarch64 -machine type=virt,help
./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu help

4.1.2 My Current ARMv8 Invocation

This is an "old school" virt machine using user networking, a custom kernel and virtio block device

./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-version=3,virtualization=on \
                                      -cpu cortex-a53 \
                                      --serial mon:stdio \
                                      -netdev user,id=unet,hostfwd=tcp::2222-:22 -device virtio-net-device,netdev=unet \
                                      -device virtio-blk-device,drive=myblock -drive file=/home/alex/lsrc/qemu/images/debian-stable-arm64.qcow2,id=myblock,index=0,if=none \
                                      -kernel /home/alex/lsrc/qemu/images/aarch64-current-linux-kernel-only.img \
                                      -append "console=ttyAMA0 root=/dev/vda1" \
                                      -display none -m 4096 -name debug-threads=on -smp 4

This is a virt machine with PCI user networking and virtio-scsci block devices with an EFI based boot into the block devices grub.

./aarch64-softmmu/qemu-system-aarch64 -machine virt,graphics=on,gic-version=3,virtualization=on \
                                      -cpu cortex-a53 \
                                      --serial mon:stdio \
                                      -nic user,model=virtio-net-pci,hostfwd=tcp::2222-:22 \
                                      -device virtio-scsi-pci
                                      -drive file=/dev/zvol/hackpool-0/debian-stretch-arm64,id=hd0,index=0,if=none,format=raw -device scsi-hd,drive=hd0 \
                                      -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd \
                                      -display none -m 8192 -name debug-threads=on -smp 8

4.1.3 Debugging

N:some source code here

-name

4.2 Foundation/FVP Boot notes

earlycon=pl011,mmio,0x1c090000 console=ttyAMA0 devicetree /boot/fvp-base-gicv3-psci.dtb

4.3 Kernel CI

ansible-playbook -i hosts site.yml -l kernel-ci-backend -c local -t install

Adding -vvv seems to unwedge ansibles install on Debian

You need to run with the virtenv python (and also start MongoDB)

/srv/.venv/kernel-ci-backend/bin/python server.py

4.4 Creating root file-systems

4.4.1 Using vmdebootstrap

This script provides an all in approach to creating images (it uses debootstrap underneath)

sudo ~/src/vmdebootstrap.git/vmdebootstrap --image jessie-arm64.img --size 5g --log test.log --arch arm64 --foreign /usr/bin/qemu-aarch64-static --no-kernel --distribution testing --package htop,tmux,emacs24,zile,openssh-server,git-core,locales
sudo chown $USER:$USER jessie-arm64.img
qemu-img convert -O qcow2 jessie-arm64.img jessie-arm64.qcow2

4.4.2 Debian Derivatives with debootstrap

The handy debootstrap command will create a first stage. You will need to run the second stage inside your real architecture.

fakeroot debootstrap --include=openssh-server --foreign --arch=arm64 trusty ubuntu-trusty.arm64 http://ports.ubuntu.com /usr/share/debootstrap/scripts/gutsy

The final line just works around the fact that older installs might not know about the new releases (this was a 12.04 box). Obviously use the debian equivalent for pure Debian.

Once set-up you'll need to run the second stage (in QEMU's case in the emulated environment, or via real hardware if you have it).

/debootstrab/debootstrap --second-stage

4.4.3 Just run the installer

4.5 Running Notes

4.5.1 Debian Jessie with virt disk, redirected serial and virt network with port forwarded sshd

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -display none -serial telnet:127.0.0.1:4444 -monitor stdio -smp 2 -m 2048 -kernel ../images/aarch64-current-linux-kernel-only.img --append "console=ttyAMA0 root=/dev/vda1" -drive file=../images/jessie-arm64.qcow2,id=myblock,index=0 -device virtio-blk-device,drive=myblock -netdev user,id=unet,hostfwd=tcp::2222-:22 -device virtio-net-device,netdev=unet

4.5.2 Example KVM aarch64 invocation

./aarch64-softmmu/qemu-system-aarch64 -enable-kvm -machine type=virt -cpu host -nographic -smp 1 -m 2048 -kernel aarch64-linux-3.15rc2-buildroot.img
 --append "console=ttyAMA0"

4.5.3 ARM Virtual Machine Invocation

~/lsrc/qemu/qemu.git/arm-softmmu/qemu-system-arm -machine virt "-cpu" "cortex-a15" "-machine" "type=virt" -display none "-smp" "1" "-m" "2048" "-kernel" aarch32-current-linux-initrd-guest.img "-append" "console=ttyAMA0" -serial mon:stdio

4.5.4 Redirect serial

-serial telnet:127.0.0.1:4444 nc -ktl 127.0.0.1 4444

4.5.5 Save/restore machine state

4.5.5.1 To save state
  • C-a c to enter console
stop
migrate "exec: cat > state_file"
4.5.5.2 To restore state
${QEMU} ${QEMU_ARGS} -incoming "exec: cat state_file"

Optionally you can add the -S flag to start in the stopped state and the "cont" in the command line.

4.5.6 Booting a virt system machine with disk and network

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel ../linux.git/arch/arm64/boot/Image  --append "console=ttyAMA0 root=/dev/vda debug=vc" -drive file=../rootfs/ubuntu-trusty.img,id=myblock,index=0 -device virtio-blk-device,drive=myblock -netdev user,id=unet -device virtio-net-device,netdev=unet

4.5.7 Booting Jessie

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel ../images/aarch64-current-linux-kernel-only.img --append "console=ttyAMA0 root=/dev/vda1" -drive file=../images/jessie-arm64.qcow2,id=myblock,index=0 -device virtio-blk-device,drive=myblock -netdev user,id=unet -device virtio-net-device,netdev=unet

4.6 Debugging Notes

4.6.1 Building with ThreadSanitizer

Current building with tsan needs -pie disabled and the right linker magic.

./configure --cc=clang --extra-cflags="-g3 -O0 -fsanitize=thread" --with-coroutine=gthread --enable-debug --enable-debug-info --target-list=arm-softmmu,aarch64-softmmu,arm-linux-user,aarch64-linux-user --disable-pie

4.6.2 Root File Systems

4.6.3 Configure Setups

This needs a proper working multiarch for debugging linux-user

cd ~/lsrc/qemu/qemu.git
./configure --disable-strip --enable-debug --enable-gcov --enable-virtfs --target-list=aarch64-linux-user,aarch64-softmmu,arm-linux-user,arm-softmmu --enable-trace-backend=ust --enable-profiler

Without tracing but multiarch-able

cd ~/lsrc/qemu/qemu.git
./configure --disable-strip --disable-guest-base --enable-debug --enable-gcov --enable-virtfs --target-list=aarch64-linux-user,aarch64-softmmu,arm-linux-user,arm-softmmu

This is a static build of Linux User only

cd ~/lsrc/qemu/qemu.git
./configure --disable-strip --enable-debug --static --target-list=aarch64-linux-user,arm-linux-user
4.6.3.1 Current setup
Install prefix    /usr/local
BIOS directory    /usr/local/share/qemu
binary directory  /usr/local/bin
library directory /usr/local/lib
module directory  /usr/local/lib/qemu
libexec directory /usr/local/libexec
include directory /usr/local/include
config directory  /usr/local/etc
local state directory   /usr/local/var
Manual directory  /usr/local/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /home/alex/lsrc/qemu/qemu.git
C compiler        cc
Host C compiler   cc
C++ compiler      c++
Objective-C compiler cc
ARFLAGS           rv
CFLAGS            -fprofile-arcs -ftest-coverage -g -pthread -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include   -g
QEMU_CFLAGS       -Werror -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common  -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all -I/usr/include/p11-kit-1     -I/usr/include/libpng12   -I/usr/include/spice-server -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/spice-1   -I/usr/include/libusb-1.0   -I/usr/include/pixman-1
LDFLAGS           -Wl,--warn-common -fprofile-arcs -ftest-coverage -Wl,-z,relro -Wl,-z,now -pie -m64 -g
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       aarch64-linux-user aarch64-softmmu arm-linux-user arm-softmmu
tcg debug enabled yes
gprof enabled     no
sparse enabled    no
strip binaries    no
profiler          no
static build      no
-Werror enabled   yes
pixman            system
SDL support       yes
GTK support       yes
VTE support       yes
curses support    yes
curl support      yes
mingw32 support   no
Audio drivers     oss
Block whitelist (rw)
Block whitelist (ro)
VirtFS support    yes
VNC support       yes
VNC TLS support   yes
VNC SASL support  yes
VNC JPEG support  yes
VNC PNG support   yes
VNC WS support    yes
xen support       yes
brlapi support    yes
bluez  support    yes
Documentation     yes
GUEST_BASE        no
PIE               yes
vde support       no
netmap support    no
Linux AIO support yes
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
RDMA support      no
TCG interpreter   no
fdt support       yes
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes
sigev_thread_id   yes
uuid support      yes
libcap-ng support yes
vhost-net support yes
vhost-scsi support yes
Trace backend     nop
Trace output file trace-<pid>
spice support     yes (0.12.6/0.12.4)
rbd support       yes
xfsctl support    yes
nss used          no
libusb            yes
usb net redir     no
GLX support       yes
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
seccomp support   no
coroutine backend ucontext
coroutine pool    yes
GlusterFS support no
virtio-blk-data-plane yes
gcov              gcov
gcov enabled      yes
TPM support       no
libssh2 support   no
TPM passthrough   no
QOM debugging     yes
vhdx              yes
Quorum            no
lzo support       no
snappy support    no

4.6.4 Launch Commands

~/lsrc/qemu/qemu.git/aarch64-softmmu/qemu-system-aarch64 -machine virt "-cpu" "cortex-a57" "-machine" "type=virt" "-nographic" "-smp" "1" "-m" "2048" "-kernel"
 "/home/alex/lsrc/qemu/linux.git/arch/arm64/boot/Image" "-append" "console=ttyAMA0" -fsdev local,id=r,path=/home/alex/lsrc/qemu/rootfs/trusty-core,security_model=none -device virtio-9p-device
,fsdev=r,mount_tag=r
mount -t 9p -o trans=virtio r /mnt
./x86_64-softmmu/qemu-system-x86_64 -display none -m 4096 -serial mon:stdio -netdev user,id=unet -device virtio-net-pci,netdev=unet -drive file=/home/alex/lsrc/qemu/images/jessie-x86_64.qcow2,id=myblock,index=0,if=none -device virtio-blk-pci,drive=myblock -append "console=ttyS0 root=/dev/vda1" -kernel /home/alex/lsrc/qemu/kernel-x86-plain.build/arch/x86/boot/bzImage

4.6.5 General approach

  • set QEMUGDB
export QEMU_GDB=9999
  • run qemu and it will freeze
  • attach gdb to QEMU
3:28 alex@zen/x86_64 [qemu.git] >ps ax | grep qemu
30445 pts/15   S+     0:00 sudo chroot --userspec 1000:1000 /home/alex/lsrc/qemu/rootfs/saucy-arm64 /bin/qemu-aarch64 /root/risu/risu /root/r
isu/simd_nwshifts.risu.bin -h 10.6.2.5
30446 pts/15   S+     0:00 /bin/qemu-aarch64 /root/risu/risu /root/risu/simd_nwshifts.risu.bin -h 10.6.2.5
30471 pts/35   R+     0:00 /usr/bin/perl /usr/bin/ack-grep qemu
13:28 alex@zen/x86_64 [qemu.git] >gdb aarch64-linux-user/qemu-aarch64 -p 30446   N:some source code here
4.6.5.1 Breaking in generated code
  • Set a breakpoint at cputbexec
  • tbptr points to the code generation buffer
  • b *0x…… in the code generation buffer

4.6.6 System emulation userspace/kernel space

4.6.6.1 Attaching GDB to guest
gdb-multiarch vmlinux -ex "target remote localhost:1234"
4.6.6.2 Other notes
  • put a break in handleexceptionreturn which is used by the kernel to change mode
  • for return to aarch32 it's especially easy

4.6.7 Coverage Report

4.6.7.1 Using lcov
lcov -c -i -d . -o baseline.info
lcov -c -d . -o stress_run.info
genhtml -b baseline.info -o aarch64-softmmu/coverage stress_run.info
4.6.7.2 Using gcovr
/home/alex/.local/bin/gcovr -r . --html --html-details -o gcovr.html

4.6.8 TODO Investigate rr for debugging QEMU

4.6.9 Peters sed trace trick

> davidgiluk: would there be any way to abuse the migration code to run two qemu's in lock step until they diverge? <davidgiluk> stsquad: that depends how close a lock step you want <davidgiluk> stsquad: taking a migration snapshot is pretty damn expensive, although it would be nice if it was cheaper <davidgiluk> stsquad: 'colo' lets you run until the two send different packet outputs and then resync <pm215> my wild-ass guess is that lock-step running would probably be better done as an offshoot of the record-n-replay support > davidgiluk: currently I'm doing -d exec,cpu -D working/D failing and then diff -y the two log files <pm215> stsquad: there are worse approaches than that > pm215: it works fine - it's just fiddly and can generate gigs of logs which you are only interested in a little bit of <pm215> I have some sed runse that convert '-d inasm,exec,cpu,int,unimp,guesterrors,nochain -singlestep' trace logs into a one-line-per-executed-PC log, which is quite handy for looking at divergences (especially divergences compared to Other Emulators) <pm215> yes, you do have to stick to analysing relatively tractable program sizes > although maybe it would make more sense just to have a live diff consumer you can pipe the logs to <davidgiluk> stsquad: How do you select when to stop and dump them? > davidgiluk: currently I re-run with gdbstub and stick breakpoints just before the divergence… <pm215> sed> also useful for removing the bits of the log that have host addresses in them > pm215: do you publish your sed tricks anywhere? <pm215> nope <pm215> sed -ne 's/…really an SG instruction at 0x\([0-9a-f]*\),.*$/\1/p;s/Trace 0: 0x[0-9a-f]* \[[0-9a-f]*\/\([0-9a-f]*\)\/0x[0-9a-f]*\].*\(/\1/p;s/^Stopped execution of TB chain before.*/Stopped/p' | sed -ne '1{x;d};/Stopped/{x;d};x;//!p;\){x;p}'

From https://pastebin.ubuntu.com/p/R5GN3k4Ct3/

#!/bin/sh -e
# Convert QEMU trace to output PC values
# Assumes run with -d in_asm,exec,cpu,int,unimp,guest_errors,nochain -singlestep
#
# We also convert
# "...really an SG instruction at 0x2c000000, executing it"
# to a PC line of 0x2c000000
# (this is part of the -d int tracing, SG is a special case)
#
# The tricky part here is that we need to suppress printing of the
# PC line if there's a following "Stopped execution of TB chain" line.
sed -ne 's/...really an SG instruction at 0x\([0-9a-f]*\),.*$/\1/p;s/^Trace 0: 0x[0-9a-f]* \[[0-9a-f]*\/\([0-9a-f]*\)\/0x[0-9a-f]*\].*$/\1/p;s/^Stopped execution of TB chain before.*/Stopped/p' |
  sed -ne '1{x;d};/Stopped/{x;d};x;//!p;${x;p}'

4.7 Testing Notes

4.7.1 Boot benchmark   benchmark

See the expect script

time ~/lsrc/lava/test-definitions.git/ubuntu/scripts/qemu-boot-speed.expect ./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic -smp 1 -m 2048 -kernel ../linux.git/arch/arm64/boot/Image --append "console=ttyAMA0"
4.7.1.1 Comparing 10bit and 12bit page tables

12 bit (boot & quit)

real user sys
2.645s 0.021s 0.015s
2.654s 0.029s 0.017s
2.595s 0.026s 0.019s
2.516s 0.018s 0.012s
2.565s 0.026s 0.017s
2.595 s 0.024 s 0.016 s

10 bit (boot & quit)

real user sys
3.329s 0.021s 0.014s
3.329s 0.018s 0.012s
3.353s 0.022s 0.017s
3.189s 0.017s 0.011s
3.299s 0.018s 0.013s
3.2998 s 0.0192 s 0.0134 s
4.7.1.2 Comparing 10bit/12bit tables with and without victim cache

Time in seconds, smaller is better Percentage is amount of time compared to run to the left

Code 10 bit 10 bit + victim 12 bit 12 bit + victim
  12.783 11.664 10.348 9.527
Runs 13.046 11.971 10.123 9.326
  12.929 11.673 11.130 9.858
  12.981 11.941 10.223 9.673
Avgs 12.93475 11.81225 10.456 9.596
%prev 100% 91.321827 88.518276 91.775057

4.7.2 GCC Tests

make -k check-c RUNTESTFLAGS="aarch64.exp"

4.7.3 emerge eix

  • unsupported syscall 267

4.8 Foundation Model

./Foundation_v8 --network=nat --network-nat-ports=10000=9191 --image=img-foundation.axf --block-device=vexpress64-openembedded_lamp-armv8-gcc-4.8_20131023-504.img

4.9 RISU Notes

4.9.1 Generating instructions

./risugen --numinsns 100000 --no-fp --pattern "ADD.* A64" --pattern "SUB.* A64" aarch64.risu add_addi.risu.bin

4.9.2 Dumping risu binary files

aarch64-linux-gnu-objdump -b binary -m aarch64 -D rev.risu.bin > rev.risu.asm
  • "make blah.risu.asm"

4.10 Development Process

qemu-dev-process.png

4.11 UEFI Firmware

The latest builds can be found on snapshots.

5 Analysis

5.1 Total Instructions

Count and summarise the total number of instructions in a binary.

objdump --no-show-raw-insn -d $binary | egrep '^\s+[0-9a-f]+:' | sed 's/ \+ /\t/g' | cut -f 3 | wc -l
40762

objdump --no-show-raw-insn -d $binary | egrep '^\s+[0-9a-f]+:' | sed 's/ \+ /\t/g' | cut -f 3 | sort | uniq -c | sort -rn

5.2 Travis Failures

On –disable-user

ERROR:tests/ipmi-bt-test.c:319:test_connect: assertion failed: (rv == 1)
GTester: last random seed: R02Se1c1331ba30cc477576f9690cb31b287
 **
ERROR:tests/ipmi-bt-test.c:141:read_emu_data: assertion failed: (rv > 0)
GTester: last random seed: R02S5e7bc3627a7fcb661a87a2628990c638
 **
ERROR:tests/ipmi-bt-test.c:141:read_emu_data: assertion failed: (rv > 0)
GTester: last random seed: R02Sfdfa543f178fab6d61d5ac3873407a0e
 **
ERROR:tests/ipmi-bt-test.c:141:read_emu_data: assertion failed: (rv > 0)

The ever faithful time is wrong:

ERROR:tests/tpm-emu.c:27:tpm_emu_test_wait_cond: code should not be reached

Maybe related - with clang build (MacOS clang)

ERROR:tests/test-aio.c:501:test_timer_schedule: assertion failed: (aio_poll(ctx, true))

Replaced the assert

ahci-test: tests/libqtest.c:113: kill_qemu: Assertion `!(((__extension__ (((union { __typeof(wstatus) __in; int __i; }) { .__in = (wstatus) }).__i))) & 0x80)' failed.

Needs a better report, Added a debug runner, possible compiler bug?

ERROR:tests/rcutorture.c:384:gtest_stress: assertion failed (n_mberror == 0): (1 == 0)

6 Actions

6.1 Submit QEMU Patch Series

6.1.1 Cover Letter

Hi,

This is a small re-factoring series which I'll be needing for adding guest architecture awareness to plugins. There is a little clean up of concerns by removing the "template" type behaviour from elf.h into a new elf-types.inc.h file. I then rationalise the ELF related headers to all be in the same place. Finally the actual useful piece of moving the definition of ELFARCH out of the two loader files and into an stand alone header.

6.1.2 Check it compiles and builds

set -e
git describe
echo "Configuring build: `date`"
cd builds/all
../../configure --python=python3 > /dev/null
make clean > /dev/null
echo "Configure and clean complete: `date`"
set -e
git describe
cd builds/all
if test -z $prefix; then
    echo "Starting build: `date`"
    make -j9 > /dev/null 2>&1
    echo "Build complete: `date`"
 else
    # Done manually?
    echo "Skipping build (done manually?): `date`"
    echo "Dumping last linked binaries:"
fi
find . \( -regex ".*linux-user/qemu.*$" -or -regex ".*softmmu/qemu-system.*$" \) -and -printf '%T+ %p\n' | sort | tail -n 5
set -e
git describe
cd builds/all
if test -z $prefix; then
    echo "Starting check: `date`"
    make check | tail -n 5
    echo "Check complete: `date`"
    make check-tcg | tail -n 5
    echo "TCG Check complete: `date`"
else
    # Done manually?
    echo "Skipping check: `date`"
fi

6.1.3 Export, run checkpatch and prepare the cover letter

./scripts/checkpatch.pl -q ${series}.patches/* > ${series}.checkpatch
lines=$(cat ${series}.checkpatch | wc -l)
if test "$lines" -gt 150 ; then
  echo "Results too long ($lines), see ${series}.checkpatch"
else
  cat ${series}.checkpatch
  rm -f ${series}.checkpatch
fi
echo "Finished running @ `date`"
(save-excursion
  (goto-char (point-min))
  (when (re-search-forward "^*** Cover Letter")
    (goto-char (match-beginning 0))
    (next-line)
    (let ((beg (point)))
      (outline-next-heading)
      (previous-line)
      (concat
       (buffer-substring-no-properties beg (point))))))

6.1.4 Final send

git send-email --confirm=never --dry-run --quiet ${mailto} ${series}.patches/*
set -e
git send-email --confirm=never --quiet ${mailto} ${series}.patches/*
rm -rf ${series}.patches

6.2 Submit a QEMU PULL request

6.2.1 Check commits are good

We need to ensure we have added our signoff and there is no — ephemera left from commit history.

errors=0
commits=0
while read rev; do
    author=$(git show -s --format='%an <%ae>' $rev)
    body=$(git show -s --format='%B' $rev)

    # Check for Author signoff
    if ! grep -q "^Signed-off-by: $author" <<< "$body"; then
        errors=$((errors+1))
        echo $(git log -1 --pretty=format:"missing author signoff - %h - %an: %s" $rev)
    fi

    # Check for my signoff
    if ! grep -q "^Signed-off-by: $signoff" <<< "$body"; then
        errors=$((errors+1))
        echo $(git log -1 --pretty=format:"missing my signoff - %h - %an: %s" $rev)
    fi

    # check for unreviewed patches for patches I authored
    if [ "$author" = "$signoff" ]; then
        if ! grep -q "^Reviewed-by:" <<< "$body"; then
            echo $(git log -1 --pretty=format:"unreviewed - %h - %an: %s" $rev)
        fi
    fi

    # Check for stray history
    if grep -q "^--" <<< "$body"; then
        errors=$((errors+1))
        echo $(git log -1 --pretty=format:"has commit history - %h - %an: %s" $rev)
    fi

    commits=$((commits+1))
done < <(git rev-list "origin/master..HEAD")

echo "Found $errors errors over $commits commits"

If we want to strip history we can run the following:

current=$(git rev-parse --abbrev-ref HEAD)
if [ -d "${series}.pull" ]; then
  rm -rf ${series}.pull
fi
git format-patch origin/master..${current} -p -o ${series}.pull
# need to delete old fixup branches
prb=${current}-pr
echo ${prb}
git branch -f -D ${prb}
git checkout -b ${prb} origin/master
git am ${series}.pull/*
rm -rf ${series}.pull

6.2.2 Preparing a QEMU Pull Request

(let ((tag (format
            "pull-%s-%s-%d"
            series
            (format-time-string "%d%m%y")
            version)))
  (magit-tag-create tag "HEAD" "--sign")
  tag)
set -e
tag=$(git describe)
git push github $tag
git push gitlab $tag
echo "pushed $tag"
(if (= 1 version)
    "PULL"
  (format "PULL v%d" version))
if [ -d "${series}.pull" ]; then
  rm -rf ${series}.pull
fi
git format-patch --subject-prefix="$subjprefix" --cover-letter origin/master..HEAD -p -o ${series}.pull

You can use the $pull macro to fill in the details

6.2.3 And send the pull request

Using the prefix will limit the send to just the cover letter, useful for v2+ pull requests

if test -z "$prefix" ; then
  git send-email --confirm=never --dry-run --quiet ${mailto} ${series}.pull/*
else
  git send-email --confirm=never --dry-run --quiet ${mailto} ${series}.pull/0000*
fi
if test -z "$prefix" ; then
  git send-email --confirm=never --quiet ${mailto} ${series}.pull/*
else
  git send-email --confirm=never --quiet ${mailto} ${series}.pull/0000*
fi
rm -rf ${series}.pull

6.3 Submit RISU Patch Series

set -e
git describe --all
echo "Running build: `date`"
./build-all-archs --static > /dev/null
./build-all-archs --use-docker > /dev/null
echo "Complete: `date`"

6.3.1 Export, run checkpatch and prepare the cover letter

6.3.2 Final send

git send-email --confirm=never --dry-run --quiet ${mailto} ${series}.patches/*
set -e
git send-email --confirm=never --quiet ${mailto} ${series}.patches/*
rm -rf ${series}.patches

Author: Alex Bennée

Created: 2019-09-11 Wed 17:29

Validate