BHyVe internals
@syuu1228

BHyVe

BHyVe

FreeBSDLinux KVM

Intel VT

WebGSoC

web site
http://www.bhyve.org/
GSoC 2012: BHyVe BIOS emulation to boot legacy systems
http://bit.ly/bhyve_bios

Intel VT-x, EPT (= Nehalem)

BIOS (disk)

:virtio-net, virtio-blk

pci passthrough(VT-d)

paravirtual console/debug port

UART

OS:FreeBSD 8, 9, 10

/usr/sbin/bhyvevmm.koBSD kernel

IOCTL(VM_RUN)

Guest kernelUser program

VMLAUNCH

VMExit

/boot/loader.confhw.physmem="0x100000000"

kldload vmm.ko/usr/sbin/bhyveload -m ${lowmem} -M {highmem} -h {bootdir} ${vmname}/usr/sbin/bhyve -c ${cpus} -m ${lowmem} -M{highmem} \
-s 1,virtio-net,tap0 -s 2,virtio-blk,${diskdev}

/usr/sbin/bhyveload
VMBSDVM

/usr/sbin/bhyve
bhyveloadVMNIC
VM/dev/vmm/${vmname}
read(), write(), mmap()VM

bhyveload

sysctl(hw.vmm.create, vm_name) /dev/vmm/${vm_name}

open(/dev/vmm/${vm_name})

seg.gpa = 0
seg.len = mem_size
ioctl(fd, VM_MAP_MEMORY, seg)
membase = mmap(NULL, mem_size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0)

userboot.somembaseBSDmemcpy()

userboot.so

FreeBSD

wrap
mmapioctlVMM

kload
Linuxkexec

bhyve

open(/dev/vmm/${vm_name})

pthread_create(fbsdrun_start_thread)
fbsdrun_start_thread() {
while(1) {
ioctl(VM_RUN, &vmexit)
handler[vmexit.exitcode](&vmexit, &vcpu);
}
}

kevent()

http://www.youtube.com/watch?v=N2TbKzE_puA

CPU
OS

vCPU

bhyveloadCR0 = PE | PG | NE #

CR4 = PAE | VMXE # PAEVMX

EFER = LME | LMA # long mode

GDT

CR3

RSP

mptable

/usr/sbin/bhyveCPU

ACPICPUOS

BIOS ROM

PCI

bhyveload

64bit

IO

devicepcidevicebvmconsoledevicebvmdebugdevicemptable
ACPI
virtio.ko, if_vtnet.ko, virtio_pci.ko, virtio_blk.ko

/usr/sbin/bhyvevmm.koBSD kernel

IOCTL return

Guest kernel

VMExit

IO

IOconsolePCInetblk

io emulation

IO

OSin/outVMExit

vmm.koEXIT_REASON_INOUT

VM_EXITCODE_INOUTioctlreturn/usr/sbin/bhyve

handler[VM_EXITCODE_INOUT]()

inout_handlers[port].handler()
IO

bvm_console

IO0x220inl / outl

int getc(void) { return inl(0x220); }

void putc(int c) { outl(0x220, c); }

VMMif (in) {
read(fd, &c, 1);
*eax = (c & 0xff);}else
write(fd, *eax, 1);

PCI

CONFIG_ADDRESS0xcf8

CONFIG_DATA0xcfc 0xcff

PCI

PCIIO

PCIMSI

PCIMSI

configuration spaceMSI capable

/usr/sbin/bhyveVM_LAPIC_IRQ ioctlvmm.ko
cpuidvector

vmm.koioctlvlapic

vmlaunchvlapicVMCS

MSR register

Local APICwrmsr/rdmsrVMExit

vmm.ko/usr/sbin/bhyve

Local APIC Timer
vmlaunchvlapic
vmm.ko

TSC
rdtsctrap

PIT 8254
/usr/sbin/bhybeIO

virtio-net, blk

PCIIOMSI

/usr/sbin/bhyve

MMIO
MMIO

pci passthrough(VT-d)

Intel VT-dPCI

vmm.koio/usr/sbin/bhyve

MMIOEPT

UART

bvm_console

PCCOMPCI

Siig CyberSerial 1-port

IOAPIC(^o^)

OS

FreeBSDFreeBSDbhyveloadforkOS

BHyVe Hackathon

http://bit.ly/bhyve_hackathon
@syuu1228