Virtualization

A guest is three capabilities and some tasks#

Kore's hypervisor has two halves. In the kernel, the world switch runs each virtual CPU as an ordinary scheduled task. In user space, a virtual machine monitor decides what the guest's devices are and services everything the guest does that the hardware can't. The contract between them is a handful of capabilities and one page of rings per vCPU.

A vCPU is a Task founded into a guest space. Its handle carries all four rights, and like a founded process it is dormant until started.

Who can build a guest#

The Hypervisor capability reaches a task only by founding, under role::HYPERVISOR. In Cosmos, the hypervisor service is the one task founded with it, together with a bounded memory supply (HYPERVISOR_SUPPLY_BYTES, 8 MiB), so a guest that doesn't fit fails its allocation cleanly instead of drawing on kernel memory. Programs use guests through that service rather than holding the facility themselves.

The verbs below are therefore what a monitor uses. They are public, because they are part of the contract, and they refuse without the capability.

Building one: the monitor's side#

kore-rt carries the verbs, reachable from libkore as kore::runtime::cap::verbs:

A region's access must include read, and may not include both write and execute: second-stage mappings are W^X like everything else. Its length must equal the lease's contiguous span. On a refusal before commit, found_guest_vcpu hands back every input it was given, so no half of the vCPU's plumbing is lost.

use kore::abi::guest::GuestVcpuConfig;
use kore::abi::session::{SHARE_ACCESS_R, SHARE_ACCESS_W, SHARE_ACCESS_X};
use kore::runtime::cap::verbs::{self, GuestVcpuInputs};
use kore::runtime::cap::{
    GuestRegionToken, GuestSpaceToken, HypervisorToken, StashedLease, TaskStartOutcome,
    TaskToken,
};

/// RAM the monitor already carved from its supply for one guest.
pub struct GuestRam<'a> {
    pub code: &'a StashedLease,
    pub code_bytes: u64,
    pub data: &'a StashedLease,
    pub data_gpa: u64,
    pub data_bytes: u64,
}

/// One running guest, and the handles that own it.
pub struct Guest {
    pub space: GuestSpaceToken,
    pub code: GuestRegionToken,
    pub data: GuestRegionToken,
    pub vcpu: TaskToken,
}

pub fn build_guest(
    hypervisor: &HypervisorToken,
    ram: GuestRam<'_>,
    first_entry: GuestVcpuConfig,
    window: GuestVcpuInputs,
) -> Option<Guest> {
    // One guest-physical address space, seen identically by the guest's
    // CPUs and by DMA.
    let space = verbs::create_guest_space(hypervisor).ok()?;

    // Stage-2 W^X: code is readable and executable, data readable and
    // writable. A region asking for both W and X is refused.
    let code = verbs::share_stashed_into_guest(
        ram.code,
        &space,
        0,
        ram.code_bytes,
        SHARE_ACCESS_R | SHARE_ACCESS_X,
    )
    .ok()?;
    let data = verbs::share_stashed_into_guest(
        ram.data,
        &space,
        ram.data_gpa,
        ram.data_bytes,
        SHARE_ACCESS_R | SHARE_ACCESS_W,
    )
    .ok()?;

    // The vCPU is an ordinary task, founded dormant: the window and both
    // doorbell endpoints move into it only when the founding commits.
    let vcpu = verbs::found_guest_vcpu(&space, first_entry, window).ok()?;
    match vcpu.start() {
        TaskStartOutcome::Active => Some(Guest { space, code, data, vcpu }),
        _ => None,
    }
}

GuestVcpuConfig is the architecture-neutral first entry: entry, stack and page_table_root, the guest-physical root of the guest's own page tables. GuestVcpuInputs carries the three things that move into the vCPU: the lease holding its formatted window page, the channel end it waits on to resume (resume_recv), and the one it kicks the monitor's doorbell with (exit_send).

On the wire these are session derives: GuestShareReq (kind = GuestRegion) for a region, and DERIVE_KIND_GUEST_FOUND with GuestFoundReq for a vCPU. See sessions.

The vCPU window#

Each vCPU has one 4096-byte page holding two single-producer rings of 512-byte records:

Records are bytes and indices only: no capability ever rides a guest ring, and every field decodes from arbitrary bytes, because the peer is untrusted. Each record carries the guest's register image as 40 quadwords in the executing architecture's own wire order. The protocol names registers only by semantic role (state_index::IO_DATUM, state_index::IP), so a consumer never carries a per-architecture register map.

A second-stage fault names an address, not an instruction. To complete an MMIO access without a second exit, the record's decode-assist tail carries whatever the hardware knew: the raw instruction bytes (assist::INSN, up to 15), an already-decoded access (assist::ACCESS, as Arm and RISC-V report it), or an honest assist::NONE.

Doorbell notices (NOTICE_COOKIE_GUEST_EXIT, NOTICE_COOKIE_GUEST_CONTROL) only wake the other side. The rings are the source of truth, and a woken consumer drains its ring completely.

Using one: the client's side#

Programs talk to the hypervisor service over the ordinary service protocol: HypervisorReqWire and HypervisorRepWire in kore::abi::svc. A guest's console doesn't ride the verbs. It has its own planes in the pair's window: an event ring of output from the guest, and an input ring from the client.

Today the protocol's verbs start the service's built-in guests (an echo console, a halting guest, compiled C, a multi-segment image, MicroPython, an MMIO-console guest and a multi-CPU guest), attach and detach the console, report status and stop the guest, with one live guest at a time. They exist to prove the path end to end, and Cosmos's shell uses them to run MicroPython.