Building a Kernel-Mode Hypervisor

Why build a hypervisor?

This project was big during a time in which I was invested heavily in the Cheat development scene, and developed this along with WdFilter to create a robust and stealthy hypervisor-based cheat protection system.

Architecture overview

ArkVisor is structured as a Windows kernel-mode driver that initializes VMX operation on each logical processor at load time. The high-level components are:

VMX initialization — VMXON, VMCS setup, and host/guest state configuration
VM exit handler — routing and dispatching VM exits to the appropriate handler
EPT — extended page tables for memory virtualization and monitoring
Hypercall interface — a VMCALL-based channel between guest and hypervisor
Stealth layer — techniques to make the hypervisor invisible to detection

VMX initialization

The VMXON sequence is deceptively simple on paper but requires precise setup of the VMCS (VM Control Structure). The most critical fields are HOST_CR3 and GUEST_CR3 — getting these wrong causes immediate triple-faults when entering the guest. I learned that host state must be explicitly saved before VMLAUNCH because the CPU doesn't preserve it automatically. The surprise was how much of the VMCS is "don't care" for basic operation but becomes essential once you handle exits.

; Example: VMXON sequence (illustrative, not actual code)
; Check CPUID for VMX support
; Enable VMX in CR4
; Allocate and initialize VMXON region
; Execute VMXON
; Allocate and configure VMCS
; Execute VMLAUNCH

VM exit handling

Every VM exit passes through a central handler that reads the exit reason from the VMCS and dispatches to the right function. The exits I had to handle included:

CPUID — to mask hypervisor presence from guest queries
VMCALL — for the hypercall interface
CR access — to monitor and optionally intercept control register writes
EPT violations — triggered when the guest accesses a monitored page
RDTSC/RDTSCP — to filter timing information and hide VM exit latency
INVEPT/INVVPID — cache invalidation requests from the guest

The exit handler's latency is critical — every VM exit is a context switch that costs hundreds of cycles. The edge case that broke me was nested exits: handling a CPUID exit that triggered another exit during the handler. I had to carefully manage the guest RIP advancement to avoid infinite loops. RDTSCP was particularly tricky because some anti-cheat tools use it for timing checks, requiring precise TSC offset manipulation.

Extended Page Tables (EPT)

EPT is the part of the hypervisor I spent the most time on. I use a 4-level page walk structure mirroring the guest's paging setup. EPT serves two purposes: hiding the hypervisor's own memory from the guest (shadow space), and monitoring specific guest pages by setting them execute-only or read-only to trigger violations on access.

EPT violations are expensive if you get them wrong. My first implementation triggered violations on nearly every memory access because I had forgotten to set the read and write permissions on the EPT entries for regular guest memory. The fix was ensuring all guest-legal memory has appropriate R/W/X bits set, while only the pages I want to monitor have restricted permissions.

Hypercall interface

The hypercall interface uses VMCALL with a custom ABI: RCX holds the command ID, RDX-R8 contain arguments, and RAX returns status. Commands include reading/writing arbitrary physical memory, installing hooks, and querying hypervisor status. The dispatcher validates command IDs and argument bounds before executing. Guest-to-hypervisor communication is essential for the cheat framework to query game state without being detected by user-mode anti-cheat scans.

Stealth and anti-detection

Making the hypervisor invisible involves intercepting several detection vectors. The main ones I addressed:

CPUID leaf 0x1 — masking the hypervisor present bit
CPUID leaf 0x40000000 — returning a spoofed or absent hypervisor signature
Timing attacks — RDTSC/RDTSCP interception to reduce VM exit overhead visibility
System register hiding — intercepting MSR reads that could reveal VMX state
Exception injection — simulating #GP for suspicious instructions

Working on stealth taught me how modern anti-cheat builds a fingerprint of the system. They look for timing anomalies, unexpected CPU features, and memory artifacts. The arms race is continuous: every detection method I bypass could be patched tomorrow. Understanding VMX from the inside revealed just how much trust we place in the CPU's virtualization extensions.

What I got wrong

My first attempt at VM exit handling crashed the system within seconds. I was incorrectly advancing the guest RIP, causing instruction replay loops that the watchdog timer caught. Another painful bug: forgetting to invalidate EPT caches after modifying entries, leading to stale translations and memory corruption. The worst was a race condition during multi-processor startup where cores tried to VMXON simultaneously without proper serialization. I learned to respect the SDM's every word and test on a VM before bare metal.

What I'd do differently

If I rebuilt this today, I'd start with a more modular exit handler design using function pointers rather than a giant switch statement. I'd also implement proper logging infrastructure from day one — debugging a hypervisor with DbgPrint is painful. For EPT, I'd use a bitmap-based permission system instead of page-by-page flags to reduce memory overhead. Finally, I'd spend more time on the build system to make testing in nested virtualization (VMware/VirtualBox) easier before moving to real hardware.

Resources that helped

A few things I found genuinely useful while building this:

Intel SDM Volume 3C — Chapters 23–27 (VMX)
Drew's Infosec blog posts on hypervisor development
HyperDbg repository — for VM exit handler patterns
SimpleVisor by Alex Ionescu — minimal working reference