# Where 23 seconds went: starting a GitHub Actions job VM in 7

> A fresh KVM virtual machine for one GitHub Actions job took 22.9 s to reach its runner on Skipbox's hosts. It now takes 7.25 s. Each second and how it was measured, from the runner at a flat $39 a month with unlimited minutes.

A fresh virtual machine for one GitHub Actions job took 22.9 seconds on a Skipbox host to get from "the host accepted the job" to "the runner process starts". It now takes 7.25 seconds. This post is where the other 15 seconds were, how each one was measured, and what we tried that did nothing.

Skipbox is a GitHub Actions runner at a flat $39 a month with unlimited minutes. Your jobs run on your account's own machine, which is kept between jobs, and the jobs that need a fresh VM get one in 7 seconds. Change one `runs-on` line to move.

- A fresh job VM reaches its runner in 7.25 s, down from 22.9 s
- On your own warm machine, a job's setup steps take 7 s against 52 s on GitHub's `ubuntu-latest`
- 20,000 minutes a month on 4 vCPU: $222 on GitHub, $39 on Skipbox

**GitHub Actions at a flat price, unlimited minutes** Skipbox runs your GitHub Actions jobs on your own dedicated machine: $39 a month, however many minutes you run. Docker layers and dependency caches stay between jobs, so the next job starts right away.

A 4 vCPU runner, 20,000 minutes a month: GitHub $222 → Skipbox $39. You save $183 every month.

Moving takes one line: `runs-on: skipbox`.

[Start my 7-day free trial](https://app.skipbox.dev/sign-in?ref=blog-github-actions-job-vm-boot-7-seconds-en)

## What is timed, and on what

A Skipbox host runs KVM virtual machines through Incus on a ZFS pool. The runner image is a stopped VM with a snapshot. A job that needs a fresh VM gets a ZFS clone of that snapshot, boots it, receives a just-in-time runner config through the guest agent, runs one job, and is deleted with its disk.

The host daemon logs the end of every step of a start in milliseconds from the moment it accepts the job (`startMs`: `cloned`, `started`, `agent`). The numbers below are from a 16-thread Intel i7-7820X host and a VM of 4 vCPU and 8 GB.

| | Acceptance to the runner's start, 3 runs each |
|---|---|
| Before | 22.9 s (21.7 to 24.2) |
| After | 7.25 s (7.1 to 8.1) |

Where it went, each piece measured on its own:

| Change | Measured | Saved |
|---|---|---|
| SeaBIOS instead of OVMF | VM start to the agent's first answer, median of 3: 13.7 → 8.4 s | 5.3 s |
| `quiet` on the kernel command line | 8.4 → 6.5 s | 1.9 s |
| No initrd | 6.5 → 5.1 s | 1.4 s |
| An image already the size of its clones | Clone 1.5 → 0.1 s, first start 2.35 → 1.8 s | 1.9 s |
| Wait for Docker's socket, not dockerd | dockerd was up 3.6 s after the agent | 3.6 s |
| Ask the guest agent every 100 ms, not every second | 0.5 s on average, up to 1 s | 0.5 s |

The pieces add up to 14.6 s of the 15.65 s end-to-end difference.

## Firmware: 5.3 seconds before GRUB

Incus boots a VM with OVMF and secure boot by default. On this host OVMF took 5.3 s to hand over to GRUB. SeaBIOS takes 0.5 s. Two instance keys switch it: `security.csm=true` and `security.secureboot=false`.

This was the largest single piece, and the only one that touches security, so here it is in full. With SeaBIOS the guest's boot chain is not signature-checked. What that check protects against is someone replacing the guest's bootloader or kernel. In a CI job VM that someone is the job, and the job already has root in its own VM: it can `sudo` anything, and the VM and its disk are deleted when the job ends. What keeps one customer's VM away from another's is KVM and a per-VM firewall chain on the host's bridge, and neither depends on the guest's firmware. The image boots under both, and one flag on the host puts UEFI back.

## A quiet kernel, and no initrd

A job VM boots once for every job, and nobody reads its serial console. `quiet` took the median from 8.4 to 6.5 s.

The initrd exists to load the drivers that find the root file system. Ubuntu's generic kernel has virtio-scsi and ext4 built in, and every clone shares the image's partition UUID, so the kernel mounts `/` directly from `root=PARTUUID=…`. That was another 1.4 s.

Two guards keep it from going wrong quietly. Ubuntu's GRUB boots the next time with the initrd by itself if a boot without one panicked (`GRUB_FORCE_PARTUUID`). That fallback would silently bring the 1.4 s back to every job, so the image build boots a test clone and fails if `dmesg` shows an initramfs being unpacked.

## Don't resize a disk you could have built at the right size

The image was built on a smaller disk, and every clone was grown to its plan's size at start. Growing cost 1.9 s per job: the clone took 1.5 s instead of 0.1 s, and the first start 2.35 s instead of 1.8 s. The image is now built at the smallest plan's disk size (80 GB). The volume is thin, so the image still holds only what was written to it. Incus can't shrink a clone, so the image is never larger than the smallest plan; larger plans grow at boot.

## Docker's socket is enough

The run script used to wait for `docker.service` before starting the runner. dockerd was up 3.6 s after the guest agent. But systemd listens on Docker's socket from early boot, and a client that connects before dockerd is up waits for it. So the script waits only for `docker.socket` and DNS, with a bounded wait, and dockerd finishes starting while the runner does.

## Polling at 100 ms

The runner's config is pushed into the VM through the guest agent, which isn't there until the guest has booted. The host retried once a second, so a job waited 0.5 s on average after the agent came up. Incus refuses a push in 20 ms while the agent is down, so asking every 100 ms costs nothing. After a VM's first 30 s the host goes back to once a second.

## Not a boot fix: fsync on a disk that dies with the job

While measuring, the slowest thing in the VM turned out not to be the boot. Every `fsync` in the guest waited for the host's ZFS to commit its intent log. A fresh job VM's disk is deleted with the job, so nothing on it has to survive a power cut. The run script remounts `/` with `nobarrier` before it starts the runner.

Medians of 3, in a VM of 4 vCPU and 8 GB:

| | Before | After |
|---|---|---|
| 4k random writes with fsync | 227 a second | 2,505 a second |
| `yarn install`, 1 GB of node_modules from its cache | 12.92 s | 9.99 s (23% less) |
| `docker pull node:22` | 19.12 s | 15.45 s (19% less) |
| A Docker build of redis | 92.89 s | 87.21 s |
| `make -j4`, `sysbench cpu` | unchanged | unchanged |

Only fresh VMs get this. Your own machine holds your caches between jobs, and its ext4 keeps its barriers.

What we measured and did not take:

- **ZFS `sync=disabled` on the VM's volume** measured the same (2,901 writes a second; 10.10 s; 16.17 s; 87.99 s). It needs a root `zfs` call that the host daemon's sandbox has no right to make. The guest-side remount needs nothing from the host.
- **Guest memory on huge pages:** `make` took 52.41 s against 51.72 s without. Nothing.
- **`noatime` and no `discard` on `/`:** `yarn install` 12.84 s against 12.92 s. Nothing.

## The same hunt on rented cloud servers

When Skipbox's own hosts are full, a job runs on a cloud server booted from a snapshot image. That path had its own lost time, in a different place. On a 4 vCPU Intel shared server, the job script (the server's user data) started 28.1 s after the kernel. cloud-init had the script at 19.3 s and Docker was up at 22.2 s, but cloud-init runs user data in its last stage, after everything else.

The image now has a small systemd unit that runs the user data as soon as cloud-init has written it. It reads the script through a file descriptor from a file it has already removed, so cloud-init's last stage finds nothing to run a second time. The image also makes one kind of SSH host key instead of three (2.0 s on the path everything waits for; nobody signs in to these servers), and no longer starts snapd, apport, rsyslog or a dozen maintenance timers at boot.

With that image, on a 4 vCPU AMD shared server, the job script starts 8 to 14 s after the kernel.

## The dead end that stopped a clock

One host is a desktop PC with a hybrid CPU (Core Ultra 7 265F: performance cores and efficiency cores). During a night of image builds, the kernel logged:

```
tsc: Marking TSC unstable due to KVM discovered backwards TSC
clocksource: Switched to clocksource acpi_pm
```

KVM compares the TSC when a vCPU moves to another host CPU. It read a lower value on the new core, across core types, and marked the TSC unstable. The kernel had already disabled HPET at boot ("HPET dysfunctional in PC10"), so the only clock source left was acpi_pm, which on this board barely advanced: about 1 second of uptime in 30 minutes. `sleep` never returned. `sync` hung. New SSH logins were refused. Ping still answered. Only someone at the machine could reboot it.

What did not fix it: `tsc=reliable` only turns off the clocksource watchdog; KVM's check has no switch.

What did, in two steps:

1. **A fallback that works.** Limiting the package C-state to C8 in the BIOS keeps HPET alive. KVM still dropped the TSC, in round 11 of a 20-round loop that starts and stops an 8-vCPU VM under load, but the clock moved to HPET and kept time.
2. **Pinning.** KVM runs that check only when a vCPU moves. With each vCPU pinned to one host CPU, the same loop ran 40 rounds on the TSC with no new line in the log.

The host daemon takes no job unless the clock source is `tsc` and `hpet` is available, so a host that drops its TSC drains itself and its jobs go to other machines. Every Skipbox host pins its vCPUs.

## The fastest boot is the one you skip

Seven seconds is for the jobs that need a fresh VM: code from people without write access (such as pull requests from forks), a job that arrives while your machine is busy, and accounts that choose a clean VM for every job. Every other job boots nothing. It runs on your account's own machine, with the last job's Docker layers, package caches and tools still on its disk.

That is where the minutes are. The same workflows on GitHub's `ubuntu-latest` (2 vCPU, what a private repository gets) and on a warm Skipbox S (4 vCPU), medians of warm runs:

| | GitHub `ubuntu-latest` | Skipbox S, warm | |
|---|---|---|---|
| excalidraw: install, build and tests | 613 s | 450 s | 27% less |
| Setup steps (checkout, Node, cache restore) | 52 s | 7 s | |
| Tests | 484 s | 354 s | |
| umami: multi-stage Docker build | 263 s | 188 s | 29% less |

The setup steps fall from 52 s to 7 s because there is no cache archive to download and unpack: Node is already in the tool cache and yarn's cache is on the disk. The Docker build uses the daemon's own layer cache, with nothing exported or imported. On Skipbox the workflow has no cache step at all.

## Why a flat price can keep a machine warm

An idle machine looks like waste. With the way compute is billed, it is the cheaper choice, and that is what makes a flat price work.

A rented cloud server is billed for every hour it starts, in full. A 3-minute job on its own server pays for 60 minutes. On one day of our own CI, 44 jobs averaging 2.5 minutes ran on 44 servers and were billed 44 server-hours. Replaying the same day with each server kept for the account's next job within its hour gives 13 to 14 server-hours: about 70% less. And a month has 730 hours: an account that runs more than 730 short jobs a month on per-job servers pays for more hours than a machine that never stops.

So the machine that is fastest for you (caches on disk, nothing to boot) is also the one that costs the least to run. Hosts Skipbox owns come first because they have no hourly bill at all, and rented servers take the overflow. That is why Skipbox charges for the machine, not the minute:

| Minutes a month (4 vCPU) | GitHub | Skipbox S | You save every month |
|---|---|---|---|
| 10,000 | $102 | $39 | **$63** |
| 20,000 | $222 | $39 | **$183** |
| 30,000 | $342 | $39 | **$303** |
| Always on | $500.40 | $39 | **$461.40** |

At full use that is 1/12 of GitHub's price. GitHub's figures are the bill after GitHub Team's included compute. Put your own minutes into the [calculator on the home page](https://skipbox.dev/?ref=blog-github-actions-job-vm-boot-7-seconds-en#cost); it needs no sign-in.

## Moving is one line

```yaml
jobs:
  build:
    runs-on: skipbox # was ubuntu-latest
```

1. Sign in to Skipbox with GitHub.
2. Install the Skipbox GitHub App and pick the repositories.
3. Change `runs-on` and push.

## Try it

The trial takes no card: 7 days or 3,000 job-minutes, whichever comes first. Point your slowest workflow at `runs-on: skipbox`, run it twice, and compare the step times of the second run.

## FAQ

### How long does a fresh VM for a GitHub Actions job take to start on Skipbox?

On Skipbox's own hosts, 7.25 seconds from the host accepting the job to the runner process starting, down from 22.9 seconds. Most jobs don't boot anything: they run on your account's own machine, which is kept between jobs.

### Is it safe to boot CI job VMs with SeaBIOS instead of UEFI secure boot?

Yes for a job VM. Secure boot protects the guest's boot chain from being replaced, and a CI job already has root in its own VM, which is deleted after the job. On Skipbox, VMs are kept apart by KVM and a per-VM firewall on the host, and neither depends on the guest's firmware.

### Why is fsync slow in a VM on ZFS, and how do you fix it for CI?

Every fsync in the guest waits for the host's ZFS to commit its intent log. A fresh job VM's disk is deleted with the job, so Skipbox mounts its ext4 without barriers: yarn install is 23% faster and docker pull 19% faster. A machine that is kept between jobs keeps its barriers.

### Why does KVM mark the TSC unstable on a hybrid CPU?

KVM compares the TSC when a vCPU moves to another host CPU, and across performance and efficiency cores it can read a lower value. Pinning each vCPU to one host CPU stops the check from running: Skipbox's start-and-stop loop then held the TSC for 40 rounds.

### How do I run my jobs on Skipbox?

Change runs-on to skipbox in a workflow. The trial takes no card: 7 days or 3,000 job-minutes, whichever comes first.

Updated 2026-11-12 · https://skipbox.dev/blog/github-actions-job-vm-boot-7-seconds