iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
When a CircleCI self-hosted machine job stays queued or a Machine Runner Orchestrator fails to create VMs, start with the job’s resource_class. Confirm that the class name is exact, a runner is attached, the class token is valid, and queued work has not consumed the class’s concurrency. Then separate controller/API problems from VM startup and runner-agent problems by reading the provisioner and guest logs. This sequence covers CircleCI Cloud, CircleCI Server, manually installed Machine Runner 3 agents, and the Machine Runner Orchestrator documented as version 1.0.0.
Use this order to isolate the failure
Do not infer capacity from an apparently empty runner list. CircleCI’s inventory shows tasks that are actively running; queued tasks also count toward a resource class’s concurrency. A class can therefore look idle while new jobs remain blocked. CircleCI’s troubleshooting guide states that concurrency is calculated across all tasks for a resource class, including tasks waiting in the queue. Start with the checks below, in order.
| Observed symptom | First check | What it distinguishes |
|---|---|---|
| Job remains queued immediately | Exact resource_class and attached runner |
Class-selection or registration failure |
| Inventory looks idle but jobs do not start | Runner API task query and concurrency | Hidden queued or unclaimed work |
| Orchestrator does not create VMs | Provisioner logs, class token, API address, and replica limits | Polling/authentication versus scaling configuration |
| VM exists but never accepts a job | circleci-runner service status and journal |
Guest boot or runner registration |
| macOS job is stuck at “Preparing Environment” | Launch-agent executable permission and invocation | Documented first-install macOS issue |
Keep the deployment type in view: Machine Runner Orchestrator is a Kubernetes controller that provisions KubeVirt VMs, while a manual Machine Runner 3 installation depends on the service running on machines you manage.
1. Confirm the job targets the intended resource class
Match the namespace and name exactly
A self-hosted job is selected by its resource class. The spelling and case in .circleci/config.yml must match the configured class, including its namespace. For the orchestrator, every provisioner.resourceClasses key uses namespace/name format. A typo, wrong namespace, or reference to a class from another organization leaves the job waiting even when other runners are healthy.
#1 Best Overall
jobs:
build:
machine: true
resource_class: my-namespace/linux-large
steps:
- checkout
- run: ./build.shThe quickstart pattern uses machine: true and resource_class: <namespace>/<name> for an orchestrated machine runner. Do not replace that value with a CircleCI-hosted machine.image alias; hosted image labels and self-hosted resource classes are different selectors. The orchestrator configuration reference and examples are in CircleCI’s Machine runner orchestrator guide.
Check that the class exists in the same organization
Open the CircleCI organization’s runner settings and verify that the class named by the job exists. If the class was recently renamed, update either the job configuration or the class definition; do not create a second class merely to work around a spelling mismatch without checking which one the project is authorized to use.
2. Verify registration and attachment
Runner inventory is only the first signal
Use the runner inventory to confirm that at least one machine is registered under the target resource class. The troubleshooting documentation warns that a class can exist with no runners attached, causing jobs to wait indefinitely. An inventory entry proves registration, not that the service is currently healthy, able to reach CircleCI, or free of local startup errors.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Inspect a manually managed Linux runner
On the VM or host, check the service and recent journal entries. Typical read-only checks are:
sudo systemctl status circleci-runner
sudo journalctl -u circleci-runner -n 100 --no-pagerLook for authentication failures, inability to reach the configured endpoint, repeated restarts, permission errors, or a service that is installed but inactive. The Linux installation and configuration details are documented in CircleCI’s Machine Runner 3 Linux installation guide and configuration reference.
Inspect an orchestrated VM and its controller
For Machine Runner Orchestrator, first check Kubernetes deployment and pod status, then inspect the provisioner deployment logs. A provisioner that exits at startup commonly indicates a missing secrets.yaml, malformed configuration or secrets, or a resource class entry without its token. After a VM is created, inspect its guest service with systemctl and journalctl as above. This separates a controller that never provisions from a VM whose runner agent cannot start.
3. Validate tokens and API configuration
Use the resource-class token for the class
CircleCI documentation describes the resource-class token as “the method by which CircleCI authenticates that the resource class is valid.” It authenticates the class and allows the runner or orchestrator to claim tasks for it; it is not interchangeable with a token from another class. Compare every token in the orchestrator’s configured classes with the token currently displayed for that exact class. The setup instructions say the token is shown only once, so store it as a secret rather than putting it in a repository, shell transcript, or log.
If a token is wrong or was revoked, regenerate it in the CircleCI web application and update the Helm values or secret used by the provisioner. Restart or roll the deployment according to your Kubernetes change procedure, then watch a fresh poll cycle in the provisioner logs. A token that authenticates one class does not authorize task claims for a different class.
Distinguish CircleCI Cloud from CircleCI Server
| Deployment | Task polling and endpoint checks | Common configuration error |
|---|---|---|
| CircleCI Cloud | Use each class’s resource-class token for the documented polling flow. The personal circleToken is optional and retained for legacy /tasks endpoints. |
Using a wrong-class token, or assuming a legacy personal token replaces the class token. |
| CircleCI Server | Set circleciAPIAddr to the Server instance and verify that runner VMs can reach that address over the network. |
Leaving a Cloud-oriented address in the chart, or a VM route, DNS entry, firewall, or certificate that cannot reach Server. |
Do not copy a Cloud value into Server configuration without checking the applicable deployment. The orchestrator guide documents both paths at https://circleci.com/docs/guides/execution-runner/machine-runner-orchestrator/.
4. Check queue depth and hidden concurrency consumption
Why an idle-looking inventory can still be full
The inventory reports actively running tasks, not every task counted against concurrency. Queued tasks and unclaimed tasks can consume the resource class’s concurrency limit while no machine appears busy. This is the explanation to test when a class has registered runners but new jobs remain queued.
Query tasks through the Runner API
Use the Runner API task query documented by CircleCI’s troubleshooting page, filtered for the affected resource class. Keep the credential handling secure: do not paste tokens into tickets, shell history, CI output, or articles. Compare the returned queued, unclaimed, and running work with the class’s configured concurrency. If stale or abandoned unclaimed tasks are consuming the limit, CircleCI recommends canceling them to release capacity.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsThe endpoint pattern, authentication requirements, and examples are maintained in CircleCI’s self-hosted runner troubleshooting guide. Use that page’s current path rather than relying on a copied URL from an older script.
Interpret orchestrator task counters
The provisioner reports unclaimed_tasks and running-task counts on each poll cycle. If jobs are visibly queued but unclaimed_tasks remains zero, check the resource token, the class key, any configured legacy personal token, and the API address. A token can be valid syntactically yet unauthorized for the class, producing the same zero-demand symptom.
5. Diagnose scaling that does not follow demand
Read the provisioner’s poll-cycle logs
Follow the provisioner deployment logs while submitting a test job. Confirm that the controller can reach the API, reports changing task counts, and computes a desired VM count. If counts never change, stay on the authentication, class-name, and endpoint branch; changing Kubernetes replica settings will not fix a controller that cannot see demand.
Check minReplicas and maxReplicas
Orchestrator scaling is bounded. A fixed VM count may be the configured maximum rather than a failure to detect jobs. If the desired count stops at a ceiling, inspect maxReplicas; if VMs disappear below the expected baseline, inspect minReplicas. Treat these as capacity controls, not task-claim credentials. Increase limits only after confirming that tokens, class names, and API polling are correct and that the Kubernetes cluster can supply the required VM resources.
Confirm KubeVirt and node prerequisites
The documented orchestrator setup requires a Kubernetes cluster with KubeVirt, kubectl, Helm 3 or later, and a resource-class token for every provisioned class. Nodes that host runner VMs require /dev/kvm. The documentation reports testing with KubeVirt v1.8; that is the documented test setup, not a blanket compatibility guarantee for every later or earlier release. A VM that cannot be scheduled, lacks KVM access, or fails its KubeVirt definition is a platform problem after task polling, not a resource-class registration problem.
Best Value
6. Follow the logs according to the symptom
- Provisioner exits before polling: inspect pod status, pod description, and deployment logs for missing secrets, malformed values, or a class without a token.
- Provisioner polls but demand is always zero: verify class-key spelling, token authorization, the optional legacy token if configured, and the Cloud or Server API address.
- Demand is visible but no VM appears: inspect desired count,
maxReplicas, Kubernetes scheduling, KubeVirt status, and node access to/dev/kvm. - VM exists but job remains queued: inspect the guest’s
circleci-runnerservice, registration output, network reachability, and whether the VM registered to the same namespace/name class. - Runner registers then disappears: inspect service restarts and journal errors, then correlate the timestamp with provisioner and Kubernetes events.
This division prevents mixing controller configuration with guest-agent troubleshooting: provisioner logs explain polling and scaling decisions; VM journals explain startup and registration behavior.
7. Fix the documented macOS “Preparing Environment” case
CircleCI documents a specific first-install scenario in which a self-hosted macOS job remains at “Preparing Environment” without an obvious error. The remedy applies to that macOS launch-agent case, not automatically to every Machine Runner 3 platform.
- On the macOS host, ensure
/opt/circleci/circleci-launch-agentis executable by root. - Run the launch agent with the launch-agent configuration file created during installation, using the invocation shown in CircleCI’s troubleshooting instructions.
- Cancel the stuck job and rerun it after the agent is running.
- If the rerun remains stuck, use CircleCI support as directed by the troubleshooting page rather than repeatedly rotating unrelated tokens.
See the macOS-specific guidance in CircleCI’s self-hosted runner troubleshooting documentation. This case is distinct from orchestrator scaling: there is no reason to change maxReplicas when a manually installed macOS launch agent is not executable.
Free tools Windows power users keep installed
One-click scans. No signup required.
8. Separate OIDC runtime failures from registration failures
CircleCI notes a different failure mode in which OIDC token environment variables are missing because the default temporary directory is not writable or is mounted with noexec. The system temporary directory used by the OIDC plugin must be writable and permit execution. Check this only when the runner has already registered and the job reaches runtime; it is not the usual explanation for an absent runner, a zero task count, or a job that cannot claim a resource class.
Manual Machine Runner 3 or Machine Runner Orchestrator?
| Operational path | Capacity model | Primary failure surface | Best first evidence |
|---|---|---|---|
| Manual Machine Runner 3 | Static machines that you install, register, patch, and scale yourself. | Local service, launch agent, network, and class registration. | Runner inventory plus host service status and journal. |
| Machine Runner Orchestrator | Kubernetes-managed VMs bounded by minReplicas and maxReplicas. |
Provisioner secrets, API polling, KubeVirt scheduling, VM boot, and guest registration. | Provisioner poll-cycle logs, Kubernetes events, then VM service logs. |
Neither path is universally better. Static capacity can be predictable when hosts are already managed; orchestration can react to demand but adds Kubernetes, KubeVirt, token, and scaling configuration. Choose the diagnostic branch that matches how the runner is provisioned.
Version and upgrade precautions
The Machine Runner Orchestrator page describes version 1.0.0 and says its chart and image are publicly available. It also warns that field names and defaults may change before general availability. Pin the chart version in production and read the changelog before upgrading. Recheck the live documentation for command syntax and values whenever you change versions; an old values file can otherwise produce a startup failure that resembles a bad token.
Quick Recap
Final verification checklist
- The job’s
resource_classexactly matches the configured namespace/name. - The class exists in the correct organization and has an attached, running machine runner.
- The class token is the current token for that class and is stored as a secret.
- Cloud polling uses the documented class-token flow; Server uses a reachable
circleciAPIAddr. - Runner API results account for queued and unclaimed tasks, not only active inventory entries.
- Provisioner logs show nonzero demand when jobs are queued and a desired VM count within replica limits.
- Kubernetes, KubeVirt,
/dev/kvm, and VM startup are healthy for orchestrated deployments. - Guest service logs show successful runner startup and registration.
- Only the documented macOS launch-agent fix is applied to the macOS first-install symptom.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →
