How to Deploy and Run AI Models on Intel Gaudi Accelerators

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

To deploy an AI model on an Intel Gaudi accelerator, first match the Gaudi2 or Gaudi3 hardware, host driver, SynapseAI release, operating system, Python, framework, and container image in Intel’s support matrix. Then verify the node with hl-smi, register the Habana container runtime, launch the matching Intel Gaudi image, and run your PyTorch, TensorFlow, DeepSpeed, distributed-training, vLLM, SGLang, or quantized-inference workflow. The same process works on a self-managed server and on hosted capacity, but hosted nodes often arrive with drivers already installed.

This guide shows the exact checks, container settings, training and serving paths, and recovery steps. It also explains when to choose Gaudi2 or Gaudi3, Docker or Kubernetes, source installation or a prebuilt image, and a hosted instance or your own server.

What do I need before deploying a model on Gaudi?

You need four compatible layers: an accelerator node, the host software, a framework environment, and the model application. A failure in any one layer can prevent the container from seeing devices or can produce package errors after startup.

Choose Gaudi2 or Gaudi3 capacity

Pick a Gaudi2 or Gaudi3 node according to the model size, memory requirement, concurrency target, and scale-out plan. The available documentation covers both generations, but exact support depends on the software release rather than the accelerator name alone. Do not assume that an image validated for one generation is valid for the other.

You can install the hardware in a compatible server or use hosted capacity. Intel’s setup documentation identifies Intel Tiber AI Cloud and Amazon EC2 DL1 as access routes. For either route, confirm the target region, quota, networking, storage, and the provider’s supported Gaudi software release before you reserve capacity. Enterprise availability can differ by region.

Plan the host and operating system

Your host must have a Gaudi driver and firmware combination supported by the selected SynapseAI release. The displayed Gaudi3 configuration for Intel Gaudi software 1.24.1 lists Ubuntu 22.04.5 or 24.04.2, and RHEL 9.6 or 9.8. It lists Python 3.10–3.12 depending on the operating system, Docker 29.1.5, Kubernetes 1.34, 1.35, or 1.36, and PyTorch 2.11. Treat these values as a release-specific matrix entry, not as universal requirements for every Gaudi installation.

Decide how the model will run

For training, the supported stack includes PyTorch and TensorFlow, with DeepSpeed and distributed-training paths. For inference, the documentation covers PyTorch inference, vLLM, SGLang, and quantized FP8 and INT4 workflows. Select the framework and serving path before choosing an image, because the image tag must contain the corresponding SynapseAI and framework versions.

How do I use the Gaudi support matrix?

Use the support matrix as the authority for every version combination. It is more reliable than copying a command from an older blog post or choosing the newest generic PyTorch image.

1. Identify the accelerator generation. Confirm whether the node exposes Gaudi2 or Gaudi3.
2. Record the installed driver. Run hl-smi | grep Driver on the host and save the reported version.
3. Check the operating system. Match the exact distribution and point release listed for your target SynapseAI release.
4. Check Python and framework versions. Use the matrix entry for your operating system, then select the supported PyTorch or TensorFlow build.
5. Check container and orchestration versions. For containerized work, verify Docker; for cluster work, verify Kubernetes or the documented Slurm path for the release.
6. Select the image or plugin tag. Use the image tag associated with that matrix row. For vLLM, also select the Gaudi plugin release that matches the installed Gaudi software.

Intel’s current Gaudi documentation groups the relevant workflows under migration, training, distributed training, PyTorch inference, vLLM, SGLang, and quantized inference. Read the workflow page after selecting the matrix row; it supplies model-specific flags and any release limitations.

How do I verify that a Gaudi node is ready?

Run the checks on the host before debugging your model code. They distinguish a hardware or runtime problem from an application problem.

Confirm that the accelerator is visible

Start with:

hl-smi

The command should list the Gaudi devices and their status. To identify the installed driver specifically, use:

hl-smi | grep Driver

If no devices appear, verify that you are on a Gaudi node, that the host driver and firmware are installed, and that your user has permission to access the device files. A container cannot repair a missing host driver.

Confirm the Habana container runtime

Docker must know the habana runtime before you launch an Intel Gaudi image. Check that the habanalabs-container-runtime package is installed and registered with the container engine. If it is absent, install the package supplied for your host’s driver and SynapseAI release, configure the container service, and restart the service as required by that release.

The expected failure when this step is missing is an error such as Unknown runtime specified habana. Fix the registration first; changing the model command will not resolve a runtime that Docker does not recognize.

Check permissions and shared resources

Containerized distributed jobs commonly require host networking, shared IPC, and permission to adjust process scheduling. Confirm that your account can invoke Docker and that the host firewall and network policy permit traffic between training processes. Validate storage for model weights, checkpoints, and logs before starting a long job.

How do I start a compatible Intel Gaudi container?

Use a versioned Intel Gaudi image selected from the support matrix. Intel’s example passes the Habana runtime, exposes all devices, disables a specific Open MPI shared-memory mechanism, grants the scheduling capability, and shares the host network and IPC namespaces.

Set the image variable to the exact tag from the matrix, then run:

export GAUDI_IMAGE='<matrix-selected-intel-gaudi-image>'
docker pull "$GAUDI_IMAGE"
docker run -it --runtime=habana 
  -e HABANA_VISIBLE_DEVICES=all 
  -e OMPI_MCA_btl_vader_single_copy_mechanism=none 
  --cap-add=sys_nice 
  --net=host 
  --ipc=host 
  "$GAUDI_IMAGE"

The angle-bracket value is a shell variable value, not a tag to copy literally. Replace it with the image reference listed for your Gaudi generation, driver, SynapseAI release, operating system, and framework. Do not reuse an old tag simply because it starts successfully.

Why each option matters

• --runtime=habana selects the Habana device runtime rather than a generic container runtime.
• HABANA_VISIBLE_DEVICES=all makes all assigned Gaudi devices available inside the container. Restrict it to a device list when the workload is intentionally partitioned.
• OMPI_MCA_btl_vader_single_copy_mechanism=none applies Intel’s documented Open MPI setting for the example environment.
• --cap-add=sys_nice grants the scheduling capability used by some accelerator workloads.
• --net=host and --ipc=host simplify high-performance communication and shared-memory behavior, especially for distributed jobs.

After the shell opens, run a small framework smoke test before downloading a large model. Confirm that the framework imports, the device is enumerated, and a few tensors or a short forward pass execute successfully.

How do I run PyTorch or TensorFlow training?

Use the framework environment supplied by the Intel Gaudi image whenever possible. Intel states that its optimized framework containers include the required dependencies and the complete SynapseAI software, and that SynapseAI is integrated and validated with recent TensorFlow and PyTorch versions.

PyTorch training

Start with a single-device or small-batch smoke test. Adapt the model’s device-selection code to the Gaudi framework path documented for the selected release, then run a short training loop and confirm loss updates and checkpoint creation. Only after that test should you increase sequence length, batch size, or device count.

Keep the host driver, container image, PyTorch build, and SynapseAI release aligned. Installing a newer generic PyTorch package inside a validated image can silently replace a compatible dependency and produce runtime errors. If you need a different PyTorch version, choose a different matrix row and image rather than upgrading packages ad hoc.

TensorFlow training

TensorFlow2 is listed among the supported frameworks in Intel and Habana Labs’ Gaudi2 white paper. Use the TensorFlow environment supplied for the matching Gaudi release, run a small dataset first, and verify that the accelerator is selected rather than the CPU. Framework support does not mean every third-party layer is validated; test custom operations and data pipelines separately.

DeepSpeed and distributed training

The Gaudi documentation provides DeepSpeed and distributed-training routes. Follow the release-specific launch command and environment variables for your selected framework. Confirm device visibility on every process, use the host-network and IPC settings required by the image, and make sure all nodes run the same driver, image, SynapseAI, and model code.

For multi-node work, test a communication-only job before a full model run. A successful single-node training job does not prove that routing, firewall rules, process discovery, or collective communication are configured correctly across nodes.

How do I serve a model with vLLM on Gaudi?

vLLM on Gaudi uses a Gaudi-specific plugin and release combination. The documented installation choices are Docker Compose, a Dockerfile, or a source installation, and the documentation covers Gaudi2 and Gaudi3. Use the route that matches your operational needs, but keep the plugin tag aligned with the installed Gaudi software.

Choose an installation route

Route	Best fit	What it provides
Docker Compose	Repeatable local or small-cluster service deployment	Ready-to-run service definition and container parameters
Dockerfile	A customized image with controlled build steps	A reproducible image that can add approved application dependencies
Source installation	Development or a change not yet packaged in an image	Direct control over the plugin and Python environment

Use a ready-to-run image when you want the fewest moving parts. Choose a Dockerfile when you need to add your own serving layer while retaining a fixed base. Use source installation only when you can reproduce the supported Linux, Python, Gaudi, and plugin combination and are prepared to maintain it.

Match the plugin and model

The vLLM quickstart documents custom runtime parameters, benchmarking, and validated model families including LLaMA, Mistral, and Qwen. Model-family validation is release-dependent, so check the plugin’s matching tag and model notes before deploying a production endpoint. A model that loads is not automatically validated for your context length, quantization mode, or concurrency target.

Validate serving before opening traffic

1. Launch the documented Gaudi vLLM image or build the matching Dockerfile.
2. Confirm that the process can enumerate the assigned Gaudi devices.
3. Load a small or representative model and send a short prompt.
4. Check generated tokens, latency, memory use, and logs under the intended sequence length.
5. Run the plugin’s benchmark procedure with your production-style prompts before setting concurrency or autoscaling limits.

Do not mix generic vLLM instructions with CUDA-oriented assumptions. The Gaudi plugin, image, runtime parameters, and supported model list are part of one release-specific stack.

Can I use SGLang or quantized inference?

Yes. Intel’s Gaudi documentation includes SGLang and quantized inference routes, including FP8 and INT4 workflows. Select the exact guide for your Gaudi generation and software release, then verify that the model architecture and quantization format are listed for that release.

Quantization changes memory use and may change numerical behavior or throughput. Validate output quality on your own prompts, especially when moving from full-precision weights to FP8 or INT4. Keep the quantization conversion and serving versions tied to the same support-matrix row.

Should I deploy with Docker, Kubernetes, or Slurm?

Use Docker for the fastest single-node path, Kubernetes when you need cluster scheduling and service management, and Slurm when your organization already runs batch-oriented high-performance computing. The Gaudi2 support table lists Docker as the container runtime, Kubernetes as an orchestration option, and PyTorch distributed data parallel as a training path.

Docker

Docker is the simplest way to reproduce Intel’s documented runtime flags. It is appropriate for development, one server, and a controlled inference service. You remain responsible for image lifecycle, device allocation, restarts, secrets, and log collection.

Kubernetes

Kubernetes adds scheduling, rolling deployment, and multi-tenant controls, but the cluster must expose Gaudi devices through the supported device integration and use versions accepted by the matrix. For the displayed Gaudi3 software 1.24.1 configuration, the listed Kubernetes versions are 1.34, 1.35, and 1.36. Match the node image, device plugin, container runtime, and workload manifest to that release.

Slurm and batch scheduling

Use the distributed-training instructions for your release when integrating with Slurm or another batch scheduler. Device assignment, host lists, environment propagation, and network interfaces must be consistent across the allocation. Avoid launching a scheduler-managed job with Docker flags copied from a single-node shell unless the cluster administrator has approved that pattern.

Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

What is the difference between hosted Gaudi access and a self-managed server?

Hosted access reduces hardware installation work; self-managed hardware gives you control over the host image, networking, storage, and maintenance schedule. Neither option removes the need to match the software matrix.

Decision factor	Hosted Gaudi node	Self-managed Gaudi server
Initial setup	Provider may preinstall the appropriate driver; load the compatible Intel Gaudi image	Install and maintain the supported driver, firmware, runtime, and operating system
Capacity	Request the provider’s available region and quota	Provision the accelerator capacity you own or lease
Networking	Use the provider’s network and security controls	Design host, cluster, and inter-node networking yourself
Maintenance	Coordinate with provider image and release availability	Schedule upgrades, repairs, and compatibility testing

Intel’s setup documentation says that Gaudi nodes or virtual machines from a cloud provider usually have the appropriate drivers preinstalled and that users should load the latest compatible Intel Gaudi Docker image. “Latest” still means the image supported by the installed driver and release, not necessarily the newest tag published for another configuration.

How do I troubleshoot a failed Gaudi deployment?

Classify the failure before changing versions. The symptom usually identifies whether the problem is hardware visibility, runtime registration, package compatibility, networking, or the serving plugin.

hl-smi shows no devices

• Confirm that the machine is a Gaudi node and that the accelerator is attached to the instance or server.
• Check the host driver and firmware status.
• Verify permissions for the user and container engine.
• Run hl-smi outside the container before debugging the model.

Docker reports Unknown runtime specified habana

Install and register habanalabs-container-runtime, configure the container engine, restart the service if required, and retry with --runtime=habana. If the runtime package came from a different SynapseAI release, replace it with the package that matches the host driver and selected image.

The image starts but imports or packages fail

Compare the host driver, firmware, SynapseAI, PyTorch or TensorFlow version, Python version, and image tag against one support-matrix row. Remove ad hoc package upgrades from the container and rebuild from a validated base image. A package mismatch can appear as an import error, an unavailable operator, or a device initialization failure.

Distributed training fails while single-node training works

Check host networking, IPC sharing, device visibility, process discovery, firewall rules, and the distributed-training requirements for the release. Ensure every node uses the same image and software stack. Run a communication test before restarting the full model job.

vLLM will not start

Use the vLLM Gaudi plugin release and container or source path that match the installed Gaudi software. Remove generic vLLM flags intended for another accelerator backend, then verify that the model family, Python version, and Linux distribution are supported by that plugin release.

What deployment checklist should I use in production?

Record the following information for every node and workload so that a later upgrade can be reproduced:

• Gaudi generation and host identifier.
• Driver and firmware versions reported by hl-smi.
• SynapseAI release and support-matrix row.
• Operating system point release, Python version, and framework version.
• Exact container image digest or tag.
• Habana runtime package and container-engine configuration.
• Model revision, tokenizer or preprocessing revision, and quantization format.
• Device allocation, network and IPC settings, and distributed launch configuration.
• Benchmark prompts, sequence lengths, batch or concurrency settings, and quality checks.
• Rollback image and the last known-good configuration.

For a production change, upgrade one layer at a time in a staging node. Re-run device visibility, a framework smoke test, model loading, a short workload, and the serving or distributed benchmark before changing the next layer.

What performance claims can I safely make?

Do not generalize performance from the accelerator name, a model-family example, or a vendor quickstart. The available official material does not provide one current, directly comparable benchmark figure for every Gaudi2, Gaudi3, model, and workload combination.

Measure your own workload with fixed model weights, prompt or training data, sequence lengths, batch size, precision, concurrency, and software versions. Report throughput, latency, memory use, and output-quality checks together, and identify whether the result came from one node or a multi-node run.

Frequently Asked Questions

Do I have to rewrite a CUDA model before using Gaudi?

Often you can keep the model’s high-level PyTorch or TensorFlow structure, but accelerator-specific kernels, memory assumptions, and custom extensions may require changes. Start with Intel’s model-migration guidance for the selected release and test every custom operation rather than assuming CUDA-only code will run unchanged.

Can I install the newest PyTorch package inside an Intel Gaudi image?

Do not do that by default. The supported PyTorch build is coupled to the driver, SynapseAI release, and image; choose a different support-matrix row when you need another version instead of upgrading the container blindly.

How do I limit a job to only some Gaudi devices?

Set the device visibility environment variable to the device list assigned to that workload rather than exposing all devices. Confirm the resulting list from inside the container and keep the scheduler’s allocation consistent with it.

Is a hosted Gaudi instance automatically ready for vLLM?

A hosted node may include the host driver, but vLLM still requires a compatible Gaudi plugin, image or source environment, Python version, and model support. Check the provider’s installed software release and select the matching plugin tag before deploying the endpoint.

The Bottom Line

A reliable Gaudi deployment is a compatibility exercise: select the hardware and access model, match one support-matrix row, verify the host and Habana runtime, launch the corresponding image, and validate the framework and model with a small workload before scaling out.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.