What to Check Before Buying an AI Server: GPUs, Memory, Networking, and Support

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

Before buying an AI server, match its full configuration to the models and workloads you intend to run—not just the number of GPUs. Check GPU memory and interconnect, CPU and system RAM topology, storage and network paths, facility requirements, software fit, and support terms. NVIDIA’s HGX and DGX documentation provides useful concrete examples, but those configurations are not universal requirements for every organization.

What should I look for in an AI server?

Start by defining the work the server must do, then compare complete configurations against it. A high GPU count or a headline throughput figure cannot tell you by itself whether a system can hold your model, feed the GPUs with data, communicate efficiently across devices, or be installed and supported at your site.

#	Preview	Product	Price
1		Nimo AI NAS, Agentic Computer Mini PC and AI Server, AMD Ryzen 7 PRO 8845HS(up to 5.1 GHZ, beat...	$1,999.99	Buy on Amazon

Before requesting quotes, write down the workload and operating conditions you need to support. Include whether you will train, fine-tune, or serve models; whether work stays on one server or spans several; and the model, dataset, batch, context length, precision, and concurrency you expect. Those details shape nearly every hardware decision that follows.

• Workload: Training, fine-tuning, inference, HPC, or a mix; single-node or distributed.
• GPU capacity: GPU model and generation, number of GPUs, memory per GPU, and total memory.
• GPU fabric: Interconnect type and topology, considered separately from compute throughput.
• Host platform: CPU model and count, system RAM capacity and population, PCIe lanes, and slot layout.
• Storage and networking: Local and shared data paths, compute fabric, and management networks.
• Operations: Power, cooling, rack fit, installation, software compatibility, and support.

Use the same workload assumptions when you compare vendors. Otherwise, quotes may describe different GPU counts, networking, storage, or support scopes and will not be meaningfully comparable.

#1 Best Overall

Nimo AI NAS, Agentic Computer Mini PC and AI Server, AMD Ryzen 7 PRO 8845HS(up to 5.1 GHZ, beat i5-1235u) up to 132TB ZFS Hybrid Storage, Dual 10GbE for 24hr AI Agent

• [Local AI Inference & 70B Model Ready] Equipped with the AMD Ryzen 7 PRO 8845HS processor, NEXUS is engineered for heavy local AI workloads. With a full-size GPU bay, it runs 70B LLMs natively without an internet connection. Ideal for AI developers and tech enthusiasts who need private environment for coding and model testing.
• [132TB Mass Storage with ZFS Integrity] Features a hybrid storage architecture (3×NVMe + 4×3.5" HDD) supporting up to 132TB. Utilizing the enterprise-grade ZFS file system and ECC memory, it prevents data corruption and bit rot—a must-have for professional photographers and video editors safeguarding 4K/8K RAW footage.
• [OpenClaw-Driven Automation Workflow] The built-in OpenClaw execution layer allows complex automated tasks to be processed locally. Even when offline, your backup schedules and AI file organization continue seamlessly. Say goodbye to monthly cloud subscriptions and high latency.
• [Dual 10GbE & USB4 Ultra-Connectivity] Experience server-class speeds with dual 10GbE ports and a 40Gbps USB4 interface. It enables multi-user real-time collaboration on large project files directly from the NAS, ensuring zero-lag editing for creative studios and production teams.
• [Open-Source ZimaOS for Total Privacy] Running on the fully open-source ZimaOS, NEXUS ensures your data stays physically on-premise with no backdoors. It acts as a "Digital Fortress" for privacy-conscious families and small businesses who demand absolute data sovereignty.

$1,999.99

Buy on Amazon

How do I size GPUs and GPU memory for my workload?

GPU memory is a capacity constraint: it affects whether the model and the work you intend to perform can fit in memory. GPU interconnect and bandwidth are separate performance considerations. Estimate both, and do not treat a node’s combined GPU-memory total as proof that a single workload can use it as one undivided pool.

Estimate capacity from the actual workload

Record the model size, sequence or context length, precision, batch size, and expected inference concurrency. For training or fine-tuning, include the intended training setup and whether it spans multiple GPUs or nodes. For inference, include the number of simultaneous requests and the context lengths you need to serve. The capacity you need depends on how your target software maps that work to GPU memory, so confirm the requirements with the framework and deployment configuration you plan to use.

Ask the vendor to identify memory per GPU and total GPU memory, and clarify how the proposed system’s software uses multiple GPUs. If your workload must fit on one GPU, aggregate memory across a node does not answer that requirement. If it is distributed across GPUs, validate that the model-serving or training software supports the distribution you intend to use.

Use published specifications with their scope attached

NVIDIA’s HGX AI Factory reference architecture lists these specifications for its named SXM GPU configurations. They are vendor specifications, not independent end-to-end application benchmarks:

The HGX figures describe these configurations; they do not establish what your application will achieve. Ask vendors for the exact GPU model, memory type and capacity, and the interconnect topology in the quoted system. For application performance, request evidence tied to your workload rather than inferring results from a component specification.

Why should I check GPU interconnect separately from GPU memory?

Memory capacity answers how much data can be held; interconnect affects communication between GPUs. A system can have substantial aggregate memory yet still be a poor fit if the GPU topology or communication path does not suit the way your software splits work across devices.

Ask for the interconnect type, topology, and aggregate bandwidth for the exact configuration. NVIDIA HGX examples use NVLink and NVSwitch; other platforms may use different arrangements. Confirm how GPUs are connected to one another and how the proposed topology aligns with your training or inference software. Do not substitute an advertised compute-throughput number for this information.

If a workload uses several GPUs, request a demonstration or benchmark that matches the intended model, precision, batch or concurrency, and software stack. A result from a different workload can help describe a system, but it does not establish how your workload will perform.

What CPU, system memory, and PCIe details matter?

GPUs do not operate in isolation. Check the host CPU and memory configuration, plus the PCIe layout that connects GPUs, network adapters, and storage devices. These details matter particularly in systems that combine several GPUs with multiple NICs and NVMe drives.

NVIDIA’s HGX 8-GPU design lists at least 1.5 TB of system memory and recommends even distribution across CPU sockets and memory channels. Treat that as a reference-design example, not a minimum for every AI server. Ask for the actual quoted system’s CPU model and count, memory capacity and population, socket and channel layout, and PCIe topology.

• Confirm that the listed RAM capacity is the installed configuration, not only the platform’s maximum.
• Ask how memory is distributed across CPU sockets and memory channels.
• Request a slot and lane map showing GPU, NIC, and NVMe placement.
• Check that the configuration supports the adapters and drives you need without relying on unspecified expansion.

A quote that names CPUs and RAM capacity but omits the topology may leave a key compatibility question unanswered. Have the supplier explain the placement and connectivity of the components in the proposed build.

How much storage should an AI server have?

Size storage for the data path and lifecycle of your workload, not only for the operating system. Account for dataset staging, training data access, model files, cache, checkpoints, and scratch space, then decide what belongs on local NVMe versus shared storage.

NVIDIA’s HGX reference recommends local NVMe per CPU socket, with different recommendations for inference, training or deep learning, and HPC, plus a separate boot drive. The appropriate capacity and performance depend on the workload, so request a configuration tied to your workflow rather than assuming a boot drive can serve as the data store.

Map storage to the work it performs

• Boot: Identify the separate drive or drives reserved for the operating system.
• Dataset staging: Estimate the local data needed for the portion of a dataset accessed during a job.
• Cache and scratch: Include temporary files and intermediate data created by your software.
• Checkpoints: Estimate how much checkpoint data must be written and how frequently your workflow writes it.
• Shared storage: If jobs read from or write to shared storage, specify the required path and bandwidth as part of the system design.

Ask for NVMe capacity, drive count and placement, and the intended boot-versus-data arrangement. If local storage is only a staging tier, establish how data moves between it and shared storage and whether the network path is part of the proposed configuration.

What networking does an AI server need?

Separate the network used for distributed compute from the paths used for storage, customer traffic, and management. Those jobs have different purposes, and a quote that gives only one NIC speed may not explain how the complete system will connect.

NVIDIA’s HGX reference architecture distinguishes east-west compute traffic from north-south storage and customer traffic. In the cited design, it recommends up to 400 Gb/s per adapter for the compute fabric. That is a reference configuration, not a universal requirement for all servers or clusters.

Specify each network path

• Compute fabric: State whether the cluster will use InfiniBand or Ethernet/RoCE, and specify adapter count and speed per node.
• Switching and links: Include switches, optics or cables, connector type, compatibility, and the intended oversubscription.
• Storage and customer traffic: Identify the north-south path and the storage or service traffic it must carry.
• In-band management: Ask how the system’s operational management traffic is connected.
• Out-of-band management: Confirm the separate path used for baseboard management controller (BMC) access.

NVIDIA’s physical topology document describes a converged storage and in-band management network as well as an isolated out-of-band network for BMC management. The important buyer question is whether the quoted topology provides the paths your environment needs—not whether it copies one reference design exactly.

Check that adapters, switches, link protocol, and connectors are compatible across the proposed configuration. Include cabling and optics in the quote; a fast adapter alone does not establish that the network fabric is complete or ready to deploy.

How do I know whether my site can run the server?

Confirm power, cooling, rack space, rack depth, weight, and installation requirements for the exact system and configuration before purchase. A facility’s ability to host a smaller server does not establish that it can support a high-power GPU system.

For a product-specific reference, the NVIDIA DGX H100 datasheet lists system power at approximately 10.2 kW maximum. NVIDIA’s DGX H100/H200 user guide describes those systems as 8U rack systems. These figures apply to the named products; verify the exact system revision and installation requirements with the vendor and your facilities team.

Have the supplier provide the complete system’s power and cooling figures, dimensions, weight, and rack-depth requirements. Confirm the required electrical service and cooling capacity with the people responsible for the site. Include delivery, installation, and any site preparation in the purchasing plan and quote.

What software and support should I confirm?

Verify software compatibility and support scope before treating the hardware as ready for production. Ask which operating systems, drivers, orchestration tools, and workload software the vendor supports for the quoted configuration, and identify any required software subscriptions or services.

Support terms vary by product and seller. The NVIDIA DGX H100 datasheet gives a product-specific example of three-year business-standard hardware and software support; it is not a universal term for AI servers. Compare the exact quote’s coverage and cost rather than assuming another vendor includes the same service.

Questions to put in the quote

• What hardware and software support is included, and for what term?
• What response coverage applies, and when is support available?
• Which software components are covered, and are any subscriptions separate?
• What is the process and expected logistics for replacement parts or systems?
• What do support renewals cost after the included term?
• Are deployment, installation, or service costs included in the delivered price?

Ask for the answers in writing and compare them alongside the hardware configuration. A support term without its coverage, response arrangement, and renewal cost is not enough to establish the ongoing cost or service level.

How can I compare two AI server quotes fairly?

Compare like-for-like delivered configurations, including software, network components, installation, and support. GPU-only headline numbers can conceal meaningful differences in memory, topology, storage, fabric, or service.

Request comparable quotes with the same workload assumptions and the same stated scope. Where a supplier does not specify a detail, ask for it rather than treating an omitted item as included.

What should I put on my AI server buying checklist?

Use this checklist to turn your requirements into questions suppliers can answer against a specific configuration.

1. Record intended model sizes, sequence or context length, batch size, precision, concurrency, and whether training spans multiple nodes.
2. Estimate GPU memory needs; compare memory per GPU with total memory, and confirm how your target software uses multiple GPUs.
3. Compare interconnect type and topology separately from advertised compute throughput.
4. Check CPU sockets, installed RAM capacity and population, PCIe lanes and slot layout, and NIC/GPU placement.
5. Calculate local NVMe requirements for boot, datasets, cache, checkpoints, and scratch; identify shared-storage bandwidth needs.
6. Specify the cluster fabric, adapter count and speed, switches, optics or cables, oversubscription, and separate management paths.
7. Obtain power and cooling figures for the exact configuration; confirm rack dimensions, weight, facility readiness, and installation.
8. Ask what hardware and software support is included, its term and response coverage, replacement process, and renewal price.
9. Compare complete delivered configurations and support terms across vendors rather than GPU-only specifications.

Frequently Asked Questions

Can I use the total memory across eight GPUs as if it were one large GPU?

Do not assume so. Whether a workload can distribute its memory across devices depends on the GPU topology and the software’s multi-GPU behavior; verify that exact arrangement with the vendor and the software provider.

Does a 400 Gb/s adapter mean the whole cluster network runs at 400 Gb/s?

No. The cited HGX figure is up to 400 Gb/s per adapter in that reference design, not a guarantee of end-to-end cluster throughput. Switches, cabling, topology, and oversubscription also shape the network path.

Should I buy the server before my facility is ready?

First obtain the exact configuration’s power, cooling, dimensions, weight, and installation requirements, then confirm them with your facilities team. The DGX H100’s approximately 10.2 kW maximum power figure is product-specific, so it should not be applied to a different system.

Quick Recap

Bestseller No. 1

Nimo AI NAS, Agentic Computer Mini PC and AI Server, AMD Ryzen 7 PRO 8845HS(up to 5.1 GHZ, beat i5-1235u) up to 132TB ZFS Hybrid Storage, Dual 10GbE for 24hr AI Agent

$1,999.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.