iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Short answer: start with NVIDIA’s published Nemotron 3.5 Lightning NVFP4 checkpoint when your target GPU and runtime support it. That checkpoint uses a W4A16 four_over_six recipe and quantization-aware distillation. For other Nemotron 3.5 checkpoints, select INT4 AWQ, FP8, or INT8 SmoothQuant only when the exact checkpoint, GPU architecture, JetPack/CUDA stack, and TensorRT runtime are listed as compatible.
A 4 GB device requires the smallest compatible checkpoint, batch size one, a short context, and a conservative generation limit. A 24 GB device may run a larger checkpoint or higher-precision format, but neither capacity guarantees a particular model or context length. Weights are only part of the allocation: KV cache, activations, quantization scales, engine workspace, CUDA allocations, metadata, and the operating system all need headroom. Measure peak allocation on the exact board instead of treating the advertised memory size as a weight-file limit.
What quantization changes in a Nemotron 3.5 deployment
Quantization stores some model values with fewer bits. A lower-bit representation can reduce weight memory and sometimes improve throughput, but it can also change task accuracy, instruction following, and long-context behavior. The result depends on the checkpoint, calibration data, kernels selected by the runtime, and the target GPU.
The published Nemotron 3.5 Lightning recipe starts with a BF16 checkpoint, applies quantization-aware distillation, and publishes an NVFP4 checkpoint using W4A16 four_over_six. This is different from taking an arbitrary BF16 checkpoint and converting it with an unverified script: the distilled checkpoint is the one NVIDIA prepared for that recipe.
#1 Best Overall
- 【Core Parameters】★AI Perf: 34/67 TOPS ★GPU:1024-core official Ampere architecture GPU with 32 Tensor Cores ★CPU:6-core Arm Corte-A78AE v8.2 64-bit CPU 1.5MB L2 + 4MB L3 ★Memory:8GB 128-bit LPDDR5 68 GB/s ★Storage: external NVMe via M.2 Key M
- 【Empowered by Large Al Model, Enhanced Human-Computer Interaction】Jetson Orin Super leverages three AI models and incorporates an AI voice interaction module. This multimodal visual system matches the scene being described, enabling environmental awareness and AI visual gameplay. Combined with a large-scale voice module and camera, it enables speech-to-text, semantic analysis, natural conversation, and real-time video analysis, enabling advanced embodied AI applications.
- 【AI Upgrade】Jetson Orin Nano series modules are compact in size but can deliver up to 34-67 TOPS of AI performance, with power consumption ranging from 7 watts to 25 watts. Compared to the Jetson Nano B01, it offers up to 80 times the performance and sets a new standard for entry-level edge AI.
- 【Highly compatible carrier board】Yahboom's carrier board is fully compatible with orin nano module. Compared to carrier boards that use Jetson Nano on the market, the newly upgraded circuit supports 25W power mode, which enables larger and more complex neural networks and fully leverages the performance of the core module. The resources, size, and interfaces of the Yahboom carrier board are consistent with the official board, with the only difference addition of power switch button.
- 【Tutorial materials provided】The JETSON system based on Ubuntu 22.04 provides a complete desktop Linux environment with accelerated graphics, supporting CUDA 12.6, TensorRT 10.7.0, cuDNN 9.6.0, OpenCV 4.10.0, etc. The performance on AI LLM, VLM and visual Transformer is significantly improved compared with the previous generation.
Available alternatives are exposed through NVIDIA Model Optimizer. Its documented methods include FP8, INT8 SmoothQuant, and INT4 AWQ. TensorRT Edge-LLM also lists Nemotron checkpoints in its support matrix, but support is version- and architecture-dependent. Always check the current matrix before exporting an engine.
Choose the format before you build
| Format or recipe | What it represents | When to prefer it | Important qualification |
|---|---|---|---|
NVFP4 W4A16 four_over_six |
The published Nemotron 3.5 Lightning checkpoint format; weights are 4-bit and activations use a 16-bit path. | First choice when deploying the published Lightning checkpoint on a supported NVIDIA edge stack. | Use the NVIDIA checkpoint and its documented export path; do not assume every Nemotron variant has an equivalent published NVFP4 file. |
| INT4 AWQ | Four-bit weight quantization using activation-aware weight calibration. | Useful for a supported checkpoint when the target TensorRT stack lists INT4 AWQ. | Quality and build behavior depend on calibration data and supported kernels. Verify the exact GPU and runtime combination. |
| FP8 | A lower-precision floating-point representation. | Useful on hardware and software that provide a validated FP8 path, especially when INT4 quality is insufficient. | Do not select it solely because the file is smaller; confirm model, architecture, and TensorRT support first. |
| INT8 SmoothQuant | Eight-bit integer quantization that smooths activation and weight ranges before conversion. | A conservative option when INT8 kernels are well supported and you need more numerical margin than INT4. | It generally uses more weight memory than INT4, so it may not be viable on a strict 4 GB budget. |
Compare candidates on five axes: memory fit including runtime overhead, quality on your workload, first-token and decode speed, compatibility with the exact software stack, and operational behavior such as power and temperature. A format that fits the weights but leaves no space for the KV cache is not a usable deployment.
Pin the software and checkpoint release
Quantization support changes as checkpoints and runtimes evolve. Create a release record before you download or convert anything.
- Record the exact Hugging Face or NVIDIA checkpoint name and revision. Include the model variant, not only the Nemotron family name.
- Save the tokenizer revision and license information with the deployment package.
- Record the TensorRT Edge-LLM or TensorRT-LLM version, CUDA version, JetPack version, operating-system image, and Jetson SKU or other GPU model.
- Check NVIDIA’s current TensorRT Edge-LLM support matrix for the checkpoint, precision, GPU architecture, and required build options.
- Keep the BF16 reference checkpoint, if licensing and storage permit, so that you can compare quality after conversion.
Do not mix an engine built with one TensorRT or CUDA release with a different runtime image unless that combination is explicitly supported. A successful export is not proof that the resulting engine is valid on another board or software image.
Budget memory as a runtime allocation
Use this accounting model before choosing a checkpoint:
- Compressed weights: the quantized parameter storage, including packing overhead.
- Scales and metadata: per-group or per-channel scales, zero points where applicable, layer metadata, and tokenizer or embedding data.
- KV cache: storage for keys and values grows with sequence length, number of layers, attention dimensions, batch size, and concurrent requests.
- Activations: temporary tensors needed during prefill and decoding.
- Engine workspace: scratch memory requested while TensorRT builds or executes kernels.
- CUDA and runtime allocations: allocator pools, libraries, context memory, and buffers.
- Operating-system reservation: memory used by the board’s services, display stack, monitoring, and other processes.
Estimate the compressed weights, add every category above, then leave a safety margin. Finally, profile peak allocation during both engine build and inference; build-time workspace can exceed steady-state inference memory. Reject a configuration that only works when the operating system is nearly out of memory.
Build a 4 GB deployment
The 4 GB tier is a constrained single-request target. Treat it as an explicitly bounded profile rather than a smaller version of a server deployment.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →1. Start with the smallest compatible checkpoint
Choose the smallest Nemotron 3.5 checkpoint that meets the task requirement and appears in the support matrix for your board. Prefer the published NVFP4 W4A16 four_over_six Lightning checkpoint when it is supported. If that checkpoint is unavailable for your variant, evaluate INT4 AWQ only through the documented NVIDIA path.
2. Fix the initial runtime envelope
- Use batch size one.
- Allow one request at a time while establishing a baseline.
- Set the shortest context that still covers the application’s prompts.
- Set a conservative maximum output length.
- Disable optional services and background workloads during profiling.
These settings reduce KV-cache and activation growth. Increase only one variable at a time after recording peak memory.
3. Prepare representative calibration data
For AWQ, SmoothQuant, or another conversion that requires calibration, use a fixed, license-compatible sample representing the production language, prompt style, and sequence lengths. Include long examples if the application will use long context; a short-only calibration set can hide degradation at the upper context limit. Hash and version the sample so that later comparisons use the same data.
4. Export with the version-matched NVIDIA workflow
Follow the Model Optimizer and TensorRT Edge-LLM instructions that match the pinned release. Do not substitute flags from a different version. Inspect export logs for unsupported layers, precision fallbacks, and unexpected workspace requests.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minute5. Externalize weights when the build requires it
NVIDIA’s TensorRT Edge-LLM guidance warns that INT4 engine builds on low-memory Jetson Orin devices, including Orin Nano-class systems, may need --externalize-weights to reduce engine-build memory. Apply that documented option when the support matrix or build log calls for it, and verify where the external weight files are stored and loaded at runtime.
Rank #2
- 【Core Parameters】★AI Perf:34-67 TOPS ★GPU:512-core NVIDIA Ampere architecture GPU with 16 Tensor Cores ★CPU:6-core Arm Corte-A78AE v8.2 64-bit CPU 1.5MB L2 + 4MB L3 ★Memory:4GB 64-bit LPDDR5 51 GB/s ★Storage: external NVMe via M.2 Key M (NOTE:SUB Board No SD Card Slot)
- 【Empowered by Large Al Model, Enhanced Human-Computer Interaction】Jetson Orin Super leverages three AI models and incorporates an AI voice interaction module. This multimodal visual system matches the scene being described, enabling environmental awareness and AI visual gameplay. Combined with a large-scale voice module and camera, it enables speech-to-text, semantic analysis, natural conversation, and real-time video analysis, enabling advanced embodied AI applications.
- 【AI Upgrade】Jetson Orin Nano series modules are compact in size but can deliver up to 34-67 TOPS of AI performance, with power consumption ranging from 7 watts to 25 watts. Compared to the Jetson Nano B01, it offers up to 80 times the performance and sets a new standard for entry-level edge AI.
- 【Highly compatible carrier board】Yahboom's carrier board is fully compatible with orin nano module. Compared to carrier boards that use Jetson Nano on the market, the newly upgraded circuit supports 25W power mode, which enables larger and more complex neural networks and fully leverages the performance of the core module. The resources, size, and interfaces of the Yahboom carrier board are consistent with the official board, with the only difference addition of power switch button.
- 【Tutorial materials provided】The JETSON system based on Ubuntu 22.04 provides a complete desktop Linux environment with accelerated graphics, supporting NVIDI-ACUDA 12.6, TensorRT 10.7.0, cuDNN 9.6.0, OpenCV 4.10.0, etc. The performance on AI LLM, VLM and visual Transformer is significantly improved compared with the previous generation.
6. Profile prefill and decode separately
Prefill can create a higher activation peak than token-by-token decoding, while decoding grows the KV cache as the sequence becomes longer. Record peak allocation for an empty prompt, a typical prompt, and the maximum allowed context. Also record time to first token and decode tokens per second at batch size one.
7. Increase limits cautiously
If the baseline leaves measured headroom, increase context or output length in small increments and rerun the same prompt suite. Stop when peak allocation approaches the device budget, latency becomes unacceptable, or quality falls below the application threshold. Do not infer a safe context length from the model’s nominal maximum.
Use a 24 GB device without wasting the extra memory
A 24 GB-class NVIDIA edge or GPU platform gives you more choices, not an automatic guarantee that every Nemotron 3.5 variant will run. The practical context length depends on the checkpoint’s architecture, precision, batch size, concurrency, and the memory consumed by the operating system and runtime.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Establish the same conservative baseline
Begin with one request, batch size one, a short context, and a conservative output limit. This makes it easier to identify the model’s steady-state memory and confirms that the software stack is correct before you add pressure.
Choose what to spend the headroom on
- More context: useful when prompts contain documents or conversation history; KV-cache growth must be measured at each increase.
- Higher precision: FP8, INT8 SmoothQuant, or BF16 may preserve quality better than INT4 for sensitive tasks, provided the target hardware and runtime support the format.
- A larger checkpoint: possible only if compressed weights plus runtime allocations leave sufficient space.
- Batching or concurrency: increases throughput but multiplies activation and KV-cache demand.
Change one dimension at a time. A 24 GB board may run a larger model at short context but fail at the same model’s long-context or multi-request target. Publish the measured context, batch, and concurrency rather than claiming that “24 GB supports” a universal limit.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Calibration and export practices that preserve quality
Match calibration to production
Calibration examples should reflect the languages, formatting, instruction types, and sequence lengths users will send. Keep a separate, untouched evaluation set. Reusing calibration prompts for quality claims can hide errors introduced by quantization.
Keep precision boundaries visible
Record which layers remain at a higher precision, whether the runtime inserted fallback kernels, and whether embeddings or output heads were treated differently. A nominal INT4 or NVFP4 label does not describe every tensor in the engine.
Inspect build output
Save logs showing selected tactics, workspace allocation, rejected tactics, fallback layers, and externalized-weight handling. A build that silently falls back to a slower or higher-memory implementation can change both performance and fit.
Separate engine-build memory from inference memory
Build the engine on the target board when possible, or reproduce its memory constraints in the build environment. If the build requires more temporary memory than the device has, reduce workspace according to the documented runtime controls or use the supported externalized-weight option; do not delete safety reservations from the operating system.
Validate quality, speed, and thermal behavior
Use a fixed prompt suite and compare the quantized engine with BF16 or another higher-precision baseline whenever that baseline can run the same tasks. Record:
- Task accuracy or a clearly defined human-evaluation rubric.
- Instruction-following and formatting success.
- Errors at short, typical, and maximum tested context.
- Time to first token and decode tokens per second.
- Peak memory during build, prefill, and decode.
- Power mode, temperature, throttling events, and sustained throughput.
Run repeated trials after warm-up and report the exact model revision, tokenizer, software versions, board SKU, power mode, context length, batch size, and generation limit. Published NVIDIA material does not establish a universal benchmark proving that every Nemotron 3.5 variant fits every 4 GB or 24 GB device, so your measurements must define the claim.
Troubleshoot failures by symptom
| Symptom | Likely cause | Action |
|---|---|---|
| Engine build runs out of memory | Temporary TensorRT workspace, tactic selection, or weight duplication during build. | Check the version-matched build guidance; on supported low-memory Orin systems, try --externalize-weights. Build with nonessential services stopped and inspect workspace logs. |
| Engine builds but inference allocation fails | KV cache, activations, runtime pools, or system reservation were omitted from the estimate. | Reduce context, output limit, batch, or concurrency; then remeasure peak allocation before trying another format. |
| Unsupported-layer or precision fallback appears | The checkpoint, GPU architecture, or runtime does not support the requested path. | Verify the support matrix and pinned versions. Select a documented alternative such as INT4 AWQ, FP8, or INT8 SmoothQuant only when the combination is listed. |
| Quality drops on long prompts | Calibration data does not represent long sequences, or the lower precision is sensitive to the task. | Add representative long-context calibration examples, retest against the fixed baseline, or move to a higher-precision format if memory permits. |
| Throughput declines over time | Thermal throttling, an aggressive power mode, or memory pressure. | Record temperature and power settings, confirm cooling, and retest at the intended sustained workload rather than a short burst. |
| Results differ after rebuilding | Changed checkpoint revision, tokenizer, calibration set, tactic choices, or software image. | Compare the release record and calibration-data hash; rebuild from the same pinned inputs. |
Package a reproducible deployment
Store these artifacts together:
- Checkpoint identifier, revision, tokenizer revision, and license record.
- Quantization method and complete configuration, including the W4A16
four_over_sixdesignation where applicable. - Calibration-data hash and a description of its language, task mix, and sequence-length distribution.
- Exact export and engine-build commands or scripts, including any
--externalize-weightsuse. - TensorRT Edge-LLM or TensorRT-LLM, Model Optimizer, CUDA, JetPack, and operating-system versions.
- Device SKU, memory capacity, power mode, cooling configuration, and storage layout.
- Profiling logs for build, prefill, decode, peak memory, latency, throughput, power, and temperature.
- The fixed evaluation prompts and the acceptance thresholds used for quality.
This record lets you distinguish a model change from a runtime change and makes a 4 GB or 24 GB deployment repeatable after an operating-system or library update.
Quick Recap
Decision rule for the two device tiers
- For 4 GB: select the smallest supported checkpoint, prefer the published NVFP4 Lightning recipe when available, otherwise use a documented INT4 path, keep batch size one and context short, and require measured headroom after KV cache and runtime allocations.
- For 24 GB: begin with the same baseline, then spend verified headroom on context, precision, model size, or concurrency—never all four at once.
- For either tier: reject unsupported combinations, preserve calibration and release metadata, and publish measured memory and quality results for the exact board and software stack.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
