NVIDIA H100 NVL 94GB PCIe Gen5
900-21010-0020-000
GPUs
FP32
67 TFLOPS
FP64
34 TFLOPS
PCIe
PCIe Gen5
VRAM
94 GB HBM3 with ECC
Memory Bandwidth: 3.9 TB/s
Memory Bandwidth: 3.9 TB/s
TDP
300-350W (configurable)
Warranty
3 Years Warranty
Why inferencing is at the heart of your AI strategy
In the world of artificial intelligence, there are two crucial phases: training and inferencing. While training involves neural networks “learning” from vast amounts of data, inferencing is the moment of truth. This is when the model applies its learned knowledge to new, real-world data to make predictions, generate images or answer complex questions in milliseconds.
In the age of generative AI and large language models (LLMs), the efficiency of inference has become a decisive competitive advantage. It is no longer just a question of whether AI works, but how fast, cost-efficient and scalable it is in live operation.
Areas of application: Where AI inference is making a difference today
Generative AI & chatbots
Intelligent customer support systems that, thanks to RAG (Retrieval-Augmented Generation) technology, securely access a company’s own data and provide accurate answers in real time.
Edge AI & Robotics
In manufacturing, robots must process visual data instantly in order to react to obstacles within milliseconds. Local inference without the need for cloud processing (edge computing) is essential here.
Medical diagnosis
AI models analyse CT and MRI scans during the examination to immediately alert medical staff to any abnormalities.
Neural Search & Recommendation
Modern search engines understand not only fixed keywords, but also the underlying intent – made possible by lightning-fast vector searches during the inference process.
Which inference class is right for your application?
| Use Case | Typical Requirements | Suitable platforms |
|---|---|---|
LLM and RAG Applications in the Data Center | GPU memory, low token latency, high user throughput, secure API deployment | NVIDIA H100 NVL, L40S, larger multi-GPU systems, NVIDIA AI Enterprise |
Computer Vision and Video Analysis | Video decoding/encoding, multiple parallel streams, low latency, good energy efficiency | L4, L40S, Jetson Orin |
Generative Image and Multimodal Applications | Tensor performance, GPU memory, FP16/FP8/INT8, high throughput | NVIDIA L40S, H100 NVL, Larger Server GPUs |
Robotics and Autonomous Machines | Local processing, sensor I/O, deterministic response, low power consumption | Jetson AGX Orin, Jetson Orin NX, Jetson Orin Nano, and Jetson Thor for new designs |
Medical and Industrial Image Processing | Data protection, validatable pipelines, stable runtime, and, if necessary, edge operation | L4, L40/L40S, Jetson Industrial, certified systems |
Search, Recommendations, and RAG | Embeddings, vector search, database and model pipelines, parallel requests | L4, L40S, server GPU with compatible storage and networking |
AI Inference Cards & Platforms
Hardware requirements have changed dramatically. Whereas simple classifications used to suffice, today’s applications demand extremely low latencies for real-time interactions. Discover our certified NVIDIA solutions for data centres and the edge:
NVIDIA L40
900-2G133-0010-000
GPUs
CUDA Cores
18176
Tensor Cores
568
NVIDIA RT Cores
142
PCIe
PCI Express PCIe 4.0 x16
VRAM
48 GB GDDR6 with ECC
Memory Bandwidth: 864 GB/s
Memory Bandwidth: 864 GB/s
TDP
300 W
Warranty
3 Years Warranty
NVIDIA L40S
900-2G133-0080-000
GPUs
CUDA Cores
18176
Tensor Cores
568
NVIDIA RT Cores
142
PCIe
PCI Express PCIe 4.0 x16
VRAM
48 GB GDDR6 with ECC
Memory Bandwidth: 864 GB/s
Memory Bandwidth: 864 GB/s
TDP
350 W
Warranty
3 Years Warranty
NVIDIA L4
900-2G193-0000-001
GPUs
CUDA Cores
7680
Tensor Cores
240
NVIDIA RT Cores
60
PCIe
PCI Express Gen 4 x 16
VRAM
24 GB GDDR6 with ECC
Memory Bandwidth: 300 GB/s
Memory Bandwidth: 300 GB/s
TDP
72W
Warranty
3 Years Warranty
NVIDIA Jetson AGX Orin Industrial 64GB
900-13701- 0080-000
NVIDIA JETSON
GPU
2048-core NVIDIA Ampere architecture GPU with 64 Tensor Cores
CPU
12-core Arm Cortex-A78AE v8.2 64-bit CPU
Deep-Learning Accelerator
2x NVDLA v2
Memory
64GB 256-bit LPDDR5
Storage
64GB eMMC 5.1
NVIDIA Jetson AGX Orin 64GB Module
900-13701-0050-000
NVIDIA JETSON
GPU
2048-core NVIDIA Ampere architecture GPU with 64 Tensor Cores
CPU
12-core Arm Cortex-A78AE v8.2 64-bit CPU
Deep-Learning Accelerator
2x NVDLA v2
RAM
64GB 256-bit LPDDR5
Storage
64GB eMMC 5.1
NVIDIA Jetson Orin NX 16GB
900-13767-0000-000
NVIDIA JETSON
GPU
1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores
CPU
8-core Arm Cortex-A78AE v8.2 64-bit CPU
Deep-Learning Accelerator
2x NVDLA v2
RAM
16GB 128-bit LPDDR5
NVIDIA Jetson Orin Nano 8GB
900-13767-0030-000
NVIDIA JETSON
GPU
1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores
CPU
6-core Arm Cortex-A78AE v8.2 64-bit CPU
RAM
8GB 128-bit LPDDR5
NVIDIA Jetson Orin NX 8GB
900-13767-0010-000
NVIDIA JETSON
GPU
1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores
Deep-Learning Accelerator
1x NVDLA v2.0
CPU
6-core Arm Cortex-A78AE v8.2 64-bit CPU
RAM
8GB 128-bit LPDDR5
NVIDIA Jetson Orin Nano 4GB
900-13767-0040-000
NVIDIA JETSON
GPU
512-core NVIDIA Ampere architecture GPU with 16 Tensor Cores
CPU
6-core Arm Cortex-A78AE v8.2 64-bit CPU
RAM
4GB 64-bit LPDDR5
NVIDIA Jetson AGX ORIN Module 32GB
900-13701-0040-000
NVIDIA JETSON
GPU
1792-core NVIDIA Ampere architecture GPU with 56 tensor cores
Deep Learning Accelerator
2x NVDLA v2.0
CPU
8-core Arm Cortex-A78AE v8.2 64-bit CPU
RAM
32GB 256-bit LPDDR5
Storage
64GB eMMC 5.1
NVIDIA Jetson Nano Module
900-13448-0020-000
NVIDIA JETSON
GPU
128-core NVIDIA Maxwell GPU
CPU
Quad-core ARM A57 CPU
RAM
4 GB 64-bit LPDDR4
Storage
16 GB eMMC 5.1
Optimum performance with NVIDIA
NVIDIA takes a full-stack architectural approach that ensures AI-powered applications can be run with maximum cost-efficiency. The result is faster inference results whilst reducing operating costs.
NVIDIA AI Enterprise, an enterprise-ready inference platform, comprises best-in-class software, reliable management, enterprise-grade security and API stability to guarantee maximum performance and reliability in live operations.
Further questions about AI Inferencing?
Scalability is key
A successful AI project does not end with training. To maximise the ROI of your AI investments, you need an infrastructure that can scale flexibly to meet your requirements. From single-GPU workstations for development and local testing to highly scalable multi-GPU server clusters, sysGen offers bespoke solutions for every application scenario.
FAQ: Frequently asked questions about AI inferencing
- What is the difference between AI training and AI inferencing?
During training, model parameters are adjusted using vast amounts of data so that the neural network ‘learns’. During inference, a fully trained model uses new inputs to generate predictions, classifications or output in real time.
- What does latency mean in the context of AI applications?
Latency is the time interval between the input (prompt or sensor signal) and the relevant output. In the case of Large Language Models (LLMs), the metrics ‘time to first token’ (the time until the first word is output) and ‘inter-token latency’ are particularly crucial.
- Why are specialised GPUs needed for AI inference?
Modern AI models require matrix and tensor operations that lend themselves well to parallelisation. Specialised data centre GPUs and edge platforms offer enormous computing power, high VRAM bandwidth, dedicated tensor cores and video engines for this purpose.
- Is more VRAM (GPU memory) always automatically better?
It’s not a one-size-fits-all situation. Whilst more memory does allow larger models or longer contexts to be loaded, it does not in itself guarantee faster speeds. Memory bandwidth, tensor performance, software optimisation and the number of parallel requests must always be considered in relation to one another.
- Which GPU is best suited for LLM inference?
This depends largely on the model size, quantisation, context length and user throughput. For small to medium-sized models, the NVIDIA L4 or L40S are ideal, whilst computationally intensive enterprise applications benefit from the H100 NVL or multi-GPU systems
- What is the difference between the NVIDIA L40 and L40S graphics cards?
Although both GPUs feature 48 GB of GDDR6 memory and identical memory bandwidth, the L40 is heavily optimised for graphics, visualisation and mixed workloads (including video engines). The L40S, on the other hand, is specifically designed for AI inference and training, with optimised FP8 tensor performance.
- What is edge inference?
Edge inference refers to the local execution of AI models directly at the data source – for example, on machines, cameras, robots or autonomous vehicles. This helps to minimise latency, reduce bandwidth costs and decrease reliance on cloud systems.