NVIDIA H100 NVL 94GB PCIe Gen5
900-21010-0020-000
FP32
67 TFLOPS
FP64
34 TFLOPS
PCIe
PCIe Gen5
VRAM
94 GB HBM3 with ECC
Memory Bandwidth: 3.9 TB/s
Memory Bandwidth: 3.9 TB/s
TDP
300-350W (configurable)
Warranty
3 Years Warranty
Why inferencing is at the heart of your AI strategy
In the world of artificial intelligence, there are two crucial phases: training and inferencing. While training involves neural networks “learning” from vast amounts of data, inferencing is the moment of truth. This is when the model applies its learned knowledge to new, real-world data to make predictions, generate images or answer complex questions in milliseconds.
In the age of generative AI and large language models (LLMs), the efficiency of inference has become a decisive competitive advantage. It is no longer just a question of whether AI works, but how fast, cost-efficient and scalable it is in live operation.
Generative AI & chatbots
Intelligent customer support systems that access company data using RAG (Retrieval Augmented Generation) technology and provide accurate answers in real time.
Edge AI & Robotics
In manufacturing, robots must process visual data immediately in order to react to obstacles. Local inferencing without cloud detours (edge computing) is essential here.
Medical diagnosis
AI models analyse CT images during the examination to immediately alert doctors to any abnormalities.
Neural Search & Recommendation
Modern search engines understand not only keywords, but also the intent behind them – made possible by lightning-fast vector searches in the inference process.
AI inferencing cards & platforms
Hardware requirements have changed dramatically. Whereas simple classifications used to suffice, today's applications demand extremely low latencies for real-time interactions.
NVIDIA L40
900-2G133-0010-000
CUDA Cores
18176
Tensor Cores
568
NVIDIA RT Cores
142
PCIe
PCI Express PCIe 4.0 x16
VRAM
48 GB GDDR6 with ECC
Memory Bandwidth: 864 GB/s
Memory Bandwidth: 864 GB/s
TDP
300 W
Warranty
3 Years Warranty
NVIDIA L40S
900-2G133-0080-000
CUDA Cores
18176
Tensor Cores
568
NVIDIA RT Cores
142
PCIe
PCI Express PCIe 4.0 x16
VRAM
48 GB GDDR6 with ECC
Memory Bandwidth: 864 GB/s
Memory Bandwidth: 864 GB/s
TDP
350 W
Warranty
3 Years Warranty
NVIDIA L4
900-2G193-0000-001
CUDA Cores
7680
Tensor Cores
240
NVIDIA RT Cores
60
PCIe
PCI Express Gen 4 x 16
VRAM
24 GB GDDR6 with ECC
Memory Bandwidth: 300 GB/s
Memory Bandwidth: 300 GB/s
TDP
72W
Warranty
3 Years Warranty
NVIDIA Jetson AGX Orin development kit 64GB
945-13730-0055-000
GPU
2048-core NVIDIA Ampere architecture GPU with 64 Tensor Cores
CPU
12-core Arm Cortex-A78AE v8.2 64-bit CPU
Deep-Learning Accelerator
2x NVDLA v2
Memory
64GB 256-bit LPDDR5
Storage
64GB eMMC 5.1
NVIDIA Jetson AGX Orin Industrial 64GB
900-13701- 0080-000
GPU
2048-core NVIDIA Ampere architecture GPU with 64 Tensor Cores
CPU
12-core Arm Cortex-A78AE v8.2 64-bit CPU
Deep-Learning Accelerator
2x NVDLA v2
Memory
64GB 256-bit LPDDR5
Storage
64GB eMMC 5.1
NVIDIA Jetson AGX Orin 64GB Module
900-13701-0050-000
GPU
2048-core NVIDIA Ampere architecture GPU with 64 Tensor Cores
CPU
12-core Arm Cortex-A78AE v8.2 64-bit CPU
Deep-Learning Accelerator
2x NVDLA v2
RAM
64GB 256-bit LPDDR5
Storage
64GB eMMC 5.1
NVIDIA Jetson Orin NX 16GB
900-13767-0000-000
GPU
1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores
CPU
8-core Arm Cortex-A78AE v8.2 64-bit CPU
Deep-Learning Accelerator
2x NVDLA v2
RAM
16GB 128-bit LPDDR5
NVIDIA Jetson Orin Nano 8GB Development Kit
945-13766-0005-000
GPU
1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores
CPU
6-core Arm Cortex-A78AE v8.2 64-bit CPU
RAM
8GB 128-bit LPDDR5
NVIDIA Jetson Orin Nano 8GB
900-13767-0030-000
GPU
1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores
CPU
6-core Arm Cortex-A78AE v8.2 64-bit CPU
RAM
8GB 128-bit LPDDR5
NVIDIA Jetson Orin NX 8GB
900-13767-0010-000
GPU
1024-core NVIDIA Ampere architecture GPU with 32 Tensor Cores
Deep-Learning Accelerator
1x NVDLA v2.0
CPU
6-core Arm Cortex-A78AE v8.2 64-bit CPU
RAM
8GB 128-bit LPDDR5
NVIDIA Jetson Orin Nano 4GB
900-13767-0040-000
GPU
512-core NVIDIA Ampere architecture GPU with 16 Tensor Cores
CPU
6-core Arm Cortex-A78AE v8.2 64-bit CPU
RAM
4GB 64-bit LPDDR5
NVIDIA Jetson AGX ORIN Module 32GB
900-13701-0040-000
GPU
1792-core NVIDIA Ampere architecture GPU with 56 tensor cores
Deep Learning Accelerator
2x NVDLA v2.0
CPU
8-core Arm Cortex-A78AE v8.2 64-bit CPU
RAM
32GB 256-bit LPDDR5
Storage
64GB eMMC 5.1
NVIDIA Jetson Nano Module
900-13448-0020-000
GPU
128-core NVIDIA Maxwell GPU
CPU
Quad-core ARM A57 CPU
RAM
4 GB 64-bit LPDDR4
Storage
16 GB eMMC 5.1
Optimum performance with NVIDIA
NVIDIA pursues a full-stack architecture approach that ensures that AI-supported applications can be operated with optimum cost efficiency. The result is faster results with reduced operating costs. NVIDIA AI Enterprise, an enterprise-grade inference platform, includes best-in-class software, reliable management, security and API stability to ensure performance and high availability.
Further questions about AI Inferencing?
Scalability is key
A successful AI project does not end with training. To maximise the ROI of your AI investments, you need an infrastructure that grows with your requirements. From single GPU workstations for development to multi-GPU server clusters, sysGen offers tailor-made solutions for every scenario.
FAQ: Frequently asked questions about AI inferencing
- What is the difference between AI training and AI inferencing?
During training, a model is ‘fed’ billions of data points in order to recognise patterns. This is extremely computationally intensive and takes days or weeks. Inferencing is the application of the finished model. The model receives a new input (e.g. a question) and immediately delivers a result. Inferencing must be fast and efficient, as it often takes place in real time.
- Why are special GPUs required for inferencing?
Although inferencing can theoretically run on CPUs, CPUs are often too slow for modern applications (such as LLMs or image recognition). GPUs such as the NVIDIA L4 or L40S are specialised in performing thousands of calculations simultaneously. This reduces latency (response time) and significantly lowers the cost per query.
- What does ‘latency’ mean in inferencing?
Latency is the time between input (e.g. ‘Write a poem’) and the start of output. For chatbots or autonomous driving, low latency is crucial to ensure that interaction appears natural and safety is guaranteed.
What is "Edge Inferencing"?
With edge inferencing, data processing takes place directly on the device (e.g. in a camera, robot or vehicle) rather than in a remote data centre. This saves bandwidth, protects privacy and eliminates delays caused by internet transmission.
- How can I optimise the costs of AI inferencing?
Optimisation takes place on two levels:
- Hardware: Choosing the right GPU (e.g. NVIDIA L40S instead of H100, if the memory capacity is sufficient).
- Software: Techniques such as quantisation (reducing computing accuracy without noticeable loss of quality) can double the speed and halve the memory requirements.
- What is RAG (Retrieval Augmented Generation) in the context of inferencing?
RAG is a method in which the AI model queries external, up-to-date information (e.g. your company database) during the inference step. This ensures that the AI does not provide outdated or fabricated answers, but instead accesses your specific facts.