Understanding Modern AI – Part 1: The Infrastructure Behind Large Language Models (LLMs)
Large Language Models (LLMs) such as those powering ChatGPT, Gemini, and Claude may appear almost magical from a user’s perspective.
Behind the scenes, however, they rely on massive and highly specialized IT infrastructures consisting of GPU-accelerated servers, Linux, high-speed networking, distributed storage, containers, and workload orchestration platforms such as Kubernetes.
In this article, we will take an infrastructure-focused look behind the scenes of modern LLM platforms. Starting with the physical compute and GPU layer, we will work our way through Linux, CUDA, containers, Kubernetes, distributed inference, networking, and storage to understand the infrastructure required to train and operate today’s large AI models.
The goal is to understand how the different infrastructure components work together to make modern AI services possible, particularly from the perspective of traditional IT infrastructure, virtualization, networking, and cloud computing.
- Understanding the Infrastructure Behind Large Language Models
- CPU vs. GPU – Why Large Language Models Require Specialized Hardware
- GPU Hardware in Modern AI Data Centers
- The AI Software Stack – Linux, GPU Drivers, CUDA, and AI Frameworks
- Linux as the Foundation of AI Infrastructure
- GPU Drivers – Connecting Linux to the Hardware
- NVIDIA CUDA – GPU Computing Beyond Graphics
- AI Frameworks – PyTorch and TensorFlow
- How the Software Layers Work Together
- GPU Monitoring and Troubleshooting on Linux
- From Linux Applications to Containerized AI Workloads
- Containerized AI Workloads – Docker, NVIDIA Container Toolkit, and Kubernetes
- Why Containers Are Used for AI Workloads
- How Containers Access NVIDIA GPUs
- Installing NVIDIA Container Toolkit on Ubuntu
- Running a GPU-Enabled Docker Container
- Running AI Applications Inside GPU Containers
- Kubernetes – Orchestrating GPU Workloads
- NVIDIA GPU Operator
- From GPU Containers to Distributed LLM Inference
- High-Performance Networking and Storage for AI Infrastructure
- LLM Inference Infrastructure – From Prompt to Response
- Cloud vs. On-Premises AI Infrastructure
- Conclusion – Putting Everything Together
- What's Next – The Understanding Modern AI Series
- Links
Understanding the Infrastructure Behind Large Language Models
Before diving into individual technologies, it is important to understand how the different infrastructure components of a modern Large Language Model (LLM) platform work together.
Unlike traditional applications, which primarily rely on general-purpose CPUs, modern LLM platforms depend heavily on GPU-accelerated computing, high-bandwidth memory, low-latency networking, and distributed computing architectures.
These platforms typically consist of multiple infrastructure layers, ranging from physical servers and specialized accelerators to operating systems, container runtimes, orchestration platforms, and model-serving frameworks.
The Infrastructure Stack of an LLM Platform
A simplified LLM infrastructure stack can be divided into the following layers:
| Layer | Components and Technologies |
|---|---|
| Application | ChatGPT, Gemini, Claude, custom AI applications |
| Model Serving | Inference engines, APIs, request routing, distributed inference |
| Orchestration | Kubernetes, container scheduling, GPU resource management |
| AI Software Stack | PyTorch, CUDA, ROCm, GPU drivers, communication libraries |
| Operating System | Linux, kernel, device drivers, system libraries |
| Compute | CPUs, NVIDIA/AMD GPUs, high-bandwidth memory (HBM) |
| Networking | NVLink, InfiniBand, high-speed Ethernet, RDMA |
| Storage | NVMe, distributed file systems, object storage |
| Physical Infrastructure | Servers, racks, power distribution, cooling |
These layers are not completely independent. For example, GPU drivers and CUDA provide access to specialized hardware, while frameworks such as PyTorch use these capabilities to execute the mathematical operations required by neural networks.
Container platforms such as Kubernetes can then distribute workloads across multiple GPU-equipped servers, while inference engines handle incoming requests and execute the models.
Not every LLM deployment requires all these components. A smaller open-source model might run on a single workstation, whereas training or operating a very large model can require hundreds or thousands of interconnected GPUs.
Training vs. Inference
One of the most important distinctions in LLM infrastructure is the difference between training and inference.
Training is the process of developing a model by processing large datasets and adjusting its internal parameters (weights). This involves computationally intensive operations, including forward and backward passes through the neural network.
Training large models often requires distributed GPU clusters, high-bandwidth interconnects, and storage systems capable of delivering enormous amounts of data.
Inference, on the other hand, is the process of using an already trained model to generate predictions or responses.
When a user submits a prompt to an LLM, the model processes the input and generates output tokens based on its learned parameters.
Although inference generally requires less computation per request than training a large model, operating an LLM service for millions of users introduces its own infrastructure challenges, particularly regarding GPU memory, latency, throughput, scalability, and availability.
| Characteristic | Training | Inference |
|---|---|---|
| Primary objective | Learn and update model weights | Generate responses using trained weights |
| GPU requirements | Often large distributed clusters | Single GPU to distributed clusters |
| Memory requirements | Weights, gradients, optimizer states, activations | Model weights, KV cache, activations |
| Networking | Intensive communication between GPUs and nodes | Depends on model distribution and serving architecture |
| Storage | Training datasets, checkpoints | Model weights, configuration, caches |
| Optimization focus | Training throughput and efficiency | Latency, throughput, and cost per token |
Both environments rely on similar underlying technologies, but their architectures and resource requirements can differ significantly.
In the following sections, we will examine these infrastructure components in more detail, starting with the fundamental difference between traditional CPU-based computing and GPU-accelerated computing.
CPU vs. GPU – Why Large Language Models Require Specialized Hardware
Traditional IT infrastructure has historically relied primarily on Central Processing Units (CPUs) to execute applications, operating systems, databases, and virtual machines.
Modern Large Language Models, however, perform enormous numbers of mathematical operations that can be executed in parallel. This makes Graphics Processing Units (GPUs) particularly well suited for AI workloads.
Although GPUs were originally developed to accelerate graphics rendering, their massively parallel architecture has made them essential for modern AI and high-performance computing (HPC).
The fundamental difference between CPUs and GPUs is latency versus throughput.
CPUs are optimized to execute individual instruction sequences with low latency, making them ideal for sequential and complex workloads.
GPUs, on the other hand, are designed to maximize throughput by executing thousands of mathematical operations in parallel. Consequently, GPUs can process large matrix and tensor operations significantly faster, provided the workload offers sufficient parallelism.
CPU Architecture – Optimized for General-Purpose Computing
A CPU is designed to execute a wide variety of instructions efficiently, including complex operations, conditional logic, and sequential workloads.
Modern server processors, such as AMD EPYC and Intel Xeon, can contain dozens or even hundreds of CPU cores, depending on the processor generation and model.
Each CPU core is relatively sophisticated and includes components such as execution units, registers, caches, and control logic.
CPUs are particularly effective at:
- Running operating systems, virtual machines, and traditional applications.
- Executing sequential workloads and complex control logic.
- Handling interrupts, system calls, and I/O operations.
- Managing tasks that involve frequent branching and unpredictable execution paths.
- Coordinating GPU workloads and managing system resources.
However, even high-end server CPUs have significantly fewer parallel arithmetic execution units than modern data center GPUs.
GPU Architecture – Optimized for Parallel Processing
Unlike CPUs, GPUs are designed to execute large numbers of similar mathematical operations simultaneously.
A modern GPU contains many processing units organized into groups, such as NVIDIA’s Streaming Multiprocessors (SMs).
Each SM contains execution resources that can process many threads concurrently.
For example, NVIDIA GPUs include:
- CUDA Cores: Execute general arithmetic operations, including floating-point and integer calculations.
- Tensor Cores: Specialized execution units designed to accelerate matrix multiplication and related operations commonly used in AI.
- High-Bandwidth Memory (HBM): Provides extremely high memory bandwidth for moving data between GPU memory and processing units.
- GPU Interconnects: Technologies such as NVIDIA NVLink enable high-speed communication between compatible GPUs.
This architecture is especially effective for the matrix and tensor operations used by neural networks.
Why LLMs Benefit from GPUs
Large Language Models are based on neural networks that repeatedly perform mathematical operations on large multidimensional arrays called tensors.
One of the most important operations is matrix multiplication.
For example, a simplified neural network operation can be represented as:
Y = X × W
Where:
- X represents input data or intermediate activations.
- W represents the model’s learned weights.
- Y represents the resulting output values.
In practice, LLMs execute enormous numbers of these operations across multiple neural network layers.
Many of the individual calculations involved in matrix multiplication can be performed simultaneously.
A CPU can execute these operations, but GPUs can often process them much more efficiently because they provide substantially more parallel arithmetic throughput and specialized matrix-processing hardware.
Modern GPUs also support reduced-precision numerical formats, such as FP16, BF16, FP8, and, on supported hardware, FP4-based formats. These can significantly improve performance and reduce memory requirements when appropriate for the workload.
CPU vs. GPU – Architecture Comparison
| Characteristic | CPU | GPU |
|---|---|---|
| Primary purpose | General-purpose computing | Massively parallel computing |
| Processing architecture | Relatively few sophisticated cores | Many parallel execution units |
| Strength | Sequential and complex workloads | Parallel mathematical workloads |
| Memory | System RAM with multi-level CPU caches | Dedicated GPU memory, often HBM in AI accelerators |
| Memory bandwidth | Generally lower | Very high on data center accelerators |
| AI acceleration | Vector instructions, matrix extensions on supported CPUs | Tensor Cores or equivalent matrix engines |
| Typical role in LLM platforms | Operating system, orchestration, preprocessing | Model training and inference |
It is important to understand that GPUs do not replace CPUs.
In a typical AI server, the CPU remains responsible for running Linux, managing system resources, handling networking and storage operations, and coordinating GPU workloads.
The GPU performs the computationally intensive mathematical operations required by the model.
GPU Memory – Why VRAM and HBM Matter
GPU computing performance alone is not sufficient to run a Large Language Model.
The model’s parameters must be available in memory during inference, and additional memory is required for intermediate computations and the Key-Value (KV) cache, which stores information used during attention processing.
For example, a hypothetical dense model containing 70 billion parameters requires approximately:
| Numerical precision | Approximate weight memory |
|---|---|
| FP32 (32-bit) | 280 GB |
| FP16 / BF16 (16-bit) | 140 GB |
| INT8 (8-bit) | 70 GB |
| 4-bit quantization | 35 GB |
These figures represent the approximate storage requirements for the model weights alone, using decimal units. Actual GPU memory consumption is higher because of additional runtime data, quantization metadata, and other overhead.
This explains why large models often require multiple GPUs or specialized accelerators with very large memory capacities.
Modern AI accelerators commonly use High-Bandwidth Memory (HBM), which provides substantially higher memory bandwidth than conventional system memory.
Memory bandwidth is particularly important during LLM inference, where generating individual tokens can involve repeatedly reading large amounts of model weight data.
CPUs and GPUs Work Together
A typical GPU-accelerated AI server contains one or more CPUs alongside several GPUs.
The CPUs run the operating system, manage applications, and coordinate workloads, while the GPUs execute the computationally intensive parts of the neural network.
Software frameworks such as PyTorch use GPU acceleration libraries, including NVIDIA CUDA or AMD ROCm, to execute operations on compatible accelerators.
For larger models, workloads can be distributed across multiple GPUs and even multiple physical servers.
This introduces additional infrastructure requirements, particularly high-speed GPU interconnects, low-latency networking, and distributed workload management.
We will explore these components in the following sections.
GPU Hardware in Modern AI Data Centers
As discussed in the previous section, GPUs are particularly well suited for the massively parallel mathematical operations required by Large Language Models.
However, running modern LLMs involves much more than installing a powerful graphics card in a server.
Large-scale AI platforms rely on specialized data center GPU accelerators, high-bandwidth memory, high-speed interconnects, and server architectures designed to operate multiple GPUs as a coordinated computing system.
Data Center GPUs – NVIDIA H100, H200, and Blackwell
Unlike consumer graphics cards, data center GPUs are designed for sustained computational workloads, large memory capacities, high memory bandwidth, and operation in multi-GPU environments.
NVIDIA is one of the major suppliers of AI accelerators, with several GPU generations widely used for training and inference.
Examples include:
- NVIDIA H100 (Hopper): An AI accelerator featuring fourth-generation Tensor Cores and HBM memory, widely used for large-scale training and inference.
- NVIDIA H200 (Hopper): Builds on the Hopper architecture with significantly more GPU memory and higher memory bandwidth than the H100.
- NVIDIA B200 (Blackwell): Introduces the Blackwell architecture with fifth-generation Tensor Cores, additional low-precision computing capabilities, and substantially increased memory capacity and bandwidth.
The following table compares selected specifications of the SXM-based accelerators.
| Specification | H100 SXM | H200 SXM | B200 SXM |
|---|---|---|---|
| Architecture | Hopper | Hopper | Blackwell |
| GPU memory | 80 GB HBM3 | 141 GB HBM3e | 180 GB HBM3e |
| Memory bandwidth | 3.35 TB/s | 4.8 TB/s | 8 TB/s |
| NVLink bandwidth | 900 GB/s | 900 GB/s | 1.8 TB/s |
| Tensor Core generation | 4th | 4th | 5th |
Specifications are approximate published maximums for the respective SXM configurations. NVLink bandwidth represents aggregate bidirectional bandwidth per GPU, not bandwidth to an individual peer.
Other manufacturers, including AMD with its Instinct accelerator family, provide alternative GPU computing platforms.
PCIe vs. SXM – How GPUs Are Installed in Servers
Data center GPUs are available in different physical configurations.
PCI Express (PCIe) accelerators are installed using standard PCIe expansion interfaces, similar in principle to conventional server expansion cards.
These configurations provide flexibility and can be integrated into compatible GPU servers.
SXM, on the other hand, is an NVIDIA module form factor designed for specialized GPU server platforms.
SXM-based systems can provide higher power budgets and support high-bandwidth GPU-to-GPU interconnects through NVIDIA NVLink and NVSwitch.
This makes them particularly attractive for large-scale AI workloads requiring intensive communication between GPUs.
Multi-GPU Servers – NVIDIA HGX and DGX
A single GPU is often insufficient to train or run very large LLMs.
Modern AI servers therefore combine multiple GPUs within one physical system.
NVIDIA provides two important platform families:
NVIDIA HGX
HGX is a GPU computing platform that server manufacturers integrate into their own systems. It combines multiple GPUs with high-speed interconnect technologies such as NVLink and NVSwitch.
For example, an HGX H100 configuration can contain eight H100 GPUs interconnected through NVSwitch.
NVIDIA DGX
DGX refers to NVIDIA’s integrated AI computing systems, combining GPUs, CPUs, memory, networking, storage, and a supported software stack.
A DGX system provides a more complete, validated AI infrastructure platform.
In simplified terms, HGX is a GPU computing platform for server manufacturers, while DGX is an integrated AI system delivered by NVIDIA.
NVLink and NVSwitch – Connecting Multiple GPUs
When an LLM is too large to fit into the memory of a single GPU, its parameters and computations can be distributed across multiple GPUs.
This requires efficient communication between the accelerators.
Although GPUs can communicate using PCIe, its bandwidth can become a limitation for workloads involving frequent transfers of large amounts of data.
NVIDIA developed NVLink to provide higher-bandwidth GPU interconnects.
NVSwitch extends this concept by providing a high-bandwidth switching fabric between multiple GPUs.
These technologies are particularly important for distributed model execution, where different GPUs must exchange intermediate results during training or inference.
However, connecting eight GPUs does not automatically create one large, transparent memory pool. The software framework must explicitly manage how model parameters and computations are distributed across the available devices.
Scaling Beyond a Single Server
Even a server containing eight high-performance GPUs may not provide sufficient computing power or memory capacity for the largest AI workloads.
In these cases, multiple GPU servers are connected to form a distributed computing cluster.
High-performance networks, including InfiniBand and RDMA-capable Ethernet, enable communication between the servers.
These networks must deliver high bandwidth and low latency because distributed AI workloads frequently exchange data between GPUs located on different physical machines.
For example, during distributed training, GPUs may need to synchronize gradients or exchange intermediate results across the cluster.
The performance of the network can therefore directly influence the overall efficiency of the training process.
Power, Cooling, and Data Center Requirements
High-performance GPU infrastructure introduces substantial power and cooling requirements.
A single modern AI accelerator can consume several hundred watts or more, depending on the GPU model and configuration.
A server containing eight GPUs can therefore require several kilowatts of electrical power before accounting for CPUs, memory, networking, and other components.
Large AI clusters consequently require specialized data center infrastructure, including:
- High-capacity power distribution.
- Advanced air-cooling or direct liquid-cooling systems.
- High-density server racks.
- Redundant network connectivity.
- Infrastructure monitoring and workload management.
These requirements illustrate why operating large-scale AI infrastructure is considerably more complex than deploying conventional application servers.
In the next section, we will examine the software environment that makes this specialized hardware usable, starting with Linux, GPU drivers, NVIDIA CUDA, and AI computing frameworks.
The AI Software Stack – Linux, GPU Drivers, CUDA, and AI Frameworks
Having powerful GPU hardware is only one part of running a Large Language Model. The operating system and software stack must be able to communicate with the accelerators, manage GPU memory, execute mathematical operations, and coordinate workloads across multiple devices.
Modern AI infrastructure therefore relies on several software layers, ranging from the Linux kernel and GPU drivers to specialized computing libraries and machine learning frameworks.
Linux as the Foundation of AI Infrastructure
Linux is widely used as the operating system for GPU-accelerated AI servers and large-scale computing clusters.
Popular distributions include Ubuntu Server, Red Hat Enterprise Linux (RHEL), Rocky Linux, and other enterprise Linux platforms.
Linux provides the fundamental services required to operate AI workloads, including:
- Hardware management: CPU scheduling, memory management, PCIe devices, and GPU driver integration.
- Networking: TCP/IP, high-speed Ethernet, InfiniBand, and RDMA support.
- Storage: Local NVMe devices, network file systems, and distributed storage.
- Process management: Execution and monitoring of AI applications and background services.
- Container infrastructure: Linux namespaces, control groups (cgroups), and container runtimes.
- Security: User permissions, access controls, kernel security mechanisms, and workload isolation.
Although GPU-accelerated AI applications can also run on Windows, Linux is particularly common in enterprise AI infrastructure and large-scale GPU clusters.
GPU Drivers – Connecting Linux to the Hardware
Before an application can use an NVIDIA GPU, the operating system requires a compatible NVIDIA driver.
The driver provides the software interface between the operating system, applications, and GPU hardware.
It is responsible for functions such as GPU initialization, device management, memory management, and execution of GPU workloads.
On a Linux server equipped with NVIDIA GPUs, the installed driver and detected accelerators can typically be inspected using:
nvidia-smi
The NVIDIA System Management Interface (NVIDIA-SMI) displays information such as:
- Detected NVIDIA GPUs and their models.
- Installed NVIDIA driver version.
- Supported CUDA driver API version.
- GPU memory usage.
- GPU utilization and temperature.
- Running processes using GPU resources.
It is important to distinguish between the CUDA version displayed by nvidia-smi and the CUDA Toolkit installed on the system.
The displayed version indicates the maximum CUDA driver API version supported by the installed driver, not necessarily the version of the CUDA Toolkit installed.
NVIDIA CUDA – GPU Computing Beyond Graphics
CUDA (Compute Unified Device Architecture) is NVIDIA’s parallel computing platform and programming model.
It enables software developers and frameworks to execute general-purpose computations on compatible NVIDIA GPUs rather than using them exclusively for graphics rendering.
Originally, GPUs were developed to accelerate graphics rendering, which inherently involves processing large numbers of pixels and vertices in parallel.
NVIDIA introduced CUDA (Compute Unified Device Architecture) in 2006, extending the use of GPUs beyond graphics rendering to general-purpose parallel computing, commonly referred to as GPGPU (General-Purpose Computing on Graphics Processing Units). This development laid an important foundation for GPU-accelerated scientific computing, machine learning, and modern Large Language Models.
Before CUDA, researchers were already using GPUs for non-graphics calculations, but they often had to work through graphics programming interfaces. That was cumbersome because mathematical problems had to be expressed in terms of graphics operations.
CUDA provides programming interfaces, runtime components, development tools, and access to GPU-accelerated computing libraries.
For example, CUDA allows applications to execute mathematical operations across thousands of GPU threads simultaneously.
Several important NVIDIA libraries are built around CUDA:
| Library | Purpose |
|---|---|
| CUDA Runtime | Application interface for GPU memory management and kernel execution |
| cuBLAS | GPU-accelerated linear algebra and matrix operations |
| cuDNN | Optimized operations for deep neural networks |
| NCCL | High-performance communication between multiple GPUs |
| cuFFT | GPU-accelerated Fast Fourier Transforms |
These libraries allow AI frameworks to use highly optimized GPU operations without requiring developers to implement every mathematical operation directly in CUDA.
CUDA is specific to NVIDIA GPUs. AMD provides an alternative GPU computing software platform called ROCm (Radeon Open Compute) for supported AMD accelerators.
What Were the Original CUDA Use Cases?
When NVIDIA introduced CUDA (Compute Unified Device Architecture) in 2006, its primary objective was not to accelerate modern AI or Large Language Models, but to make the parallel processing capabilities of GPUs accessible to general-purpose applications.
Before CUDA, researchers were already using GPUs for non-graphics calculations, commonly referred to as GPGPU (General-Purpose Computing on Graphics Processing Units). However, developers often had to use graphics APIs such as OpenGL or Direct3D and express mathematical calculations through graphics operations, shaders, and textures.
CUDA simplified this process by allowing developers to execute general-purpose parallel programs directly on compatible NVIDIA GPUs.
Early applications and target workloads included:
| Use Case | Application |
|---|---|
| Scientific simulations | Fluid dynamics, weather modeling, and physical simulations |
| Oil and gas exploration | Seismic data processing and geological modeling |
| Medical imaging | CT reconstruction and image processing |
| Molecular dynamics | Simulating interactions between atoms and molecules |
| Financial modeling | Option pricing and Monte Carlo simulations |
| Signal processing | Mathematical transformations and large-scale data processing |
These workloads shared an important characteristic: they required enormous numbers of mathematical calculations that could be executed independently and in parallel.
Although machine learning already existed long before CUDA, the modern deep-learning revolution gained significant momentum in 2012 with AlexNet, which demonstrated the effectiveness of GPU-accelerated neural network training.
CUDA subsequently became one of the foundational technologies behind modern GPU-accelerated AI and Large Language Models.
In short, NVIDIA did not invent parallel computing with CUDA. Instead, CUDA made the existing parallel computing capabilities of GPUs significantly easier to access and program for applications beyond graphics rendering.
Example: GPU-Accelerated Seismic Processing in Oil and Gas Exploration
One of the early applications of general-purpose GPU computing was seismic data processing in the oil and gas industry.
To locate potential underground oil and gas reservoirs, exploration companies use seismic surveys. These involve generating controlled seismic waves that travel through the Earth’s subsurface and are reflected by geological structures.
Thousands of sensors, called geophones on land or hydrophones in marine surveys, record the returning seismic signals.
The collected measurements are then processed to reconstruct images of underground geological formations.
One particularly computationally intensive technique is seismic migration, which transforms recorded seismic reflections into images that more accurately represent the positions of subsurface structures.
This involves enormous numbers of mathematical calculations across large datasets.
Traditionally, such workloads were processed using high-performance computing (HPC) systems consisting of many CPUs working in parallel.
However, many seismic processing algorithms are well suited for GPU acceleration because large portions of their mathematical calculations can be executed simultaneously.
Early GPU computing research demonstrated that computationally intensive seismic algorithms could be accelerated using graphics processors, reducing execution times compared with conventional CPU-based implementations.
With the introduction of CUDA, developers gained a more straightforward way to implement these algorithms directly on NVIDIA GPUs without having to express their computations through graphics APIs.
This made GPUs increasingly attractive for scientific and industrial computing applications.
The same fundamental principle applies to modern Large Language Models: computationally intensive mathematical operations are distributed across many GPU execution units, allowing large workloads to be processed much more efficiently than with conventional CPU-only execution.
AI Frameworks – PyTorch and TensorFlow
While CUDA provides access to GPU computing capabilities, developers generally do not write complete neural networks directly using CUDA.
Instead, they use machine learning frameworks such as PyTorch or TensorFlow.
These frameworks provide higher-level interfaces for building, training, and executing neural networks.
For example, PyTorch can automatically execute supported tensor operations on an NVIDIA GPU using its CUDA backend.
A simplified PyTorch example:
import torch
device = torch.device("cuda")
a = torch.randn(4096, 4096, device=device)
b = torch.randn(4096, 4096, device=device)
c = torch.matmul(a, b)
print(c.device)
In this example, PyTorch creates two matrices in GPU memory and performs matrix multiplication using the GPU.
The framework handles the underlying GPU execution through its CUDA integration and optimized libraries.
The example requires a compatible NVIDIA GPU, driver, and CUDA-enabled PyTorch installation.
How the Software Layers Work Together
When an AI application performs a GPU-accelerated operation, several software components cooperate.
For example, a PyTorch application executing matrix multiplication follows a simplified execution path:
- The Python application calls a PyTorch operation.
- PyTorch selects the appropriate GPU execution backend.
- The backend invokes optimized GPU kernels or libraries such as cuBLAS.
- CUDA runtime and driver components coordinate GPU execution.
- The NVIDIA driver communicates with the GPU hardware.
- The GPU performs the calculations and stores the results in GPU memory.
Not every operation follows exactly the same software path. Some use custom kernels, compiler-generated code, or other specialized execution mechanisms.
Nevertheless, this illustrates how high-level AI applications ultimately execute mathematical operations on physical GPU hardware.
GPU Monitoring and Troubleshooting on Linux
GPU monitoring is an important operational task when running AI workloads.
The following commands provide a starting point for inspecting NVIDIA GPU hardware and software.
Display detected GPUs and driver information:
nvidia-smi
Continuously monitor GPU utilization and memory usage:
watch -n 1 nvidia-smi
List NVIDIA GPU devices detected by PCIe:
lspci | grep -i nvidia
Display detailed GPU information:
nvidia-smi -q
Display the installed CUDA compiler version, if the CUDA Toolkit is installed:
nvcc --version
The nvcc command is part of the CUDA Toolkit and may not be available on inference servers or container hosts that only require the NVIDIA driver.
Check whether PyTorch can access an NVIDIA GPU:
python3 -c "import torch; print(torch.cuda.is_available())"
A result of True indicates that the installed PyTorch environment can access a compatible CUDA device.
From Linux Applications to Containerized AI Workloads
Running AI applications directly on Linux is possible, but managing dependencies across multiple servers can become increasingly complex.
Different AI applications may require different versions of Python, PyTorch, CUDA libraries, and other software components.
Containers help address this challenge by packaging applications and their dependencies into reproducible execution environments.
For NVIDIA GPU workloads, the NVIDIA Container Toolkit enables compatible container runtimes to expose GPU devices and required driver interfaces to containers.
This allows AI applications to run inside containers while using the physical GPUs installed in the host system.
At larger scales, Kubernetes can orchestrate these containerized workloads across clusters of GPU-equipped servers.
In the next section, we will examine containerized AI workloads, NVIDIA Container Toolkit, and GPU orchestration with Kubernetes.
Containerized AI Workloads – Docker, NVIDIA Container Toolkit, and Kubernetes
As discussed in the previous section, AI applications depend on several software components, including GPU drivers, CUDA libraries, machine learning frameworks, and Python environments.
Installing and maintaining these dependencies directly on multiple Linux servers can become increasingly complex, particularly when different AI applications require different versions of their software libraries.
Containerization provides a more flexible approach by packaging applications and their dependencies into portable, reproducible environments.
However, unlike traditional application containers, GPU-accelerated workloads require additional integration with the underlying hardware.
Why Containers Are Used for AI Workloads
Containers package an application together with its required libraries, frameworks, and runtime dependencies.
For example, a container image for an LLM inference service might include:
- Python and the required application dependencies.
- PyTorch or another machine learning framework.
- CUDA runtime libraries and GPU-accelerated computing libraries.
- An inference engine such as vLLM or NVIDIA Triton Inference Server.
- Application configuration and startup scripts.
The container can then be deployed on compatible Linux hosts without manually installing the entire application software stack on each server.
This approach provides several advantages, including reproducible deployments, simplified dependency management, version control, and integration with orchestration platforms such as Kubernetes.
How Containers Access NVIDIA GPUs
Unlike conventional applications that only require CPU and system memory resources, GPU-accelerated containers must communicate with physical GPU devices installed in the host server.
The NVIDIA GPU driver is normally installed on the Linux host rather than independently inside each container.
Containers use the host’s GPU driver interfaces while providing their own compatible user-space application libraries.
To enable GPU access, NVIDIA provides the NVIDIA Container Toolkit.
The toolkit integrates with supported container runtimes and enables containers to access NVIDIA GPU devices and the required driver components.
A simplified architecture looks like this:
Physical NVIDIA GPU → Linux Host and NVIDIA Driver → Container Runtime with NVIDIA Container Toolkit → GPU-Enabled Container → AI Application
The container does not emulate the GPU. Instead, the application executes computations directly on the physical accelerator through the exposed device and driver interfaces.
Unlike full hardware emulation, modern hypervisors such as VMware ESXi or Microsoft Hyper-V use hardware-assisted virtualization (
Intel VT-x/AMD-V) to execute most guest CPU instructions directly on the physical processor.Memory virtualization uses technologies such as Intel EPT or AMD NPT to map guest memory to physical RAM, while virtual devices such as NICs and storage controllers are typically implemented through software-based device emulation or optimized paravirtualized drivers (e.g., VMXNET3 or VirtIO).
An important distinction: Virtualization does not necessarily mean emulation. Modern VMs execute most CPU instructions natively, whereas device emulation reproduces hardware behavior in software.
In virtualized environments, hardware resources can be presented to virtual machines using different techniques, including hardware emulation, paravirtualization, and hardware-assisted virtualization, each providing a different approach to accessing the underlying physical hardware.
- Emulation: Software reproduces the behavior of a hardware device, such as an emulated Intel E1000 NIC. The guest operating system communicates with it using a conventional hardware driver.
- Paravirtualization: The guest uses virtualization-aware drivers, such as VMware VMXNET3 or VirtIO, to communicate efficiently with the hypervisor. This eliminates the need to emulate all the behavior of a traditional physical device.
- Hardware-assisted virtualization: The guest executes most CPU instructions directly on the physical CPU using
Intel VT-xorAMD-V.
Emulation makes the guest believe it is communicating with conventional physical hardware, whereas paravirtualization provides an optimized interface specifically designed for communication between the guest and the hypervisor.
Installing NVIDIA Container Toolkit on Ubuntu
The following example assumes an Ubuntu Server with a supported NVIDIA GPU, a compatible NVIDIA driver, and Docker Engine already installed.
First, verify that the GPU is detected by the host:
nvidia-smi
Next, configure the official NVIDIA Container Toolkit package repository:
sudo apt-get update sudo apt-get install -y curl gnupg
Download and install the repository signing key:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
Add the NVIDIA Container Toolkit repository:
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
Install the toolkit:
sudo apt-get update sudo apt-get install -y nvidia-container-toolkit
Configure Docker to use the NVIDIA container runtime integration:
sudo nvidia-ctk runtime configure --runtime=docker
Restart Docker:
sudo systemctl restart docker
The NVIDIA Container Toolkit is now configured for Docker GPU workloads.
Running a GPU-Enabled Docker Container
Once the NVIDIA Container Toolkit is installed, GPU access can be tested using a CUDA container.
For example:
sudo docker run --rm --gpus all ubuntu nvidia-smi
This command launches a temporary Ubuntu container with access to all GPUs exposed by the host.
The nvidia-smi utility is made available through the NVIDIA runtime integration when the required driver capabilities are enabled.
The command should display the GPUs available to the container, including their driver information, memory usage, and utilization.
The important Docker parameter is:
--gpus all
It instructs Docker to make all available NVIDIA GPUs accessible to the container.
Alternatively, individual GPUs can be selected using device identifiers.
For production workloads, access should generally be restricted to the GPU devices actually required by the application.
Running AI Applications Inside GPU Containers
Once GPU access is available, containerized applications can execute CUDA workloads using frameworks such as PyTorch.
For example, an AI application running inside a container can use:
import torch print(torch.cuda.is_available()) print(torch.cuda.get_device_name(0))
If the container has access to a compatible NVIDIA GPU and the required software libraries, PyTorch can execute tensor operations directly on the accelerator.
The same principle applies to containerized LLM inference engines, which load model weights into GPU memory and execute inference requests using the available accelerators.
Kubernetes – Orchestrating GPU Workloads
While Docker can run GPU-enabled containers on individual servers, large-scale AI environments often require workloads to be distributed across multiple GPU-equipped machines.
Kubernetes provides an orchestration platform for deploying, scheduling, scaling, and managing containerized applications across a cluster.
However, Kubernetes does not automatically discover and allocate NVIDIA GPUs using only its standard CPU and memory resource management.
Additional components are required, such as the NVIDIA Kubernetes Device Plugin.
The device plugin advertises GPU resources to Kubernetes, allowing workloads to request them through resource specifications.
For example, a Kubernetes Pod can request one NVIDIA GPU:
apiVersion: v1
kind: Pod
metadata:
name: gpu-test
spec:
restartPolicy: Never
containers:
- name: cuda
image: ubuntu:24.04
command: ["nvidia-smi"]
resources:
limits:
nvidia.com/gpu: 1
The resource definition:
nvidia.com/gpu: 1
instructs Kubernetes to schedule the Pod on a node with an available NVIDIA GPU resource.
The example assumes the node has the required NVIDIA driver, container runtime integration, and device plugin configured.
For a more detailed introduction to Kubernetes architecture and its fundamental concepts, including the control plane, worker nodes, Pods, Deployments, Services, and Namespaces, refer to my article below. This article is part of an ongoing series covering the deployment and operation of Kubernetes workloads in Microsoft Azure.
NVIDIA GPU Operator
Managing GPU drivers, container runtime integration, device plugins, and monitoring components across many Kubernetes nodes can become complicated.
NVIDIA provides the GPU Operator to automate much of this lifecycle management.
Depending on its configuration, the GPU Operator can manage components such as:
- NVIDIA GPU drivers.
- NVIDIA Container Toolkit.
- Kubernetes GPU Device Plugin.
- GPU Feature Discovery.
- NVIDIA Data Center GPU Manager (DCGM) monitoring components.
The GPU Operator simplifies the deployment and maintenance of GPU infrastructure in Kubernetes environments.
However, the exact installation requirements and supported configurations depend on the Kubernetes distribution, operating system, GPU hardware, and driver versions.
From GPU Containers to Distributed LLM Inference
Containerization and Kubernetes provide the foundation for operating AI workloads across multiple servers.
However, Kubernetes alone does not automatically distribute an LLM across multiple GPUs or coordinate the mathematical operations required for inference.
These functions are handled by specialized AI frameworks and inference engines, such as vLLM, TensorRT-LLM, and other distributed inference solutions.
Kubernetes manages the deployment and scheduling of the containers, while the inference framework manages model execution and, where supported, distribution across multiple GPUs.
This distinction is important because container orchestration and distributed model execution are separate responsibilities.
In the following section, we will examine how these technologies work together to provide scalable LLM inference services.
High-Performance Networking and Storage for AI Infrastructure
Modern AI infrastructure requires more than powerful GPUs. When training Large Language Models (LLMs) across multiple servers, network bandwidth, latency, and storage performance become critical to overall performance.
High-Performance Networking
Distributed AI workloads require GPUs to exchange model parameters, gradients, and intermediate results efficiently.
Common technologies include:
- InfiniBand: High-bandwidth, low-latency networking commonly used in HPC and large AI clusters.
- High-Speed Ethernet: 100/200/400/800 GbE networking, often combined with RDMA over Converged Ethernet (RoCE).
- RDMA (Remote Direct Memory Access): Enables direct memory transfers between servers with minimal CPU involvement.
- NVIDIA GPUDirect RDMA: Allows compatible network adapters to access GPU memory directly, reducing unnecessary data copies.
- NVIDIA NCCL: Optimizes collective GPU communication across technologies such as NVLink, InfiniBand, and RoCE.
While NVLink and NVSwitch primarily connect GPUs within supported systems, InfiniBand and Ethernet provide connectivity between GPU servers.
High-Performance Storage
AI workloads require fast access to training datasets, model weights, and checkpoints.
Common storage technologies include:
- NVMe SSDs: High-performance local storage for datasets, model files, and temporary caching.
- Parallel and distributed file systems: Technologies such as Lustre, IBM Storage Scale, and WEKA provide scalable shared storage for large AI clusters.
- Object storage: Services such as Azure Blob Storage and Amazon S3 provide durable storage for datasets, model artifacts, and checkpoints.
During training, checkpoints periodically save model parameters and training states so that processing can resume after interruptions or failures.
Ultimately, scalable AI infrastructure requires a balanced combination of GPU computing power, high-speed networking, and storage throughput. Even powerful GPUs can become bottlenecked when data cannot be delivered or exchanged quickly enough.
LLM Inference Infrastructure – From Prompt to Response
Inference refers to the process of using an already trained AI model to process new input and generate predictions or responses. In the context of Large Language Models (LLMs), inference occurs whenever a user submits a prompt and the model generates an answer.
Unlike training, which adjusts the model’s internal parameters (weights) using training data, inference uses the existing model weights without modifying them.
For example, when you ask ChatGPT a question, the model performs inference to generate its response. This requires computational resources, typically GPUs, but generally far less processing power per request than training a large model from scratch.
In simple terms: Training is where the model learns; inference is where it applies what it has learned.
After a Large Language Model (LLM) has been trained, it can be deployed for inference, the process of generating responses to user requests using the model’s existing parameters.
Unlike training, inference does not normally modify the model weights. Instead, it uses the trained model to predict and generate output tokens.
A token can represent:
- An entire word
- Part of a word
- A punctuation mark
- A space or another text fragment
A token is a small unit of text that a language model processes. Depending on the tokenizer, it can represent a complete word, part of a word, punctuation, or another text fragment.
And the important distinction: Tokens are not necessarily words. That’s why providers charge per token rather than per word.
Unlike complete words, tokens can also represent word fragments (e.g., splitting unbelievable into smaller pieces), punctuation marks (such as
!or?), parts of numbers, or text fragments containing spaces. Consequently, the number of tokens processed by an LLM can differ significantly from the number of words in a prompt or response.
How an LLM Processes a Request
A typical LLM inference workflow consists of the following steps:
- User Prompt: A user submits a request through a web interface, application, or API.
- API Gateway: The request is authenticated, routed, and forwarded to an available inference service.
- Tokenization: The input text is converted into tokens that the model can process.
- GPU Processing: The inference engine loads or accesses the model weights and executes the required calculations on GPUs.
- Token Generation: The model generates output tokens sequentially, using previously processed tokens as context.
- Response: The generated tokens are converted back into readable text and returned to the user, often through streaming.
Inference Engines and GPU Memory
Specialized inference engines such as vLLM, NVIDIA Triton Inference Server, and Hugging Face Text Generation Inference (TGI) help optimize model execution, request scheduling, and GPU resource utilization.
An important factor is GPU memory (VRAM), which must accommodate model weights and runtime data, including the KV cache used to store attention-related information from previously processed tokens.
Techniques such as quantization, batching, and optimized attention implementations help reduce memory requirements and improve inference performance.
Scaling LLM Inference
In production environments, inference services are commonly deployed as containers and orchestrated using Kubernetes.
Multiple inference instances can process requests in parallel, while load balancers distribute incoming traffic across available services.
For models too large to fit on a single GPU, techniques such as tensor parallelism or pipeline parallelism can distribute model execution across multiple GPUs.
Ultimately, efficient LLM inference depends on balancing GPU compute capacity, available VRAM, response latency, and the number of concurrent requests.
In production environments, LLM inference infrastructure is operated both by major AI providers such as OpenAI, Google, Anthropic, and DeepSeek, and by enterprises hosting their own AI models.
While large AI providers operate highly specialized infrastructure at massive scale, enterprises can deploy pretrained or customized open-weight models on their own GPU servers or cloud platforms.
In these environments, inference services are frequently containerized and can be orchestrated using Kubernetes, providing scalable, API-accessible AI capabilities for internal applications.
Cloud vs. On-Premises AI Infrastructure
Organizations can deploy AI infrastructure either in public cloud environments, on their own physical hardware, or through a combination of both.
Cloud-Based AI Infrastructure
Cloud providers such as Microsoft Azure, AWS, and Google Cloud offer GPU-accelerated virtual machines, managed Kubernetes platforms, and AI services.
Advantages:
- On-demand access to powerful GPU resources.
- Scalability without purchasing physical hardware.
- Managed infrastructure and AI services.
- Faster deployment and experimentation.
Challenges: Ongoing GPU costs, resource availability, data sovereignty requirements, and dependency on cloud providers.
On-Premises AI Infrastructure
Organizations can also operate AI workloads on dedicated GPU servers within their own data centers, using Linux, NVIDIA CUDA, containers, and Kubernetes.
Advantages:
- Greater control over hardware, data, and infrastructure.
- Predictable hardware capacity and potentially lower long-term costs at high utilization.
- Suitable for workloads with strict data residency or isolation requirements.
Challenges: High initial hardware investment, power and cooling requirements, maintenance, and limited scalability compared to public cloud resources.
Hybrid AI Infrastructure
A hybrid approach combines both environments. For example, organizations might use cloud GPU resources for occasional training workloads while hosting inference services on-premises to maintain control over sensitive corporate data.
Ultimately, the choice depends on cost, scalability, performance, security, and regulatory requirements.
Conclusion – Putting Everything Together
Modern Large Language Models (LLMs) rely on much more than sophisticated AI algorithms. Behind every prompt and generated response is a complex infrastructure combining high-performance GPUs, Linux operating systems, CUDA, containerization, networking, and storage technologies.
Throughout this article, we explored how these components work together, from GPU acceleration and distributed computing to Kubernetes orchestration and the infrastructure required for model training and inference.
We also examined the differences between cloud-based, on-premises, and hybrid AI infrastructure, including how enterprises can operate their own AI platforms using pretrained models.
Understanding these underlying technologies provides the foundation for exploring how LLMs actually process information.
What’s Next – The Understanding Modern AI Series
This article is the first installment of our ongoing Understanding Modern AI series, in which we explore the technologies behind modern artificial intelligence, from fundamental concepts and infrastructure architecture to practical deployments.
Coming Soon: The following articles are planned as part of my ongoing Understanding Modern AI series:
Part 2: How Large Language Models Actually Work – From Tokens to Transformers
We move inside the model to understand tokenization, embeddings, neural networks, Transformer architectures, attention mechanisms, and how LLMs generate responses.
Part 3: Understanding Agentic AI
We explore how AI agents extend traditional LLM capabilities through reasoning, planning, memory, tool calling, and interaction with external systems to perform increasingly complex tasks and automated workflows.
Part 4: Building Our AI Lab with a Two-Node Ubuntu Server and Kubernetes Cluster
We move from theory to practice by repurposing two Dell XPS 15 notebooks to build a self-hosted AI infrastructure lab.
Using Ubuntu Server, upstream Kubernetes, and NVIDIA GeForce RTX GPU acceleration, we will configure a two-node Kubernetes cluster, enable GPU support for containerized workloads, deploy locally hosted LLMs, and experiment with inference, GPU scheduling, and eventually AI agents.
Beyond Part 4 – An Ongoing Journey
The series will not end with our initial AI lab deployment. Future articles will build on this foundation as we explore additional AI technologies, infrastructure architectures, model deployment strategies, distributed workloads, and practical AI applications.
Our objective is to continuously expand our understanding of modern AI by combining technical fundamentals, real-world infrastructure, and hands-on experimentation.
Links
CUDA Programming Guide
https://docs.nvidia.com/cuda/cuda-programming-guideNVIDIA Container Toolkit
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latestNVIDIA – NCCL Documentation
https://docs.nvidia.com/deeplearning/ncclKubernetes – Concepts
https://kubernetes.io/docs/concepts/PyTorch documentation
https://docs.pytorch.org/docs/2.15/index.htmlvLLM – Official Documentation
https://docs.vllm.ai/en/latestNVIDIA GPU Gems 3 – Imaging Earth’s Subsurface Using CUDA
https://developer.nvidia.com/gpugems/gpugems3/part-vi-gpu-computing/chapter-38-imaging-earths-subsurface-using-cuda
Tags In
Related Posts
Latest posts
Understanding Modern AI – Part 1: The Infrastructure Behind Large Language Models (LLMs)
Follow me on LinkedIn
