Advanced module for AI DevOps/MLOps infrastructure engineer interns on the Nvidia ecosystem, covering hardware installation combined with software optimization to support large machine learning models. The NVIDIA ecosystem includes components such as Triton Inference Server, Cuda, Driver, Toolkit...
Knowledge requirements to be grasped after the internship
Deep System Knowledge: Linux Kernel management, hardware driver management on Ubuntu/Rocky Linux.
AI Core Knowledge: Concepts of AI Inference vs. AI Training.
Nvidia Stack: Clearly distinguish the roles of Nvidia Driver, CUDA Toolkit, NVIDIA Container Toolkit.
Execution sequence
Register GPU resources: Log in to the system https://dev.mdcgroup.vn, submit a request for access to the company's physical server cluster equipped with NVIDIA graphics cards (standalone machines with 24Gb VRAM are available).
Install NVIDIA Driver: Remove old open-source drivers, proceed with installing the NVIDIA Driver version compatible with the device code. Verify with nvidia-smi.
Install CUDA Toolkit & cuDNN: Install CUDA Toolkit compatible with the required model version, set up environment variables ($PATH, $LD_LIBRARY_PATH).
Install NVIDIA Container Toolkit: Configure Docker so that Docker can intervene and directly use GPU hardware resources from the physical machine (--gpus all).
Deploy Nvidia Triton Inference Server: Run Triton Server via Docker. Prepare a simple Model Repository (e.g., Onnx or TensorRT image classification model).
Test Inference Client: Write a Client script to send inference requests (send images/text) to Triton Server via gRPC or HTTP protocol and receive the returned results.
Result acceptance
Quiz: Complete the hardware & AI infrastructure test on the internal server.
Questions & Proposals: Propose a method to optimize GPU memory when running multiple models simultaneously on Triton Server (Dynamic Batching).
Report results: Send screenshots of nvidia-smi command results, successful Triton Server startup logs, and client-received results to the manager.
Weekend Discussion: Report installation progress, common Kernel/Driver conflict errors at the weekend meeting.