Advanced module for AI DevOps/MLOps infrastructure engineer interns in the Nvidia ecosystem, handling hardware installation combined with software optimization to serve large machine learning models.

Required knowledge to grasp

  • Deep System Knowledge: Linux Kernel management (Kernel modules), hardware driver management on Ubuntu/Rocky Linux.

  • AI Core Knowledge: Concepts of AI Inference vs AI Training.

  • Nvidia Stack: Clearly distinguish the roles of Nvidia Driver, CUDA Toolkit, and NVIDIA Container Toolkit.

Execution sequence

  1. Register GPU resources: Log in to the system https://dev.mdcgroup.vn, submit a request for access to the company's physical machine cluster equipped with NVIDIA graphics cards (independent machines with 24 Gb VRAM are available).

  2. Install NVIDIA Driver: Remove old open-source drivers, proceed with installing the proprietary NVIDIA Driver suitable for the device code. Verify using the command nvidia-smi.

  3. Install CUDA Toolkit & cuDNN: Install CUDA Toolkit suitable for the required model version, set up environment variables ($PATH, $LD_LIBRARY_PATH).

  4. Install NVIDIA Container Toolkit: Configure Docker so that Docker can directly access and use GPU hardware resources from the physical machine (--gpus all).

  5. Deploy Nvidia Triton Inference Server: Run Triton Server via Docker. Prepare a simple Model Repository (e.g., Onnx or TensorRT image classification model).

  6. Test Inference Client: Write a Client script to send inference requests (send images/text) to Triton Server via gRPC or HTTP protocol and receive the returned results.

Result acceptance

  • Multiple-choice test: Complete the hardware & AI infrastructure test on the internal server.

  • Questions & Suggestions: Propose a method to optimize GPU memory when running multiple models simultaneously on Triton Server (Dynamic Batching).

  • Report results: Send screenshots of the nvidia-smi command output, successful Triton Server startup logs, and client-received results to the manager.

  • Weekend discussion: Report installation progress, common Kernel/Driver conflict errors at the weekend meeting.