GPU Access & Scheduling¶

GPU hardware¶

The PAIS development cluster has one NVIDIA L40S GPU with 48 GB VRAM, shared across all research groups via time-slicing.

Property	Value
GPU model	NVIDIA L40S
VRAM	48 GB
Sharing	Time-sliced ×4 (4 virtual slices)
Scheduler	KAI Scheduler (fair-share)

Time-slicing model¶

The GPU is time-sliced into 4 virtual slices. Each slice appears as a full GPU to workloads but shares time on the physical hardware. This means:

Up to 4 workloads can run concurrently
Each slice gets 12 GB of effective VRAM (48 GB ÷ 4)
GPU context switching is handled by the NVIDIA driver
Workloads do not need to be aware of slicing

Requesting a GPU in your notebook¶

In the notebook creation form, set GPU to 1. This requests one time-slice (12 GB effective VRAM).

Verify GPU access inside a notebook:

import torch
print(torch.cuda.is_available())          # True
print(torch.cuda.get_device_name(0))      # NVIDIA L40S
print(torch.cuda.get_device_properties(0).total_memory // 1e9, "GB")

KAI Scheduler allocates GPU time fairly across research groups. If your group has contributed GPU hardware, you receive priority access to your quota. Idle capacity flows to other groups.

When the GPU is fully utilised: - Jobs from groups with quota are scheduled first - Jobs from groups over their quota wait in queue - The Kueue admission controller manages the queue

Check your group's current GPU usage:

kubectl get resourcequota -n rg-compsci
kubectl describe clusterqueue rg-compsci-queue

Training job best practices¶

Checkpoint frequently — save to VAST storage every N epochs so a preempted job can resume
Use mixed precision — torch.autocast("cuda") halves VRAM usage with minimal accuracy loss
Release GPU when idle — stop or delete notebook servers when not training; idle servers still hold their GPU slice

Long-running training jobs¶

For jobs that take hours, use a Kubeflow Pipeline instead of a notebook — pipelines can run unattended and resume after preemption. See Kubeflow Pipelines.