Skip to content

Chapter 04: Resource Scheduling, Custom Constraints & Placement Groups

  • Topic Focus • Hardware Resource Demands, Custom Resource Tags, Placement Groups, and Gang Scheduling
  • Interactive Challenges • 6 Hands-on Exercises
  • Launch Playground in Wasm →

1. Architectural Overview & Control Plane Mechanics

Ray's distributed scheduler uses logical resource requirements (num_cpus, num_gpus, resources={"custom_accelerator": 1}) to match tasks and actors to available node capacity across the cluster.

flowchart TD
    Req["Placement Group Request<br/><code>bundles=[{GPU:1}, {GPU:1}], strategy='STRICT_SPREAD'</code>"] -->|"1. Submit Reservation"| GCS["GCS Resource Manager<br/>(Global Coordination)"]
    GCS -->|"2. Atomic Gang Lock"| Node1["Node 1 (Host A)<br/>Bundle 0: 1 GPU"]
    GCS -->|"2. Atomic Gang Lock"| Node2["Node 2 (Host B)<br/>Bundle 1: 1 GPU"]
    Node1 -.->|"3. Schedule Worker"| Actor1["Worker Actor Rank 0"]
    Node2 -.->|"3. Schedule Worker"| Actor2["Worker Actor Rank 1"]

    style Req fill:#1e293b,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
    style GCS fill:#1e293b,stroke:#f59e0b,stroke-width:2px,color:#f8fafc
    style Node1 fill:#0f172a,stroke:#34d399,stroke-width:2px,color:#f8fafc
    style Node2 fill:#0f172a,stroke:#06b6d4,stroke-width:2px,color:#f8fafc
    style Actor1 fill:#1e1e38,stroke:#c084fc,stroke-width:1px,color:#f8fafc
    style Actor2 fill:#1e1e38,stroke:#c084fc,stroke-width:1px,color:#f8fafc

Diagram Walkthrough & Core Concepts: - Atomic Gang Scheduling: Placement groups guarantee all-or-nothing resource reservation across physical cluster hosts before workers are dispatched. - Topology Control Strategies: STRICT_SPREAD enforces anti-affinity across separate physical machines for fault tolerance, while STRICT_PACK guarantees co-location on the same machine for ultra-fast NVLink communication. - Targeted Bundle Binding: Workers and actors bind to specific bundle indices within the placement group, ensuring predictable CPU and GPU resource isolation.

2. Annotated Python Code Anatomy & API Reference

import ray
from ray.util.placement_group import placement_group
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy

# 1. Define atomic resource bundles for distributed training
pg = placement_group(
    bundles=[{"CPU": 2, "GPU": 1}, {"CPU": 2, "GPU": 1}],
    strategy="STRICT_SPREAD"
)

# 2. Block until the gang-scheduling reservation is granted
ray.get(pg.ready())

# 3. Schedule actors onto specific reserved bundles
@ray.remote(num_cpus=2, num_gpus=1)
class DistributedWorker:
    def __init__(self, rank: int):
        self.rank = rank

    def run_training_step(self) -> str:
        return f"Worker {self.rank} completed step on dedicated GPU"

workers = [
    DistributedWorker.options(
        scheduling_strategy=PlacementGroupSchedulingStrategy(
            placement_group=pg,
            placement_group_bundle_index=idx
        )
    ).remote(idx)
    for idx in range(2)
]

Key API Parameter Reference

  • @ray.remote(num_cpus=N, num_gpus=M, resources={...}): Resource demands.
  • placement_group(bundles, strategy): Allocates atomic multi-node resource reservations. Strategies: STRICT_SPREAD, STRICT_PACK, PACK, SPREAD.
  • PlacementGroupSchedulingStrategy: Binds a task or actor to an allocated bundle.
  • ray.util.remove_placement_group(pg): Releases reserved capacity back to the cluster pool.

3. Production Best Practices & Hardening Guidelines

  1. Use Fractional GPUs for Inference: Declare num_gpus=0.25 for lightweight inference tasks to multiplex up to 4 workers onto a single physical GPU.
  2. Always Check pg.ready() with Timeout: When allocating placement groups in dynamic clusters, wait for pg.ready() before scheduling dependent workloads.
  3. Release Unused Placement Groups: Always call remove_placement_group() in finally: blocks to prevent resource leaks.
  4. Prefer STRICT_PACK for All-Reduce: Co-locate distributed workers on NVLink-connected multi-GPU nodes with STRICT_PACK for maximum communication bandwidth.
  5. Use Custom Resources for Hardware Affinity: Tag specialized nodes with custom resources (e.g. {"h100": 1, "fpga": 1}).

4. Troubleshooting & Diagnostic Workflows

  1. Placement Group Pending Indefinitely:
  2. Symptom: Cluster hangs on ray.get(pg.ready()).
  3. Fix: Verify total bundle requirements do not exceed total cluster node resources; check for autoscaler node provisioning delays.
  4. Fractional GPU CUDA OOM:
  5. Symptom: Out of memory when multiple workers share a single GPU.
  6. Fix: Set PyTorch memory allocation caps (torch.cuda.set_per_process_memory_fraction()).
  7. Infeasible Scheduling Request:
  8. Symptom: Task warning "Task requires resources that cannot fit on any node".
  9. Fix: Verify individual bundle sizes fit within single node maximum resource bounds.

5. Hands-on Practice Exercises

Exercise ID Goal / Topic Playground Link
scheduling01 Fractional & Custom Resources Open Exercise scheduling01 →
scheduling02 Node Affinity Scheduling Open Exercise scheduling02 →
scheduling03 Placement Groups: SPREAD Strategy Open Exercise scheduling03 →
scheduling04 Placement Groups: PACK Strategy Open Exercise scheduling04 →
scheduling05 Gang Scheduling Multi-Bundle Open Exercise scheduling05 →
scheduling06 Dynamic Runtime Environments Open Exercise scheduling06 →