Skip to content

Chapter 05: Lineage Reconstruction, Fault Tolerance & Actor Retries

  • Topic Focus • Task Retries, Lineage Reconstruction, Actor Restarts, and Object Loss Recovery
  • Interactive Challenges • 4 Hands-on Exercises
  • Launch Playground in Wasm →

1. Architectural Overview & Control Plane Mechanics

Ray provides distributed fault tolerance by recording the lineage graph (the sequence of function calls and arguments) used to generate every ObjectRef.

flowchart LR
    TaskA["Upstream Task A<br/><code>compute.remote()</code>"] -->|"1. Produces Ref A"| LostRef["Lost ObjectRef A<br/>(Node 1 Crash)"]
    LostRef -->|"2. Detects Missing Object"| GCS["GCS Lineage Tracker"]
    GCS -->|"3. Re-execute Lineage"| TaskAReplay["Replayed Task A<br/>(Healthy Node 2)"]
    TaskAReplay -->|"4. Restored Ref A"| TaskB["Downstream Task B<br/><code>process.remote(ref_a)</code>"]

    style TaskA fill:#1e293b,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
    style LostRef fill:#1e1e38,stroke:#ef4444,stroke-width:2px,color:#f8fafc
    style GCS fill:#1e293b,stroke:#f59e0b,stroke-width:2px,color:#f8fafc
    style TaskAReplay fill:#0f172a,stroke:#34d399,stroke-width:2px,color:#f8fafc
    style TaskB fill:#1e293b,stroke:#818cf8,stroke-width:2px,color:#f8fafc

Diagram Walkthrough & Core Concepts: - Lineage Tracking: Ray automatically records the deterministic DAG of function calls and object dependencies that generated each ObjectRef. - Autonomous Fault Detection: If a worker node crashes or an object is lost from Plasma memory, the GCS detects the missing dependency via heartbeats. - Lineage Reconstruction: Upstream tasks are re-executed on healthy worker nodes without requiring application restarts, seamlessly unblocking downstream consumer tasks.

2. Annotated Python Code Anatomy & API Reference

import ray
import random

# 1. Configure automatic task retries on failure
@ray.remote(max_retries=3, retry_exceptions=True)
def flaky_external_api_task(item_id: str) -> dict:
    """Task that retries automatically upon intermittent network exceptions."""
    if random.random() < 0.5:
        raise ConnectionResetError("Transient network failure")
    return {"status": "success", "item_id": item_id}

# 2. Configure resilient stateful actor with automatic restart & state recovery
@ray.remote(max_restarts=3, max_task_retries=2)
class ResilientWorker:
    def __init__(self, partition_id: int):
        self.partition_id = partition_id
        self.state_counter = 0

    def process(self, value: int) -> int:
        self.state_counter += value
        return self.state_counter

# 3. Instantiate resilient actor
actor = ResilientWorker.remote(1)

Key API Parameter Reference

  • @ray.remote(max_retries=N): Maximum times a task is re-executed upon worker crash or application exception.
  • @ray.remote(retry_exceptions=True): Enables automatic retry on standard Python exceptions in addition to worker crashes.
  • @ray.remote(max_restarts=N): Maximum times an actor worker process is restarted if it dies.
  • @ray.remote(max_task_retries=M): Retries pending method calls on an actor following a restart.

3. Production Best Practices & Hardening Guidelines

  1. Ensure Task Idempotency: Tasks configured with max_retries > 0 must be idempotent to prevent duplicate side effects.
  2. Limit Retries for Deterministic Bugs: Only set retry_exceptions=True for transient errors (e.g., network timeouts, rate limits).
  3. Persist Actor State to External Storage: Actor restarts instantiate a fresh actor (__init__); reload persistent state from checkpoints (e.g. S3 / GCS / Redis).
  4. Disable Lineage for Volatile Generators: For long-running streaming pipelines, disable lineage reconstruction (max_retries=0) and handle errors at the pipeline boundary.
  5. Monitor Task Retry Metrics: Alert on elevated ray_task_retry_count Prometheus metrics to detect intermittent infrastructure degradation.

4. Troubleshooting & Diagnostic Workflows

  1. Infinite Crash Loop on Actor Restart:
  2. Symptom: Actor exceeds max_restarts and raises ActorDiedError.
  3. Fix: Check actor __init__ logic for unhandled exceptions or invalid external credentials.
  4. ObjectLostError during Lineage Reconstruction:
  5. Symptom: Object cannot be recovered because input arguments were freed or non-reproducible.
  6. Fix: Keep root data persisted in durable storage; avoid capturing random state without seeds in tasks.
  7. Cascading Failure from OOM Worker Death:
  8. Symptom: Node kills worker due to OOM; task retries and kills next worker.
  9. Fix: Adjust task memory resource requests (@ray.remote(memory=...)) to avoid repeated node OOM kills.

5. Hands-on Practice Exercises

Exercise ID Goal / Topic Playground Link
fault01 Automatic Task Retries Open Exercise fault01 →
fault02 Actor Failure & Restart Recovery Open Exercise fault02 →
fault03 Lineage Reconstruction Open Exercise fault03 →
fault04 Spot Instance & Preemption Handling Open Exercise fault04 →