2026-09-16 · AI-assisted

Async GRPO and LoRA Synchronization Without NCCL

According to a Hugging Face blog post, developers can implement asynchronous Group Relative Policy Optimization by training LoRA adapters and syncing them to vLLM via a shared Storage Bucket mounted via FUSE.

Removing NCCL for Asynchronous GRPO Training

The Hugging Face blog post Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL describes an alternative approach to distributed reinforcement learning. Instead of synchronizing full model weights across a high-speed cluster network using NCCL, the documented architecture uses a shared Storage Bucket mounted at the same absolute path across separate Hugging Face Jobs.

When training a 1.5B model using TRL v1.14's AsyncGRPOTrainer, the trainer outputs a rank-1 LoRA adapter that is only a few megabytes in size. This small adapter file is written to the shared storage bucket, allowing separate vLLM inference jobs to load the updated weights dynamically.

How the Components Interact

The source details a multi-job architecture divided into three main operational parts:

1. The Trainer Job: Runs AsyncGRPOTrainer with FSDP on dedicated hardware (such as an h200x2 setup) and writes saved LoRA checkpoints to the mounted storage bucket.
2. The vLLM Replicas: Run on separate individual GPU jobs using vllm/vllm-openai:v0.27.1. The source states these replicas must enable runtime LoRA updates via specific environment variables (VLLM_ALLOW_RUNTIME_LORA_UPDATING=1, VLLM_SERVER_DEV_MODE=1, and the --enable-lora flag). Operators must also set --max-loras 6 to account for a max_staleness value of 4 plus current and swapping versions.
3. The Local Proxy Server: Sits between the trainer and the vLLM replicas. It handles two distinct tasks: it routes generation requests using chained block hashes of 16-token prompt blocks (seeded with the adapter name) to maximize KV prefix cache reuse for group rollouts, and it broadcasts state-changing commands—like adapter loads, pauses, and resumes—to all connected replicas.

Practical Implications for Developers

Developers building practical AI engineering pipelines may find this setup useful for several reasons:

• Decoupled Infrastructure: Training and inference run in isolated Hugging Face Jobs, eliminating the need to configure rigid cluster networking across virtual machines.
• Reduced Transfer Overhead: Moving megabyte-scale LoRA adapters through a storage bucket avoids the bandwidth bottlenecks associated with broadcasting gigabytes of full model weights.
• Optimized KV Caching: The proxy routing logic helps multi-replica inference setups retain cache hits during group generations where prompts share identical prefixes, avoiding redundant prefill calculations.

According to the source, applying these optimizations across five runs reduced total training time from 3 hours and 27 minutes down to 53 minutes for 500 steps using the sail/Sanity-Test-R1D-1.5B dataset.

Limitations and Constraints

The source highlights several operational limitations to keep in mind:

• Strict Configuration Dependencies: The setup relies on pinned software versions (such as vLLM v0.27.1) and exact parameter matching. Features unsupported by vLLM at runtime—such as DoRA, modules_to_save, or ranks exceeding --max-lora-rank—trigger a fallback to merged-weight synchronization.
• Potential Trainer Bottlenecks: Depending on the hardware configuration, the training forward and backward passes can become a bottleneck. This leaves generation replicas waiting and underutilized if training and generation speeds are poorly balanced.
• Storage Latency and Consistency: Because adapter synchronization depends on FUSE-mounted storage buckets, eventual consistency anomalies can occur. The proxy layer must incorporate retry and rollback logic to handle situations where the bucket mount has not yet propagated the latest adapter file across isolated containers.