Improving RECAP: Better Advantage Labeling and Active Intervention Requests for Vision-Language-Action Models

RECAP ^[↓] (RL with Experience and Corrections via Advantage-conditioned Policies) enables Vision-Language-Action models to learn from both successes and failures through advantage conditioning, bypassing the need for policy gradients. However, the original method has two key limitations: (1) in failed trajectories, all actions are treated uniformly despite only some causing the failure, and (2) human supervision requires constant monitoring. We propose two improvements. First, failure point detection uses the value function to identify the exact moment a trajectory transitions from approaching success to drifting toward failure, enabling precise labeling - only post-failure actions receive negative advantage labels. Second, active intervention requests enable the policy to explicitly request human help when uncertain, shifting from passive monitoring to efficient collaboration. We implement these improvements on top of π₀.₆* and evaluate on simulated manipulation tasks.

1. Introduction

Recent work on π₀.₆* ^[↓] introduced RECAP, a method for enabling Vision-Language-Action (VLA) models to improve through experience without requiring policy gradients. By conditioning the policy on binary advantage indicators ("good" vs "bad"), RECAP converts reinforcement learning into conditional supervised learning, making RL-style improvement compatible with flow matching and diffusion policies ^[↓].

If you're unfamiliar with RECAP or π₀.₆*, I recommend reading my deep dive on the topic first. This article assumes familiarity with the core mechanism.

However, RECAP has practical limitations that reduce its effectiveness:

Problem 1: Imprecise labeling of failed trajectories. RECAP labels actions based on trajectory-level outcomes. In a failed trajectory, this means all actions receive negative advantage - but this is often incorrect. Consider a robot reaching for an object: it may move correctly toward the target for many steps before making a single mistake that causes failure. Labeling all actions as "bad" teaches the model to avoid good behaviors, contaminating the training signal.

Problem 2: Human supervision doesn't scale. RECAP relies on human interventions to demonstrate recovery behaviors and escape local optima. The current approach requires constant monitoring - one person per robot - which is impractical for fleet deployment.

We propose two improvements to address these limitations:

Contributions

1. Failure Point Detection: We use the value function to identify the failure point - the step where the robot was closest to success before things went wrong. Only actions after this point receive negative advantage labels. Actions before are discarded as ambiguous rather than mislabeled as bad.

2. Active Intervention Requests: Instead of passive human monitoring, the robot estimates its own uncertainty using an ensemble of value functions. When uncertainty is high, it requests human help instead of relying on constant human monitoring.

These improvements are complementary to RECAP's core mechanism and can be integrated into the existing π₀.₆* pipeline.

2. Background: RECAP and π₀.₆*

We briefly review RECAP as introduced in π₀.₆* ^[↓] to establish context for our improvements.

2.1 The Problem RECAP Solves

VLAs trained with behavioral cloning are limited by their demonstration data - they cannot improve beyond what they've seen. Standard RL (PPO, SAC) could enable improvement, but these algorithms require log-probabilities that flow matching policies cannot provide tractably.

2.2 Advantage-Conditioned Policies

RECAP's key insight is that RL-style improvement can be achieved through conditional supervised learning:

1. Train a value function $V_\phi(s)$ to predict steps-to-success from any state
2. Compute advantages: $A_t = G_t - V_\phi(s_t)$, where $G_t$ is the actual return
3. Binarize advantages: Top $k\%$ labeled "positive", rest "negative"
4. Train policy with conditioning: The policy learns to produce actions conditioned on this binary indicator
5. At inference: Always condition on "positive" to sample from successful behavior patterns

The theoretical justification comes from the observation that the optimal improved policy has the form:

$\pi_{\text{improved}}(a|s) \propto \pi_{\text{ref}}(a|s) \cdot P(\text{improvement} \mid a)^\beta$

When $\beta=1$, this reduces to conditioning on an improvement indicator - which is exactly what RECAP's binary advantage label provides.

2.3 Human Interventions in RECAP

When a human operator takes control during a rollout, their actions are recorded and forced to "positive" advantage (trusting expert corrections). This provides recovery demonstrations that the policy learns to imitate.

2.4 Limitations We Address

Trajectory-level labeling: The original RECAP computes advantages based on episode returns. For failed trajectories, this assigns negative advantage to all actions - even those that were making progress toward the goal.

Failed trajectory with ORIGINAL RECAP labeling:

Step 0        Step 5        Step 10       Step 15       Step 20 (FAIL)
  │             │             │             │              │
  ▼             ▼             ▼             ▼              ▼
[reach]  →  [approach]  →  [align]  →  [FUMBLE]  →  [DROP]
  ✗             ✗             ✗             ✗              ✗
NEGATIVE     NEGATIVE     NEGATIVE     NEGATIVE       NEGATIVE

↑ These were GOOD actions!    ↑ Only THESE caused the failure!

Passive supervision: Human operators must continuously monitor rollouts to catch failures and intervene. This requires one person per robot and doesn't scale.

3. Method

3.1 Failure Point Detection

Our first contribution is a method for precisely identifying which actions in a failed trajectory actually caused the failure.

Key insight: The value function $V_\phi(s)$ predicts steps-to-success. In a failed trajectory, there is typically a point where the robot was closest to success before things went wrong. Actions before this point were making progress; actions after were compounding errors.

Algorithm:

Given a failed trajectory $\tau = (s_0, a_0, s_1, a_1, \ldots, s_T)$:

1. Compute $V_\phi(s_t)$ for each step $t$
2. Find the failure point: $K = \arg\min_t V_\phi(s_t)$
   • This is the step where predicted time-to-success was lowest (closest to goal)
3. Label actions:
   • Steps $0$ to $K-1$: Discard (these were approaching the goal)
   • Steps $K$ to $T$: Negative advantage (these caused the failure)

Failed Trajectory Timeline:

Step 0 ──────────── Step K (failure point) ───────────> Step T (failure)
    │                        │                               │
    │   Value decreasing     │       Value increasing        │
    │   (approaching goal)   │       (drifting away)         │
    │                        │                               │
    │      DISCARDED         │      KEPT (advantage = -)     │
    │   (ambiguous data)     │      (caused failure)         │

Why discard instead of label positive? Actions before the failure point were moving toward the goal, but they were insufficient - they didn't prevent the eventual failure. Labeling them positive could teach suboptimal behaviors. Labeling them negative is clearly wrong. Discarding them avoids contaminating the training signal.

Why this works: The value function provides a dense progress signal that identifies exactly when the trajectory "turned bad." This is more precise than trajectory-level returns, which assign blame uniformly.

3.2 Active Intervention Requests

Our second contribution enables the robot to request help when uncertain, rather than requiring constant human monitoring.

Uncertainty estimation via ensemble: We train $M$ value functions $\{V_{\phi_1}, \ldots, V_{\phi_M}\}$ with different random initializations. Uncertainty is measured by disagreement:

$u(s) = \text{Var}_{i=1}^{M} \left[ V_{\phi_i}(s) \right]$

High disagreement indicates the ensemble hasn't seen similar states during training - the robot is in unfamiliar territory.

Intervention request protocol:

At each step:

1. Compute uncertainty $u(s)$
2. If $u(s) > \theta$:
   • Pause execution
   • Signal for human assistance (audio/visual alert)
   • Wait for human to take control
   • Record human actions with positive advantage
   • Resume autonomous execution when human releases control
3. Else: continue autonomous execution

At each timestep:

Compute uncertainty u(s) = Var[V_φ₁(s), ..., V_φₘ(s)]
     │
     ├── u(s) ≤ θ ──────────► Continue autonomous execution
     │
     └── u(s) > θ ──────────► PAUSE execution
                                  │
                                  ├── Signal for human (audio/visual)
                                  ├── Wait for human takeover
                                  ├── Record human actions (advantage = +)
                                  └── Resume when human releases

Benefits over passive monitoring:

When does the robot request help?

• Entering states not well-represented in training data
• Facing ambiguous situations where the value function is uncertain
• Approaching potential failure modes

3.3 Integration with RECAP Pipeline

Our improvements integrate into the standard RECAP training loop:

IMPROVED RECAP PIPELINE
═══════════════════════

For each training iteration:

1. COLLECT ROLLOUTS
   ├── Execute policy with advantage = "positive"
   ├── Monitor uncertainty u(s) at each step
   └── Request intervention when u(s) > θ

2. PROCESS TRAJECTORIES
   ├── Successful trajectories:
   │   └── All steps → advantage = positive
   ├── Failed trajectories:
   │   ├── Find failure point K = argmin V(s_t)
   │   ├── Discard steps 0 to K-1
   │   └── Steps K to T → advantage = negative
   └── Intervention segments:
       └── All human actions → advantage = positive

3. TRAIN VALUE FUNCTION ENSEMBLE
   └── M value functions on successful trajectories only (avoid drift)

4. TRAIN POLICY
   └── With advantage conditioning on processed data

The key differences from standard RECAP:

• Step 1: Active intervention requests replace passive monitoring
• Step 2: Failure point detection replaces trajectory-level labeling for failed trajectories
• Step 3: An ensemble of value functions is trained instead of a single value function

4. Experiments

4.1 Setup

We implement our improvements on top of the RECAP codebase and evaluate on simulated reaching tasks in Genesis.

Environment: Robot arm reaching task with random target positions.

• Observation: joint positions/velocities, end effector position, relative target position (24-dim)
• Action: delta joint positions (7-dim)
• Success: end effector within 0.08m of target

Baselines:

• RECAP (original): trajectory-level advantage labeling, passive monitoring
• RECAP + FPD: with failure point detection only
• RECAP + AIR: with active intervention requests only
• RECAP + Both: full method

4.2 Metrics

• Success rate: Percentage of episodes reaching the goal
• Label accuracy: For failed trajectories, what fraction of "negative" labels are actually on bad actions?
• Human effort: Interventions per episode, total human attention time
• Sample efficiency: Success rate vs. training episodes

4.3 Results

Failure Point Detection Analysis (planned):

• Comparison of label quality with/without FPD
• Visualization of failure points in example trajectories
• Impact on final policy performance

Active Intervention Request Analysis (planned):

• Human effort reduction vs. passive monitoring
• Comparison of intervention timing (uncertainty-based vs. human judgment)
• Impact on learning efficiency from interventions

5. Discussion

5.1 When Does Failure Point Detection Help Most?

Failure point detection provides the most benefit when:

• Failed trajectories have significant "good" portions before failure
• The value function accurately predicts progress
• Failures stem from specific actions rather than gradual drift

It helps less when:

• Failures occur immediately (no good actions to preserve)
• The value function is poorly calibrated
• The entire trajectory is uniformly suboptimal

5.2 Limitations of Active Intervention Requests

Uncertainty calibration: If the ensemble is overconfident (low uncertainty in unfamiliar states), the robot won't request help when it should. Conversely, if underconfident, it will request help too frequently.

Threshold sensitivity: The intervention threshold must balance safety (lower threshold, more requests) against human burden (higher threshold, fewer requests).

Not a replacement for good pretraining: Active intervention requests reduce monitoring burden but don't eliminate the need for human expertise. The quality of interventions still matters.

5.3 Relationship to Prior Work

Failure point detection relates to credit assignment in RL. While TD methods distribute credit temporally through bootstrapping, we use the value function directly to segment trajectories. This is simpler and doesn't require on-policy data assumptions.

Active intervention requests build on uncertainty-based methods in active learning and safe RL. ThriftyDAgger ^[↓] uses novelty detection to decide when to query an expert during DAgger-style training. The key difference is integration with advantage conditioning - uncertainty drives not just exploration decisions but also the labeling of human collaboration data within the RECAP framework.

Both DAgger ^[↓] and our active intervention requests address distribution shift between training and deployment. DAgger queries the expert at on-policy states; we request interventions at uncertain states and feed the resulting demonstrations into the advantage-conditioned training pipeline.

5.4 Future Directions

Adaptive failure margins: Rather than hard cutoff at the argmin, use a soft margin before the failure point to account for actions that led to the failure. The transition from "good" to "bad" is rarely instantaneous - a smooth weighting scheme based on $\frac{d}{dt} V_\phi(s_t)$ could capture this more faithfully.

Language-based intervention requests: Instead of a simple "help needed" signal, the robot could describe what it's uncertain about, leveraging the VLM backbone (e.g., "unsure how to orient the gripper for this object shape").

Cross-task transfer of value functions: Training a general progress predictor that transfers across tasks could improve failure point detection in new domains where task-specific data is limited.

6. Conclusion

We presented two improvements to RECAP for Vision-Language-Action models:

1. Failure point detection uses the value function to identify exactly when a trajectory transitions from progress to failure, enabling precise advantage labeling that avoids mislabeling good actions as bad.

2. Active intervention requests enable the robot to request help when uncertain, reducing human supervision burden while maintaining sample-efficient learning from expert corrections.

Together, these make RECAP's training signal cleaner and its human supervision requirements practical for multi-robot deployment.