HiRE: Hindsight Reward Editing for Policy Finetuning

Abstract

Pre-trained robot policies always require finetuning to adapt to specific environments. Reinforcement Learning (RL) offers high performance potential because it improves action optimality rather than simply mimicking data. However, such potential depends heavily on reward quality. Sparse rewards lack process feedback, human-designed rewards are costly and biased, and semantic rewards from foundation representations are often not control-centric. We propose Hindsight Reward Editing (HiRE), a training-free framework to break this reward bottleneck. HiRE bridges the broad knowledge of foundation representation models with physical control awareness, by contrasting successful and failed trajectories in hindsight. It calibrates foundation representation models by identifying “trap states” that are predicted as high-rewarding states yet eventually result in failure, and vice versa. HiRE explicitly penalizes these traps while boosting rewards for critical successful states. This approach can be flexibly compatible with any foundation representations and RL algorithms. Experiments show that HiRE consistently outperforms other reward recipes by delivering dense, control-aware feedback that prevents value function collapse and reward hacking, thereby achieving superior sample efficiency, stable policy updates, and higher performance ceilings, e.g., at least 3x performance of the base policies.

Method

Following maximum-entropy inverse RL, the ideal control-aware reward is the log-density ratio between success and failure states:

R^*(s) = \log p_{+}(s) - \log p_{-}(s)

R^*(s) = \log p_{+}(s) - \log p_{-}(s)

Instead of training a discriminator, HiRE estimates both densities non-parametrically in a frozen foundation representation space (e.g. DINOv2, SigLIP), using von Mises–Fisher kernel density estimation over a positive buffer $\mathcal{B}^+$ (successful rollouts and expert frames) and a negative buffer $\mathcal{B}^-$ (terminal frames of failed rollouts) collected during finetuning. This yields a closed-form, training-free potential:

\Phi_{\text{HiRE}}(s) = \mathbb{L}^{\kappa^+}_{\mathcal{B}^+}\!\left(\text{sim}_\phi(s, g^+)\right) \;-\; \lambda \cdot \mathbb{L}^{\kappa^-}_{\mathcal{B}^-}\!\left(\text{sim}_\phi(s, g^-)\right), \qquad \mathbb{L}^{\kappa}_{\mathcal{B}}(X) := \tfrac{1}{\kappa}\log \mathbb{E}_{\mathcal{B}}\left[e^{\kappa X}\right]

\begin{aligned}\Phi_{\text{HiRE}}(s) &= \mathbb{L}^{\kappa^+}_{\mathcal{B}^+}\!\left(\text{sim}_\phi(s, g^+)\right)\\ &\quad - \lambda \cdot \mathbb{L}^{\kappa^-}_{\mathcal{B}^-}\!\left(\text{sim}_\phi(s, g^-)\right),\\ \mathbb{L}^{\kappa}_{\mathcal{B}}(X) &:= \tfrac{1}{\kappa}\log \mathbb{E}_{\mathcal{B}}\left[e^{\kappa X}\right]\end{aligned}

where $\text{sim}_\phi$ is cosine similarity in the representation space and $\lambda = \kappa^-/\kappa^+ \in [0,1)$ balances the two manifolds. This subtractive structure partitions the representation space into three regimes. In the success-dominated regime, HiRE elevates dynamically correct states that appear visually distant from the goal, rescuing false negatives. In the failure-dominated regime, it imposes a severe penalty on deceptive trap states that look like the goal but have suffered irreversible failures, suppressing visual reward hacking. In the equilibrium regime, the two gradients cancel and the landscape flattens, so HiRE gracefully degrades to the clean sparse-reward recipe on under-covered states. The potential is integrated with the sparse task success signal via potential-based reward shaping:

R_{\text{HiRE}}(s,a,s') = R_{\text{sparse}}(s,a,s') + w\left(\gamma\,\Phi_{\text{HiRE}}(s') - \Phi_{\text{HiRE}}(s)\right)

\begin{aligned}R_{\text{HiRE}}(s,a,s') &= R_{\text{sparse}}(s,a,s')\\ &\quad + w\left(\gamma\,\Phi_{\text{HiRE}}(s') - \Phi_{\text{HiRE}}(s)\right)\end{aligned}

which provides dense guidance while provably preserving the optimal policy for task completion. The whole recipe requires no reward-model training or human labels, and is compatible with any RL finetuning backbone.

Experiments

We ask whether a better reward can improve an already strong policy-finetuning pipeline. In the main comparisons, all methods use the same DICE-RL backbone and the same initial policy within each task. We compare HiRE with sparse success rewards, the pretrained robotic reward model RoboMeter, the training-free VLM reward TOPReward, and GCR, which adapts its reward representation through gradient-based updates.

Simulation Benchmark Results

HiRE improves both final success and sample efficiency. On Stack_Three and Three_Piece_Assembly, it reaches roughly 50% and over 80% success, compared with around 30% and 70% under sparse rewards. On Tool_Hang, it enables learning where sparse-reward finetuning fails. On Threading, it reaches comparable success with markedly fewer online interactions.

RoboMeter and TOPReward achieve substantially lower final success rates, suggesting limited reward generalization to these tasks. GCR adapts rewards from online failures through gradient-based updates, but needs roughly 3× as many interactions to reach 30% success on Stack_Three and Three_Piece_Assembly, and makes little progress on the precision tasks. HiRE achieves more consistent improvement through training-free reward editing.

HiRE and four reward baselines on Stack Three, Threading, Tool Hang and Three Piece Assembly — Simulation benchmark. Success rate versus online environment steps for different reward recipes under the same DICE-RL backbone.

Real-Robot Results

On Bottle_Pick-and-Place, HiRE steadily guides the policy toward better performance, boosting success from 20% to 67%, whereas the sparse-reward and RoboMeter baselines nearly fail entirely in the early training phases. Evaluation spans six initial bottle positions, testing compliant manipulation and spatial generalization.

On Towers_of_Hanoi, HiRE reaches 53.3% success with only 20 episodes and eventually peaks at 73.3% (3.67× the base policy). Sparse reward peaks at 53.3% after 30 episodes and RoboMeter at 46.7% after 40, before declining to 30% and 20%, respectively, at 50 episodes. These comparisons show HiRE’s superior sample efficiency, stable improvement, and higher policy performance ceiling on both hardware tasks.

Successful bottle and Hanoi sequences alongside success-rate curves — Real-robot evaluation. Successful rollouts (left) and success rates over online episodes (right). `Bottle_Pick-and-Place` uses 30 evaluation trials per checkpoint across six initial positions; `Towers_of_Hanoi` uses 15. The base policies are initialized from 117 and 50 teleoperated demonstrations, respectively.

Compliant manipulation across six starting positions. The robot must grasp a deformable plastic bottle, carry it to the paper plate, and release it without slipping or bouncing away.

Position P1 · 1 / 6

All demonstrations: Bottle_Pick-and-Place: Position P1 · Bottle_Pick-and-Place: Position P2 · Bottle_Pick-and-Place: Position P3 · Bottle_Pick-and-Place: Position P4 · Bottle_Pick-and-Place: Position P5 · Bottle_Pick-and-Place: Position P6 · Towers_of_Hanoi: Rollout 1 · Towers_of_Hanoi: Rollout 2 · Towers_of_Hanoi: Rollout 3 · Towers_of_Hanoi: Rollout 4 · Towers_of_Hanoi: Rollout 5 · Towers_of_Hanoi: Rollout 6 · Recovery behaviors: Example 1 · Recovery behaviors: Example 2 · Recovery behaviors: Example 3 · Recovery behaviors: Example 4 · Recovery behaviors: Example 5 · Recovery behaviors: Example 6

Understanding HiRE

Correcting Misleading Visual Rewards

In the Threading example, correctly approaching the target initially makes the observation look less like the final goal, so raw DINO similarity decreases. Later, the robot loses its target and leaves the camera view, yet the raw score rises as the background resembles the goal frame. HiRE corrects these false negatives and false positives: successful manifold attracts robot to do correct behaviors, while failure manifold penalizes the visually plausible trap states. The edited potential follows task progress more faithfully than simply doing goal-frame resemblance alone.

Raw goal similarity and HiRE potentials along Threading and Stack Three trajectories — Reward landscapes on `Threading` and `Stack_Three`. Raw goal similarity and HiRE potentials along representative trajectories.

Adapting to Harder Failure Modes along Policy Finetuning

The negative buffer initially concentrates around early failures near the starting state. As the policy improves, older samples are replaced by localized, later-stage failure modes near the goal. This keeps reward correction focused on current traps. The joint t-SNE visualization also shows a compact success distribution and a broader failure distribution, supporting HiRE’s asymmetric kernels: sharp attraction toward success and broader repulsion from failure.

Positive and negative buffer samples in a shared t-SNE embedding across training — Buffer dynamics in a shared t-SNE space. Positive and negative buffer samples across training reveal how the negative buffer shifts toward harder failure states.

Ablations and training dynamics

Foundation representations

Both DINOv2 and SigLIP provide useful dense guidance across all four tasks. Replacing the frozen encoder leaves the hindsight-editing recipe intact. DINOv2 achieves higher success rates overall; SigLIP is closer on Tool_Hang and Three_Piece_Assembly, but learns more slowly on Stack_Three and especially Threading. The recipe transfers across representations, while encoder choice still affects efficiency and reliability.

DINOv2 and SigLIP encoder ablations across four tasks — Encoder ablation. HiRE with frozen DINOv2 and SigLIP representations.

Hyperparameter sensitivity

HiRE is broadly insensitive to hyperparameter choices. We vary the contrastive weight λ across {0.1, 0.3, 0.7, 0.9} and test a negative-buffer capacity of 512. The variants perform similarly on Stack_Three, and every configuration substantially improves Three_Piece_Assembly, with some differences in learning speed and final success. HiRE therefore retains effective learning across a broad range of settings.

Precision-critical tasks show more task-dependent behavior. On Threading, reducing λ or enlarging the buffer slows improvement, consistent with weaker repulsion or retention of outdated failures. On Tool_Hang, λ = 0.1 and 0.3 provide stronger guidance, whereas λ = 0.9 fails to sustain improvement. Together, these results support broad parameter insensitivity on multi-stage tasks while showing the value of appropriate failure correction for precision manipulation.

Contrastive weights and negative-buffer capacity across four simulation tasks — Hyperparameter sensitivity. HiRE retains effective learning across a broad range of settings on multi-stage tasks; precision-critical tasks show task-dependent preferences.

Reward composition

Hindsight editing and potential-based shaping work best together. On Stack_Three, full HiRE learns faster and reaches higher success than directly adding the edited potential or raw goal similarity. On Tool_Hang, both alternatives remain near zero while full HiRE continues to improve. The edited potential still outperforms raw similarity on Stack_Three, indicating that hindsight correction is useful even without temporal-difference shaping.

Full HiRE compared with no PBRS and raw similarity on Stack Three and Tool Hang — Reward composition. Combining hindsight editing with potential-based shaping gives the strongest results among the compared configurations.

Success and failure buffers

Failure awareness is most valuable when near-goal failures look successful. On Stack_Three, the positive-only variant closely tracks full HiRE and finishes slightly higher. On Tool_Hang, full HiRE learns faster and reaches above 90%, while the positive-only variant improves more slowly and plateaus lower. Success attraction supplies useful guidance, but the negative buffer adds the correction needed to distinguish deceptive failures in precision manipulation.

Full HiRE versus positive-buffer-only rewards on Stack Three and Tool Hang — Buffer design. Failure awareness is especially useful for contact-rich, high-precision manipulation.

Critic feedback on the real robot

Dense, outcome-grounded rewards sustain informative critic updates. During Bottle_Pick-and-Place finetuning, sparse rewards produce prematurely low critic loss alongside failed policy improvement, while RoboMeter’s loss does not converge. HiRE maintains non-trivial critic updates as success increases. The paper interprets this contrast as evidence that hindsight editing supplies useful learning signals when sparse feedback is insufficient.

Critic losses for HiRE, sparse reward and RoboMeter — Critic loss during `Bottle_Pick-and-Place` finetuning. HiRE maintains non-trivial updates, compared with prematurely low loss under sparse rewards and non-converged loss under RoboMeter.

BibTeX

@inproceedings{niu2026hire,
  title  = {HiRE: Hindsight Reward Editing for Policy Finetuning},
  author = {Niu, Haoyi and Han, Zhengtao and Ji, Yufeng and Li, Zhongyu and Sreenath, Koushil},
  booktitle = {Conference on Robot Learning (CoRL)},
  year   = {2026},
  
}

HiRE: Hindsight Reward Editingfor Policy Finetuning