Reward Shaping: Every Iteration Got Gamed

I started with the cleanest, most ideologically pure reward function I could write. It did not work. Neither did the second one. The third one is what the agent trains on today. Each one broke in a different way, and each failure taught me something specific about why the intuition “just reward the right outcome” falls apart on long-horizon sequential decisions. What follows is each reward function I shipped, what the agent did with it, and what I replaced it with.

The training loop runs in a patched Godot build of Slay the Spire 2, with the agent making card-play and end-turn decisions inside actual combat encounters. A typical fight has 20–30 actions before the combat resolves, which turns out to matter a lot for the first version below.

This is textbook credit assignment — figuring out which of those 20–30 actions caused the final outcome. At the start of training the agent wins maybe 1–2% of the time, so 98% of gradient updates push against -1.0 everywhere, with a value function that has no idea which actions were actually causal and which were incidental. The policy collapsed into randomness because that was the only signal the value head could fit to. Dense reward was the obvious next move — give the agent something to fit to on every step, not just at the end of the fight, so the value function has a chance to develop a sense of which states are better than which other states.

Win rate climbed from random to around 40–50% on Stage 0. Episode length climbed with it. Fights that a reasonable player finishes in 5–6 turns were running 40+. The agent was technically maximizing its per-turn reward — it was taking less damage per turn than a normal player would — but the fights dragged on so long that debuffs accumulated and it lost via attrition. This is textbook reward hacking: the reward function said “minimize HP loss per step,” and the stalling loop does minimize HP loss per step. The reward just did not say “minimize HP loss per fight.” Time was a free resource and the agent exploited it.

The signal that v2 was broken did not come from the win-rate graph — which actually looked plausible for a while, climbing from random into the 40s — it came from the episode-length graph, which started creeping and then spiking. That was useful to notice because it suggested the fix before I had even fully diagnosed the exploit: if episodes are long, make episodes expensive.

What I’d do first next time

Start with the dense + terminal hybrid from day one. Three components together:

1. Small step-level rewards that encode immediate feedback (HP damage, blocking, enemy kills).
2. A small penalty per turn to discourage stalling.
3. A terminal reward for win/loss.

The insight I lacked initially: pure outcome-based rewards do not work for sequential decision problems with long horizons unless you already have a decent value function. The common advice that “you should let the agent figure it out from minimal reward” is a trap at this problem size. Every real RL system I have read about in production uses some form of reward shaping — the only debate is how much.

Anticipate reward hacking. When you add any reward component, immediately ask: how could an adversarial agent exploit this? The HP-preservation reward was exploitable because it did not account for the passage of time. If I had red-teamed it mentally for five minutes before training, the stalling loop would have been obvious. The agent was not being clever; the reward was being incomplete.

Monitor episode length as a first-class metric. Along with win rate and reward. Wildly long episodes are almost always a sign of reward exploitation or broken termination conditions. It was the episode-length graph that first told me v2 was broken — win rate looked fine for a while.

Keep iteration cycles short. Each of these reward functions trained for an hour or two before I killed it. That was deliberate. There is a real temptation, when you have committed a few hours of GPU time to a run, to let it keep going to justify the sunk cost. I try to resist that. If the reward is broken, more compute makes it more broken, not less.

Technical details

• Framework: PPO (Proximal Policy Optimization)
• Step reward: -(damage_taken / max_hp) * 2.0 per transition — always ≤ 0, no positive step component (reward.py::step_reward)
• Turn penalty: -0.01 per turn end (reward.py::TURN_PENALTY)
• Terminal: win +1.0 + hp_remaining/max_hp (range +1.0 to +2.0), loss -1.0 — so terminal sits in [-1.0, +2.0]
• Discount factor: γ = 0.99
• GAE (generalized advantage estimation) lambda: 0.95

Key takeaways

1. Dense reward beats sparse reward for long-horizon tasks with random initial policies. Credit assignment is too hard with only a terminal reward; the value function cannot bootstrap itself out of the cold start.
2. Every reward signal can be exploited. Stare at your reward function and ask what a lazy, greedy agent would do to maximize it. Whatever loophole you find, the agent will find it faster.
3. Watch episode length. Regressions in episode length often reveal reward hacking before win rate does. It is the earliest public signal that the agent has found a degenerate optimum.
4. Iterate fast, measure everything. Each version trained for only an hour or two before I decided it was broken. The iteration loop matters more than per-iteration perfection — you learn more from three one-hour runs than from one carefully-tuned six-hour run. The mistakes the first run makes are rarely the mistakes the second run needs to make for you to learn the thing.
5. The reward is not the code that computes it; it is the policy it produces. The only real test of a reward function is what the agent does under it. The prose version of the reward — what I thought I was asking for — is almost always more forgiving than the actual math version the agent gets to see.