Experience Tuning on WebShop — methods brief & results

Base model Qwen3.5-9B (scale-up: Qwen3.6-27B, QLoRA 4-bit) · WebShop env, 100 held-out Word2World goals shared by every method · Score = mean reward×100, SR = % episodes with reward 1.0 · greedy decoding throughout.

1 · The task

WebShop is a simulated shopping site: the agent gets a natural-language instruction ("a 3 oz bottle of bright citrus deodorant for sensitive skin, under $50"), and must search, click a product, select options (size/color/count), and Buy Now. Reward ∈ [0,1] measures how well the purchased item matches the instruction's attributes; reward = 1.0 counts as success. An episode is a chain of Observation → Reasoning → Action steps (the ORA chain).

2 · Methods — what each one actually is

Frozen, prompted agents (no training)

Act — the model sees the few-shot example and the episode history (all past observations and actions in-context), and at each step emits exactly one environment action: search[...] or click[...]. No reasoning text at all. This is the paper's "Act" ablation: acting from memory, without thinking.

ReAct (paper protocol) — identical loop, but the action space is extended with a thought: the model may emit think[...], which does not touch the environment (it just gets "OK." back) and is kept in-context to guide later actions. Reasoning and acting interleave: think about what the results mean → act on the conclusion. In the paper (PaLM-540B) this beats Act because the thoughts plan, track sub-goals, and decide when to commit. Crucially, each thought costs one of the 15 loop steps, so thinking has a budget price. Faithful replication: their exact few-shot prompts, greedy, stop-at-newline, 6400-char context.

Native-chat Act / ReAct — the same two agents rebuilt on the model's own chat template (multi-turn: observation = user turn, action = assistant turn) instead of the GPT-3-era flat prompt, plus a worked example of the buy cadence and an anti-loop guard. The "ReAct" variant reasons in one short sentence (verify the instruction's color/size/count/price) inside the same generation as the action — so reasoning no longer costs an env step. This separates "reasoning fails" from "the foreign prompt format fails".

Trained agents (LoRA fine-tuning, r=256)

Full-history Experience Tuning (ET) — supervised fine-tuning on expert WebShop trajectories rendered as one flat sequence [obs_0][think_0][act_0][obs_1]…. Three design choices define it: (1) full history — the whole trajectory is the context, so the model learns to act conditioned on everything that happened; (2) loss masking — cross-entropy only on think+act tokens; observations are context, never predicted (this stops the model from drifting into page-text generation); (3) self-distilled reasoning — the think text ("Prediction: … Reason: …") is generated by the same model viewing the realized next observation (privileged context), then distilled in — a forward-dynamics rationale, not a teacher's plan. Trained to convergence with early stopping on a shared held-out validation set (same criterion for every cell → fair scaling comparisons).

Memoryless ablations (Obs→Act, Obs→React) — same training, but each example is a single step: current observation → action (optionally with a think in between), no history. These isolate what memory is worth: a memoryless policy cannot tell whether it already paged through results or clicked an option, so it loops.

Design-A: sliding window + intra-trajectory persistent RoPE (+ recall) — addresses trajectories longer than the 4096-token training window (they run to ~35K tokens). Sliding window tiles the full trajectory into overlapping 4096-token chunks so all of it is trained on (instead of front-truncation). Traj-RoPE gives every token its true absolute position within its trajectory (positions grow across chunks, reset at each new trajectory), so a chunk knows when in the episode it happened. Recall adds an auxiliary objective: given a short cue, reproduce a window's think/act content — training retrieval from weights. Result: null on task success (and recall hurts) — kept as an honest ablation.

The headline method: Continual Context Distillation (CCD)

CCD — the lifetime setting — everything above trains once, freezes, then evals. CCD instead mirrors human learning during a lifetime of 100 consecutive episodes: the ORA chain streams through the context window (working memory); after each episode the experience is consolidated into the LoRA ΔW by one gradient step (long-term memory) — necessary because a lifetime cannot fit in any context window; a recall buffer replays past high-reward episodes at their original time positions (memory rehearsal); and a global clock G that never resets gives every experience a unique persistent-RoPE timestamp (temporal anchoring). Performance is measured prequentially — each episode's reward is recorded before that episode is consolidated — so the reward-vs-episode curve directly shows whether accumulating experience makes the agent better. Four integration modes compared:

3 · Results

3.1 Scaling: history is the axis that scales

Full-history ET holds SR 22–37% across N = 150…20,000 training trajectories. The memoryless ablations stay near zero at every N — more data cannot fix a policy that can't remember what it already tried. Obs→Act actively collapses (it converges onto the dominant action and loops); Obs→React drifts up only slightly.

3.2 Prompted vs trained reasoning, across model scale

mode	what is consolidated after each episode	tests
frozen	nothing — pure in-context agent, ΔW never changes	control: is the climb caused by consolidation?
everything	all events (loss on obs+think+act), every episode	pure continual accumulation
reward_gated	observations always (world model); think+act only if reward ≥ 0.5	don't reinforce bad behavior
hybrid	reward_gated + recall-replay of a past high-reward episode at its original G-positions	full human-learning recipe: gated consolidation + rehearsal + time

WebShop	Act (prompt)	ReAct (prompt)	ET (trained)
Qwen 9B	SR 43	SR 10	SR 36
Qwen 27B	SR 50	SR 22	SR 39
PaLM 540B (paper, reported)	SR 30	SR 40	—

The think-trap. Prompted ReAct collapses at small scale for a verified reason (same-session traces): after an identical search returning identical results, Act clicks the product and buys; ReAct thinks "no item matches" — false — and re-searches the same query until the step budget dies (87/100 episodes). The reasoning step injects miscalibrated judgments and pays a step tax. It fades with scale (10→22→40) but only overtakes Act somewhere between 27B and 540B. Training the reasoning (ET) fixes the loop (36/39 ≫ 10/22) but still trails plain Act — on WebShop, at these scales, reasoning of any kind underperforms just acting with full history. The native-chat rebuild confirms this is robust, not a format artifact: native think SR 51 vs native act 57 at n=100 (an early n=20 think>act read was sampling noise). ALFWorld mirrors the story (9B prompted ReAct: 17.6% vs paper's 71%; only the trivial "examine" task solves).

3.3 CCD: consolidation makes the agent climb

lifetime mode	overall	first⅓ → last⅓	climb	SR
CCD everything	0.739	0.661 → 0.800	+0.138	0.40
CCD hybrid	0.740	0.709 → 0.753	+0.044	0.46
reward_gated	0.738	0.750 → 0.706	−0.043	0.43
frozen (control)	0.623	0.658 → 0.592	−0.066	0.36
static: Act / ET / ReAct	0.69 / 0.61 / 0.12	flat by construction

Headline. A single LoRA, consolidating streamed experience episode-by-episode with recall rehearsal and a persistent time clock, hill-climbs from 0.66 to 0.80 and ends above every static baseline — while the frozen control (same adapter, same rollout, no consolidation) stays flat at ~0.62 and slightly decays. The climb is therefore caused by consolidation, not by the harness. Within the same harness CCD also wins on exact success (SR 0.46 vs 0.36). This is the human-learning loop made concrete: stream → working memory → consolidate to weights → rehearse → anchor in time.

Caveats kept honest: the Score win is on partial-match reward; on exact-success SR the native-chat Act agent (0.57) is higher, but it is a different rollout harness — the like-for-like comparison is CCD vs frozen. Design-A's persistent-time machinery did not move frozen task success (null) — its value shows up here, in the lifetime setting. The reward_gated dip suggests gating starves the world-model updates early; "everything" climbs highest.

4 · One-paragraph summary

Memoryless policies fail WebShop regardless of data; history is the only axis that scales. Prompted reasoning is a tax small models cannot afford (the think-trap), and even trained or native-format reasoning does not beat plain acting in a frozen model. The decisive ingredient is not reasoning but accumulation: when experience streaming through the context is continually distilled into the weights — gated, rehearsed, and time-stamped like human memory consolidation — the same model that plateaus when frozen instead keeps getting better the longer it lives, finishing above every static baseline. Context, consolidated, is the new weight.

Artifacts: scaling_curve.html (per-cell numbers) · react_replication.html (replication tables) · scripts in exp/scripts/ (train_react_wm.py, lifetime_agent.py, react_baseline_webshop.py, react_native_webshop.py, ccd_analyze.py) · all evals on the same 100 held-out goals.