Preserving Plasticity in Continual Robot Learning

(a) MuJoCo Locomotion

(b) Real World SO-101 Arm

(c) Franka Panda Simulation

TL;DR. A robot that keeps learning new tasks from camera images slowly loses its ability to learn. Standard fixes (layer norm, L2, reward normalization, Parseval regularization) aren’t enough in this setting. Keeping the network’s weight matrices on the Stiefel manifold (\(W^\top W = I\)) inside the optimizer, using Skewon, keeps agents plastic for up to 100M frames. For pretrained models, where weights aren’t orthonormal, our new Gram-Skewon optimizer preserves each matrix’s Gram matrix \(W^\top W\) instead, and continually fine-tunes a vision-language-action model on a physical robot arm.

1. Pick

2. Place

3. Stack

4. Cup

5. Sort

Why plasticity matters for robots

Imagine a robotic arm in a warehouse. Over its lifetime it learns a stream of tasks: picking boxes, placing them on conveyors, stacking them for storage. Meanwhile its joints wear, its grippers change, and new objects show up. The big world hypothesis says that an agent’s capacity is always small relative to the world it lives in. So the robot can’t learn everything up front. It has to stay able to learn.

Deep RL agents are known to lose this ability. Their networks become harder and harder to train as the data distribution shifts, a phenomenon called loss of plasticity. This has been well studied for control from low-dimensional proprioceptive states (joint angles and velocities). It is much less studied for control from pixels, which is how real manipulation systems usually perceive the world. That is the setting we focus on.

The idea: constrain the matrices, not the loss

A deep network is mostly a composition of matrix multiplications. Many plasticity fixes act on those matrices through the loss. L2 regularization keeps them small. Parseval regularization (Chung et al., 2024) penalizes their distance from orthonormality, \(\lVert W W^\top - I \rVert_F\).

We move the constraint out of the loss and into the optimizer. Every hidden weight matrix is kept exactly on the Stiefel manifold

so all of its singular values are pinned at one. A plain gradient step would leave the manifold. Instead, we pick an ascent direction in the tangent space and then retract back onto the manifold.

We use Skewon (Solonko, 2026), which gives a closed-form, Muon-style update for this constrained problem. Given a momentum or gradient matrix \(M_X\),

where \(\operatorname{polar}(N) = UV^\top\) for the SVD \(N = U\Sigma V^\top\) (computed with Newton–Schulz iterations), and \(R_X\) is a retraction.

Why should this help? We track plasticity through the network’s input–output Jacobian,

If the Jacobian norm is large, the network’s output responds to changes in its input. The agent can then turn reward feedback into useful learning signal. If it collapses, the network goes flat and stops responding. Fixing every \(W_l\)’s singular values at one stops the linear layers from adding ill-conditioning. (The activation derivatives \(D_l(x)\) still matter, though.)

A toy problem that isolates the effect

To compare the optimizers cleanly, with full hyper-parameter sweeps, we built a small pixel-based linear–quadratic regulator (LQR) environment in JAX. Eight blobs, coupled by “strings,” must be driven onto their red crosses. The 16-dimensional state is seen only through an 8,192-pixel image. Each chain alternates between two kinds of task:

• tasks that need strong feedback control of an unstable plant;
• heavily damped tasks that only need a large constant push.

So the optimal sensitivity of the policy swings down and back up at every task change.

Even with the best of 12 hyper-parameter settings per chain, AdamW gets stuck on many chains, and the chains where it does worst are exactly the ones where its Jacobian norm has shrunk the most. Skewon solves every chain.

Continual locomotion from pixels (MuJoCo)

Next, we train PPO agents from \(84\times84\) RGB frames on four continual locomotion variants. The dynamics change every 5M frames:

• Ant-Friction (15 tasks): floor friction drawn from a bimodal distribution.
• Ant-Fluid (10 tasks): the ant is immersed in fluid of varying density.
• Humanoid-Actuator (20 tasks): motor gear scales from \([0.2, 0.3] \cup [2.4, 2.5]\).
• Humanoid-Joint-Damping (20 tasks): damping from \([2, 8] \cup [40, 80]\).

All three methods share the same architecture, layer norm, reward normalization and Swish activations, so the AdamW baseline already includes the standard plasticity fixes. They differ only in how six key matrices are treated: unconstrained (AdamW), penalized (Parseval) or constrained (Skewon).

Ant-Friction, 15 tasks

Ant-Fluid, 10 tasks

Humanoid-Actuator, 20 tasks

Humanoid-Joint-Damping, 20 tasks

Skewon Parseval AdamW

Skewon pulls ahead early and stays ahead as tasks pile up. The gap is starkest on Ant-Fluid, where AdamW’s normalized return collapses to \(0.010 \pm 0.003\) by the end of the chain. On both humanoid variants, Skewon is still ahead at 95–100M frames, after 19 dynamics changes, while AdamW and Parseval plateau or decline.

The Jacobian tells the same story. Skewon is the only method whose Jacobian norm doesn’t shrink over the chain:

Ant-Friction

Ant-Fluid

Skewon Parseval AdamW

Continual manipulation in simulation (ManiSkill)

We also train two simulated arms on chains of manipulation tasks. Each task starts from the previous task’s final checkpoint. The policy sees camera images plus proprioception, with no privileged object positions. In Sort, the arm must work out from pixels alone whether it is holding a cube or a ball.

Grasp

Place

Stack

Cup-Align

Sort

Grasp

Place

Stack

Pyramid

Chain: each task is warm-started from the previous one (mean ± s.e. over seeds). Scratch: each task is trained independently from a random initialisation.

AdamW starts out better on Grasp, but Skewon wins on every task from Stack onward in both chains. Two results stand out:

• Cup-Align (SO-100): both optimizers reach 0.81 from scratch. In the chain, AdamW collapses to 0.03 while Skewon reaches 0.92. The task is learnable, but AdamW has lost the ability to learn it.
• Pyramid (Panda): neither optimizer learns it from scratch (0.00). Sequential training makes it learnable, and Skewon gets more out of the chain (0.94 vs 0.58).

Gram-Skewon: plasticity for pretrained models

Vision-language-action (VLA) models are pretrained, and their weights are nowhere near orthonormal. Projecting them onto the Stiefel manifold would wreck what they have learned. So we generalize the constraint. Rather than \(W^\top W = I\), we preserve each matrix’s pretrained Gram matrix:

Stiefel is the special case \(G_0 = I\). This keeps every singular value of the pretrained matrix fixed and lets the matrix rotate freely. Three observations make this cheap and clean:

• The tangent space doesn’t change. For any full-rank \(W\), the tangent space is \(\{YW : Y \text{ skew-symmetric}\}\), the same parametrisation Skewon uses. So the steepest-descent direction still has a closed form,

  \[Y^\star = -\operatorname{msign}\!\big(\operatorname{skew}(G W^\top)\big), \qquad \Delta W = \eta\, Y^\star W .\]
• Steps are measured as rotations. We measure step size by the spectral norm of the generator \(Y\), which rotates the columns rigidly. This bounds the relative change in every output direction by \(\eta\), so a single learning rate works across layers whose scales differ by more than 10×. It also protects the fragile, near-singular directions of the checkpoint.
• A one-time polar split. Write \(W_0 = V_0 P\) with \(V_0\) orthonormal and \(P = (W_0^\top W_0)^{1/2}\). The Gram-Skewon step on \(W\) is then exactly a plain Skewon step on \(V\), with the gradient pulled back as \(G P\). We never update or invert \(P\), so \(W^\top W = P V^\top V P = G_0\) holds by construction.

The spectrum floor in step 1 (\(\kappa_{\max} = 30\)) only touches a handful of freshly initialized cross-attention layers in SmolVLA, changing them by 1–2% in Frobenius norm. After that, the condition number stays bounded for the whole run, with no regularizer and no weight decay. Memory is about 1× the parameter count, versus 2× for AdamW’s moment estimates.

Real-world continual fine-tuning on the SO-101

We continually fine-tune SmolVLA’s action expert (99.9M trainable parameters; the VLM backbone is frozen) by behavior cloning on teleoperated demonstrations for five physical tasks: Pick, Place, Stack, Cup, and a language-conditioned Sort. In Sort, the same scene admits two correct goals, so only the instruction (“Put the yellow cube in the green cup”) tells the policy what to do. There’s no replay of earlier data, and the optimizer is the only difference between the two arms.

Gram-Skewon improves on Stack, Cup and Sort, raising final-task success from 36.7% to 61.7% and overall success from 50.8% to 65.7%. The cost is about 2× training time per task.

Takeaways

• Loss of plasticity is real in pixel-based control, and the standard fixes (layer norm, L2, reward normalization, Swish, Parseval) don’t remove it.
• Changing only the optimizer to Skewon, a Stiefel-constrained update, sustains learning for up to 100M frames, with no extra loss terms or regularization coefficients.
• The input–output Jacobian norm tracks plasticity: it stays stable under Skewon and shrinks under AdamW and Parseval, in the same order as their returns.
• Gram-Skewon carries the idea over to pretrained models by preserving \(W^\top W\) rather than forcing it to \(I\). It gives sizeable gains in continual VLA fine-tuning on real hardware.

Open questions remain. A theoretical account of how orthogonality and Gram preservation shape the Jacobian, and through it learning, is still missing. The constrained update also costs 1.4–2× in wall-clock time.