Single GPU Trains 14 Robot Skills for a Bipedal Duck in Five Days

What was built

A developer posting on dev.to describes MicroDuck, a small bipedal robot duck built from 15 XL330 servos, a Radxa ZERO 3W compute board and a 50 Hz control loop. Rather than training a single gait, the project targeted a full skill library: walking on flat and rough terrain, skating on wheeled feet, standing up after falls, rolling, kicking a ball, picking objects off the ground, and switching between sitting and standing. According to the write-up, all 14 skill variants were trained and exported as ONNX policies for deployment on the physical robot.

The run took about five days of wall-clock time on a single GPU. Simulation ran in mjlab, a MuJoCo-based environment using the Warp GPU-parallel backend, with PPO from the rsl_rl framework driving 4,096 parallel environments on flat terrain and 2,048 on rough terrain. The training code builds on pollen-robotics/microduck_rl.

Keeping an unattended run alive

Three infrastructure choices did most of the work. Tasks were listed in bash associative arrays and executed strictly one at a time so the GPU never contended with itself; each completed task dropped a marker file, while anything exiting with a non-zero status stayed unmarked and flowed into a retry queue. When the first batch of rough-terrain tasks exhausted VRAM within seconds at 4,096 environments, the fix was blunt: cut every rough task to 2,048 environments and rerun. The author's rule of thumb is that rough terrain, which carries more observation and physics state per environment, deserves roughly 70% of the flat-terrain environment count.

The post also separates two failure classes that can look similar in a log. Out-of-memory crashes arrive within seconds and are diagnosed from the tail of the log; NaN divergences appear minutes or hours later as an error about NaN values in the critic's observations and must be traced through Weights & Biases curves. A general patch — NaN guards on rewards and advantages plus a termination flag for NaN states — carried the remaining rough-terrain skills through training.

Reward shaping, skill by skill

The longest section treats each skill's reward function as its own design problem, with shared principles: task rewards are scaled to a magnitude of about 10 so shared regularisation terms keep consistent relative strength; domain-randomisation recipes are inherited from a proven velocity baseline instead of rebuilt; motion-damping penalties are held near zero during early exploration of large-motion skills and reintroduced through a curriculum; and progress rewards pay only for movement beyond the best progress so far, so standing still earns nothing.

A few examples. For walking, an upright penalty at weight 1.0 made a small forward lean effectively free — the converged gait leaned 2–4 degrees and two-thirds of push tests tipped the robot forward — so the weight was raised to 2.0. Stand-up training combined a three-part height reward with a curriculum that spawns episodes mid-flip, splitting a motion the robot could not learn whole into a half it could. The first attempt at a rolling skill produced fast, contact-free spinning that was genuinely optimal under the reward as written; the second gated rotation on ground contact, capped progress pay at 3 rad/s and paid a landing bonus only after 260 degrees of roll. Ball kicking used an asymmetric actor-critic: the actor never sees the ball, because the real robot has no ball perception, while the critic does, letting value estimation anticipate the kick payoff. Object pickup avoided defining a target pose at all, balancing a ground-attraction term, a contact prohibition and an upright-orientation term until the mouth hovers at ground level. The combined walk-and-recover skill needed three audit passes, including gating recovery rewards on having genuinely fallen and defining recovery concretely as tilt under 25 degrees with base height above 0.09.

The training recipe

Networks are plain MLPs (512-256-128, ELU activations) with observation normalisation, a 1e-3 learning rate adapted against a target KL of 0.01, discount 0.99, GAE lambda 0.95, clip parameter 0.2, 24 steps per environment and 5 epochs over 4 mini-batches. Iteration budgets ranged from 25,000 for the velocity family down to 5,000 for ball kicking. No recurrence or vision was needed for 50 Hz servo control. Acceptance was judged by single-environment rollout animations rather than loss curves, since a converging loss says little about whether the resulting motion is usable.

Why it matters

Robot reinforcement learning is often framed as a big-compute discipline. This project is a counterexample at hobbyist scale: a complete multi-skill policy library, produced on one GPU in under a week, held together by deliberately plain infrastructure — a serial queue, marker files, retry logic and a clear split between memory failures and numerical ones. The transferable parts are the VRAM budgeting heuristic for rough terrain, the NaN-safety patch, and above all the per-skill reward notes, which read as a compact field guide to how reward shaping fails and how to fix it. As GPU-parallel simulators such as MuJoCo's Warp backend mature, building an end-to-end skill library starts to look less like a research demo and more like routine engineering.