Reinforcement Learning Robots: How Training Works

What Reinforcement Learning Means for a Robot

Reinforcement learning teaches a robot by letting it try, scoring the result, and repeating that millions of times. What we see readers get wrong most often is picturing that happening on the physical machine, when almost all of it happens in a simulator.

The robot is given a reward function, which is a formula that scores how well it did. A walking reward might add points for forward speed and staying upright, and subtract them for falling, wasting energy, or lurching sideways.

The learning algorithm then adjusts the control policy to score higher next time. A policy is just a function that turns sensor readings into motor commands, and after enough attempts it produces motion nobody wrote by hand.

That last point is the whole appeal. A hand-coded gait works on the surface it was tuned for, while a trained policy learns to stay upright across the range of conditions it saw in training.

The Sim-to-Real Loop, Step by Step

Training runs in a physics simulator and only the finished policy reaches the robot. This is the loop every current consumer robot with a trained gait follows.

• Build a model of the robot in a physics simulator, with its real masses, joint limits, and motor behaviour.
• Write a reward function that scores the behaviour you want.
• Run thousands of simulated robots in parallel, each trying slightly different actions.
• Let the learning algorithm update the policy toward higher-scoring behaviour, over millions of attempts.
• Randomise friction, mass, and sensor noise during training, so the policy does not overfit to a perfect simulated world.
• Export the trained policy to a portable format and load it onto the robot.
• Run it on hardware, watch where it fails, and change the reward or the randomisation rather than the code.

How Microduck Implements That Loop

Microduck is the clearest published example of this pipeline on a $399 robot. Working through its stack shows what each abstract step costs in practice.

Simulation runs in mjlab, built on MuJoCo Warp, and the learning algorithm is Proximal Policy Optimization (PPO). The reported figure is roughly one to two hours on a CUDA graphics card for a usable gait, running 4,096 simulated robots at once.

The trained policy exports to ONNX, an open format for shipping models between frameworks, and runs on the robot under a Rust runtime at 50 Hz. Fifty corrections per second keep a 25 cm biped upright, which is why the policy must run onboard rather than stream from a laptop.

Every behaviour on the robot came out of this loop, including standing back up after a fall. Our Microduck review covers the hardware those policies run on.

What Hardware You Need

Training needs a CUDA graphics card, or a hosted equivalent. The robot itself is often the cheaper half, and this is the cost that launch coverage consistently leaves out.

Parallel simulation makes training tractable. Running 4,096 simulated robots at once on a graphics card compresses months of physical trial into an evening. The same work on a processor alone is impractical for a beginner.

Hosted training is the alternative and works fine for occasional runs. Microduck ships this route first-party through Hugging Face Jobs. Weigh the hourly rate against how many reward-tuning cycles you expect, because the first policy is rarely the one you keep.

The robot itself needs enough onboard compute to run the finished policy, which is a much smaller requirement. Microduck does it on a Rockchip RK3566 with 1 GB of RAM, which is modest hardware by any measure.

Reinforcement Learning vs Learning by Demonstration

Reinforcement learning suits continuous control and demonstration learning suits tasks you can show. Choosing wrongly between them wastes more time than any hardware decision.

Balance, gait, terrain, and recovery are reinforcement-learning problems, because there is no way for a human to demonstrate the thousands of tiny corrections a walking robot makes each second. You can only score the outcome and let the machine find its own answer.

Picking up a mug, folding a cloth, or stacking blocks are demonstration problems. A person moves the robot through the task, cameras record it, and a model learns to reproduce the behaviour. That is what the LeRobot library and the SO-101 arm are built around, and our LeRobot guide covers it.

Hugging Face now ships both routes, which is unusual. Microduck is the reinforcement-learning path and the SO-101 arm with LeRobot is the demonstration path, and a builder who understands the split will pick the right one first time.

Which Robots You Can Train Today

Three affordable robots let you run a full training loop yourself, and each makes a different tradeoff. All three publish their stacks openly.

Microduck is the assembled option at $399, a 25 cm biped with the SDK, simulator, and reinforcement-learning pipeline published under Apache-2.0. Its mechanical designs are not open, so you can retrain it but not rebuild it.

Open Duck Mini v2 is the fully open bipedal project, targeting a bill of materials under $400, designed in Onshape and trained in MuJoCo with reinforcement learning. It lists Hugging Face and Pollen Robotics among its sponsors, and you build it yourself. Our buy-or-build comparison covers that decision.

Petoi Bittle X is the $319 quadruped route with an open control stack. Four legs make balance far easier than two. That makes it the gentler place to start and the weaker demonstration of what a trained policy can do.

Who Should Not Start with Reinforcement Learning

Three kinds of project are better served by something else, and starting here costs them weeks. Reinforcement learning is a tool for problems where you cannot write the answer down.

If your task has one correct sequence, write the motion directly. Take a pick-and-place at fixed coordinates, a repeatable inspection sweep, or a demo that must run identically every time. All three are faster and more reliable as scripted motion.

If your interest is manipulation, take the demonstration route instead. Grasping and placing are taught by showing the robot, and our SO-101 robot arm guide covers the hardware that pairs with it.

If you have no CUDA graphics card and no intention of paying for hosted runs, the loop described above stays closed to you. Buying a robot with a published training stack and never training on it is an expensive way to own seven fixed behaviours.

What would change our answer is hosted training cheap enough to run a few dozen reward-tuning cycles without thinking about the bill. The route already exists through Hugging Face Jobs, so the open question is price rather than availability. Get that cheap enough and reinforcement learning becomes the default starting point rather than the ambitious one.

Where Beginners Get Stuck

Reward design breaks more projects than compute does. A policy optimises exactly what you measured, and it will find behaviour that scores well while looking nothing like walking.

The classic failure is a reward for forward speed with no penalty for style. The robot learns to lunge, hop, or drag a leg, because those genuinely move it forward faster than a clean gait early in training. Adding penalties for energy use and body attitude usually fixes it.

The sim-to-real gap is the second wall. A policy that walks in simulation and shuffles on your floor is normally a mismatch in friction, mass, or motor response. Randomising friction, mass, and motor response during training prevents that.

Hardware calibration is the third and the most frustrating, because nothing in the software is wrong. A leg assembled a few degrees off its zero position gives the policy a bias it never trained against, and the robot falls consistently to one side.

Start by retraining a working policy rather than writing one from scratch. Change one term in an existing reward function, watch what the robot does differently, and build intuition before you design your own.