MIT Vision System Teaches Robots To Understand Their Bodies

Reference: See here

A 3D-printed robotic arm holds a pencil as it trains using random movements and a single camera — part of a new control system called Neural Jacobian Fields (NJF). Rather than relying on sensors or hand-coded models, NJF allows robots to learn how their bodies move in response to motor commands purely from visual observation, offering a pathway to more flexible, affordable, and self-aware robots. | Credit: MIT

In an office at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL), a soft robotic hand carefully curls its fingers to grasp a small object.

The intriguing part isn't the mechanical design or embedded sensors — in fact, the hand contains none. Instead, the entire system relies on a single camera that watches the robot's movements and uses that visual data to control it.

"This work points to a shift from programming robots to teaching robots," says Sizhe Lester Li, MIT PhD student in electrical engineering and computer science, CSAIL affiliate, and lead researcher on the work . "Today, many robotics tasks require extensive engineering and coding.

In the future, we envision showing a robot what to do, and letting it learn how to achieve the goal autonomously."

This decoupling of modeling and hardware design could significantly expand the design space for robotics. In soft and bio-inspired robots, designers often embed sensors or reinforce parts of the structure just to make modeling feasible.

NJF lifts that constraint. The system doesn't need onboard sensors or design tweaks to make control possible. Designers are freer to explore unconventional, unconstrained morphologies without worrying about whether they'll be able to model or control them later.

"Think about how you learn to control your fingers: you wiggle, you observe, you adapt," said Li. "That's what our system does.

It experiments with random actions and figures out which controls move which parts of the robot."

The researchers see potential far beyond the lab. Robots equipped with NJF could one day perform agricultural tasks with centimeter-level localization accuracy, operate on construction sites without elaborate sensor arrays, or navigate dynamic environments where traditional methods break down.