Encord Tests Brain Wave Technology to Train Physical AI Robots Using Jenga Game

AI data startup Encord is testing brain wave-measuring headsets developed by Zander Labs to train physical AI robots in San Leandro, California. Using a Jenga game, the trial aims to address the severe scarcity of real-world physical training data. Encord's head of robot learning, Vineeth Velmurugan, notes that overcoming this bottleneck requires a dataset five times the size of YouTube's corpus, driving Encord to manufacture data through egocentric videos, teleoperation, and muscle sensors.

Brain wave robotics training trial

• ▪Zander Labs neuroscientist Lukas Gehrke stated that the amount of brain activity used during a task can help model builders determine when to deploy their highest-effort models
• ▪Encord is conducting a trial run in San Leandro, California, using a Zander Labs headset to measure the brain waves of robotic trainers playing Jenga
• ▪The Zander Labs headset measures brain activity to deduce mental states like error, intent, and surprise to create a more useful training dataset

Physical AI training data scarcity

• ▪Encord's head of robot learning, Vineeth Velmurugan, estimates that breaking through the robotics data bottleneck will require a dataset five times the size of YouTube's video corpus
• ▪Robotics experts and Encord executives state that the primary constraint for humanoid and warehouse robotics is the scarcity of real-world physical training data

Egocentric video data collection

• ▪Encord collects egocentric video data from workers wearing cameras at several factories globally to train robot brains
• ▪Encord annotates its datasets with physical descriptions of video content, which Encord's head of robot learning Vineeth Velmurugan estimates is worth 100 times more than unannotated data

Teleoperated robot data collection

• ▪Encord uses leader-follower rigs, where a human operator directly controls one robotic arm and another mimics its movements, to generate training data
• ▪Encord is developing a data modality using forearm sensors to detect electrical signals in muscles to build 3D depictions of hand movements

Robot manipulation task demonstrations

• ▪Encord stocks household items like fake flowers, books, plastic vegetables, and kitty litter trays to train robotic manipulators for household tasks
• ▪Encord pilots use robotic arms to demonstrate tasks such as pouring coffee into mugs, stacking poker chips, and plugging ethernet cables into servers

Data generation workforce economics

• ▪Encord employs robotic trainers, referred to as pilots, including Andrew Ceja and Sofia Infante, who previously worked at the AI data annotation firm Scale
• ▪Generating physical training data is highly expensive compared to scraping text from the internet, changing the economics of building physical AI models