The data layer of physical AI, discussed in a classic garage in San Francisco

On Tuesday evening, 6 October, Physical AI Operators held its second event, titled The Data Layer of Physical AI, at the Robotics Center during Tech Week in San Francisco. The venue is a classic garage, with robot arms, hands, humanoids and quadrupeds standing where one would otherwise expect tools and parked cars, and roughly 200 people had filled it by the time the panel started, which gave the evening the energy of a room where everyone present was working on the same problem. The organiser, Rachel Wang, started the community last month with a first event on 23 September about deploying robotics in business, and Tuesday’s session was co-hosted with the Robotics Center.

The evening exists because of Jerry Huang and his co-founder Stela Tong, who built the Robotics Center into a place where hardware and data can be collected in one location, with delivery within 48 hours on both, support for multi-camera setups from stereo and wrist cameras to rigs of four to six cameras, and tactile sensors on top. The center also runs a data marketplace at dexdata.roboticscenter.ai and the XEBench cross-embodiment benchmark with a public leaderboard at dexterity.roboticscenter.ai/benchmark, and it is hiring for business development, data evaluation, model work and internships. It is a Y Combinator Summer 2025 company.

Who spoke

The panel consisted of Jordan Larot, Director of International Business at AgiBot, Chongxi Lai, Research Lead at Astera Robotics, and Benoît Koenig, co-founder of Veesion, with Rachel Wang as moderator. According to Koenig, Veesion has deployed behaviour recognition models in 10,000 locations across 70 countries and has annotated close to 300 million videos, which it uses to detect shoplifting and operational issues in retail.

Data is the bottleneck

Asked what would limit progress if models stopped improving tomorrow, most panelists named data first and the closing of the deployment loop second. On the question of what the scaling unit for robotics data should be, the panel gave no single answer, since hours, rare events and diversity of tasks and environments all count, and the quality of the hours was judged to matter more than their number. Learning from real workers in real environments was preferred to relying on simulators alone.

The panel also agreed that no single data source solves physical AI. Exocentric footage from CCTV captures context and occlusion that egocentric capture misses, egocentric data collected with handheld or hand-pose devices suffers from the difficulty of localising hands accurately, and simulation remains the cheapest source while its ability to generalise is still unproven. The combination of egocentric, exocentric and sensor data was described as the emerging direction, and Veesion said it will launch a combined capture project next month.

Diversity across operators was a recurring point, because different people perform the same task differently, and the panel leaned towards domain expertise over uniformity, since warehouse workers, chefs and technicians produce episodes that are qualitatively different and more useful than those from generic operators.

Failure and edge cases should feed back into training, although they only become available after deployment, with human intervention data and novelty detection as the main triggers for retraining and with the weighting of intervention data inside the model still an open research problem.

Large foundation models were expected to reduce the need for semantic and object labelling data, while data for dexterous manipulation and whole-body control remains scarce and hard to combine, and private data from factories and inside companies was named as the next major bottleneck now that public data is close to exhausted.

Commercialisation and moats

Larot advised against starting a data collection company as a first business, because the work is expensive, low-margin, relationship-driven and dependent on hardware. Koenig described the alternative that Veesion represents, where data is generated as a byproduct of a deployed and paying product, and where multi-view data, long-horizon behaviours and evaluation datasets become valuable to OEMs, physical AI companies and integrators once the customer base exists.

On moats, the panel located the durable advantage of a data supplier in a continuous, compounding collection engine, meaning the capacity to produce new and relevant data as models and reality evolve, which a proprietary snapshot of data cannot provide. Larot added that researchers dislike reviewing data, which makes relationships with large companies and quality control decisive for AgiBot’s Maniformer, and that in his experience who you know is the most important factor, with quality control following from it.

The panel also listed problems worth building on: plugging and unplugging Ethernet cables, which is a multi-billion dollar data centre use case that no robot solves yet, a standardised data API with consensus on what data needs to come online, and automated data quality evaluation that predicts the usefulness of a dataset before training.

What to follow up

• The XEBench benchmark is open for review at dexterity.roboticscenter.ai/benchmark.
• Ian and Siming Li announced a seven-day Shenzhen Deep Dive tour in November with a physical AI focus, with details at linkedin.com/in/siming-li-647275b.
• Questions to the Robotics Center can be sent to stela@roboticscenter.ai.