Tell the Robot What You Want
Tell the Robot What You Want
- Day
- Day 3 — Session Day 2
- Time
- 12:05pm-12:25pm
- Room
- Track 2
- Track
- Robotics & World Models
Accessible with the Engineering pass and above.
About this session
What if you could command a robot just by talking to it? This session introduces Strands Agents, an open-source framework that lets developers control physical sensors and actuators using natural language, by exposing hardware as programmable agent tools through a unified interface. The agent interprets the request, selects appropriate tools, and orchestrates execution. We explore a hybrid model where low-latency perception and actuation run locally on edge hardware, and higher-level reasoning and multi-step planning are delegated to cloud-based agents when needed. This preserves real-time responsiveness while enabling richer reasoning. A live robot demonstration anchors the session. Using the SO101 robotic arm powered by NVIDIA GR00T alongside HuggingFace LeRobot, attendees see how an instruction such as “pick up the cube” moves from conversation to perception to physical action.
