Embodied Task Agents Can Perform Fine Manipulation through Tactile In-Context Learning
Abstract
Tactile-ETA improves embodied manipulation by integrating tactile observations into in-context demonstrations, boosting success rates without model updates.
Can embodied task agents use tactile experience to perform fine manipulation without task-specific model training? We investigate this capability through Tactile-ETA, a system that incorporates tactile experience into an embodied task agent through in-context learning. Successful expert demonstrations show how contact changes during manipulation. We pair each recorded robot movement with its camera views and bilateral tactile observations. We keep tactile observations at the start and end of each movement and add two neighboring time points when the tactile images change markedly in between. The agent combines these demonstrations with current vision, tactile observations, and proprioception to select actions through general-purpose robot tools, without updating model parameters. Across four simulated manipulation tasks, adding tactile observations to the same two expert demonstrations increases average success from 15.75% to 24.00%, with current sensing and execution interfaces held fixed. Further experiments show that performance depends on the sensory content and representation of demonstrations, while comparing agent models on Lift Bottle reveals substantial differences under the same system configuration. On the real-robot tasks, demonstrations with aligned visual and tactile observations increase the task success rate from 0% to 60%. These findings demonstrate the feasibility of using tactile experience in context for fine manipulation and suggest a path toward extending embodied task agents to additional sensory inputs.
Community
Thank you for sharing your work. Tactile-ETA enables a frozen agent to use contact information from expert demonstrations. Adding tactile content to the same two demonstrations raises mean success from 15.75% to 24.00% across four simulated tasks, with current sensing and action interfaces held fixed.
We would like to discuss your work in Section 3.4 (“Skill- and agent-based execution”) of our survey, In-Context Learning for Robots: Methods and Applications, to show how demonstrated contact enters tool-based action selection, with the controlled demonstration comparison supporting our evaluation of which contextual information changes execution.
If you find our discussion of learning from demonstrations useful for your own work, we would be grateful if you would consider citing the survey.
Get this paper in your agent:
hf papers read 2610.27613 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper