See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusion

This work proposes a methodology for a robot to learn complex manipulation skills, exemplified by playing Jenga, by emulating hierarchical reasoning and multisensory fusion. The approach involves formulating game mechanics using a temporal hierarchical Bayesian model, capturing latent structures in force and visual domains. The robot then uses these learned representations to infer block behavior patterns and states, adjusting its actions and strategy similar to human gameplay. The method demonstrates fidelity in a real-world implementation compared to standard baselines.

Download paper here