Conjunctive Visual and Auditory Development via Real-Time Dialogue

Zhang, Yilu and Weng, Juyang (2003) Conjunctive Visual and Auditory Development via Real-Time Dialogue. [Conference Paper]

Full text available as:



Human developmental learning is capable of dealing with the dynamic visual world, speech-based dialogue, and their complex real-time association. However, the architecture that realizes this for robotic cognitive development has not been reported in the past. This paper takes up this challenge. The proposed architecture does not require a strict coupling between visual and auditory stimuli. Two major operations contribute to the “abstraction” process: multiscale temporal priming and high-dimensional numeric abstraction through internal responses with reduced variance. As a basic principle of developmental learning, the programmer does not know the nature of the world events at the time of programming and, thus, hand-designed task-specific representation is not possible. We successfully tested the architecture on the SAIL robot under an unprecedented challenging multimodal interaction mode: use real-time speech dialogue as a teaching source for simultaneous and incremental visual learning and language acquisition, while the robot is viewing a dynamic world that contains a rotating object to which the dialogue is referring.

Item Type:Conference Paper
Keywords:autonomous mental development, incremental hierarchial discriminant regression, language acquisition, SAIL
Subjects:Computer Science > Machine Learning
Computer Science > Robotics
ID Code:3342
Deposited By: Prince, Dr Christopher G.
Deposited On:12 Feb 2004
Last Modified:11 Mar 2011 08:55


Repository Staff Only: item control page