Learning to see in a virtual world

Large-scale computational models offer the hope of providing a systems-level account of the brain’s visual pathways and functions, but their development is often hampered by an inadequate computing infrastructure and ineffective or unrealistic interactions with the environment. With respect to the computing infrastructure, hardware and software architectures are often inadequate to implement large-scale, parallel systems capable of learning and inference. As a result, model development is a slow and tedious process and, perhaps more critically, a model’s true potential may become hidden due to constraints imposed by the computing environment. Secondly, most models are developed to perform an isolated visual task, which lacks the environmental structure needed to develop a model capable of learning to perform multiple tasks. To overcome the above limitations, our approach is to embed a neural model capable of learning in an active animat living in a realistic 3D virtual environment, and to scale and speed up simulations using recently available high performance computing resources. A primary focus of the proposed approach is on self-organized learning, with the intent to develop a model capable to cope with changes in its virtual world or when deployed on a robot. Critical to our approach is the notion that mitigating the above technical limitations will allow for further theoretical developments to be made. Although similar approaches have been proposed before, they have been limited to more primitive visual systems or to primarily non-visual tasks, due to the lack of adequate computing resources and of suitable 3D graphics software, respectively.

Our computational framework is based in large part on Cog Ex Machina (or Cog), a new software platform for large-scale heterogeneous computing. Although Cog is truly general purpose, it is especially suitable for parallel and distributed models that make use of local learning laws and minimize the shuttling of synaptic weights across the network. Cog runs on both conventional CPUs and GPUs, and is designed with a view to take advantage of dense, low-power, memristive devices, which will allow for unprecedented scaling up of simulations.

In this work, we demonstrate our modeling approach by using Cog to simulate learning in a model of the primary visual pathways embedded in an animat wandering in a 3D virtual world. The animat’s visual system implements opponent processing of cone receptor signals and learns boundary and surface feature maps replicating those found in primary visual cortex.  We conclude on our current efforts to extend our model to address the kind of tasks known to be performed by biological visual systems.

This work was partially funded by the DARPA SyNAPSE program, contract HR0011-09-3-0001. Massimiliano Versace, Ennio Mingolla, Heather Ames, and Ben Chandler were also supported in part by CELEST, a National Science Foundation Science of Learning Center (NSF SBE-0354378 and NSF OMA-0835976).