Deep dive on continual learning for robots
Interactive continual learning in neuromorphic hardware for assistive robots
This month, my team’s work on continual learning for robots was featured in a number of tech news articles (Is Intel Labs' brain-inspired AI approach the future of robot learning?) after our project was recognized with the Best Paper award at this year's International Conference for Neuromorphic Systems (https://intel-ncl.atlassian.net/wiki/spaces/INRC/blog/2022/09/13/1809580037). In this short article, I’ll dive into the research behind the headline, to give you a better understanding of the neuromorphic technology we developed. You can also read the full paper here: ACM Digital Library or PDF
Learning from examples is a key achievement of modern data-driven AI. Continual and lifelong learning from new examples, on the other hand, still eludes most deep learning-based AI systems today. The reason for this lies in the core of the deep learning algorithm: error backpropagation.
Backpropagation involves gradient-based adjustment of the neural network's parameters (weights), and it is a slow, incremental process that changes millions or billions of parameters that contribute to the errors produced by a network for a given batch of data samples.
To enable the gradient-based learning algorithm to converge to a good solution, each example must only change the network a tiny amount. Moreover, it is important that examples come in a balanced sequence: a network that has only seen "cats" and then starts seeing all the "dogs", will have a difficult time learning the second, new concept without forgetting the first one. This catastrophic forgetting is a major problem for neural networks today.
However, when we think about future robotic applications – in homes, hospitals, or retail stores – we would benefit from more flexible learning models that are small enough to compute locally and can be trained on the job.
In such a setting, a user may show objects of interest to a robot, one by one, and might add a couple more later, instead of relying only on a pretrained and rigid network model. After all, this is how we expect a human apprentice to learn new tasks.
To obtain the best performance from AI models, we need a hybrid solution that combines slow gradient-based learning for feature extraction with a different type of learning – fast, one shot learning from examples.
My research team at Intel's Neuromorphic Computing Lab develops neural algorithms that enable breakthroughs in human-centered robotics. @Elvin Hajizada focuses on continual object learning in collaboration with Prof. Gordon Cheng from TU Munich and EDPR at IIT in Genoa. The paper we presented at ICONS was our first neuromorphic architecture tackling the challenge of continual learning for robots using the Intel Loihi neuromorphic research processor.
Our neural architecture combines a deep convolutional neural network for feature extraction with a continual learning layer of dynamic weights. These weights continually change according to a dynamical rule -- a three-factor learning rule.
The learning rule determines when each weight should increase or decrease, creating a "memory trace" of the visual appearance of an object and updating it each time recognition fails or an error occurs. Each significantly different object's view is represented by its own set of weights and a separate group of output neurons, adjusting the complexity of object representation to complexity of object’s appearances under different viewing angles. Separate representations for each object alleviate the problem of catastrophic forgetting.
The third factor in the learning rule controls the learning dynamics and makes the learning process autonomous. This is done by detecting states of the network when the weights should be updated. These states are detected by a group of neurons that we call a Neural State Machine (NSM). In 2019, my colleagues and I showed how NSMs can enable robust learning for intelligent agents (pdf) like robots.
For the current work, the NSM detects different states of the learning and recognition system, e.g. "an object is present and not recognized" or "this label has been seen before". Activated state-neurons trigger different actions accordingly, e.g. asking the user for a label, signaling recognition, updating synaptic weights that represent stored patterns, or recruiting neurons for new objects or objects’ views.
The neural state machine and three-factor learning rule work together to allow a robot to memorize a handful of objects (8 in the paper) in multiple 3D views (8 per object) and recognize them with 96% accuracy in interactive learning-recognition sessions. At each presentation of the object, the robot attempts to recognize it, if recognition fails, it requests a label and updates object representation. When errors are made, both the false positive and false negative are updated. Learning can be triggered any time and fewer updates are needed as the robot learns a given set of objects.
We ran the system on Loihi using NxSDK, Intel’s previous generation of hardware and software. The classification neural network consumed 150x less energy than state-of-the-art continual learning architectures solving the same task on a conventional processor. This is one of the first examples of exploiting the vast algorithmic space of stateful neural networks with different topologies – a feedforward feature extractor, a layer of plastic weights, and a neural state machine.
In the coming months, my team and I plan to transfer these models to Lava, the open-source neuromorphic framework that supports Loihi 2, which will provide even greater performance benefits and will make the algorithms available for the whole research community to build on.
Biological neural circuits can provide inspiration for such novel neural algorithms, while neuromorphic hardware provides the right computing substrate for their efficient implementation. We believe this approach to designing neural network models with on-chip learning will lead to breakthroughs in cognitive and interactive robotics.
If you're interested in learning more about this project or want to share feedback, let me know in the comments.
References
Elvin Hajizada, Patrick Berggold, Massimiliano Iacono, Arren Glover, and Yulia Sandamirskaya. 2022. Interactive continual learning for robots: a neuromorphic approach. In Proceedings of the International Conference on Neuromorphic Systems 2022 (ICONS '22). https://doi.org/10.1145/3546790.3546791 (pdf)
Dongchen Liang, Raphaela Kreiser, Carsten Nielsen, Ning Qiao, Yulia Sandamirskaya, and Giacomo Indiveri. 2019. Neural state machines for robust learning and control of neuromorphic agents. In IEEE Journal on Emerging and Selected Topics in Circuits and Systems. https://doi.org/10.1109/JETCAS.2019.2951442 (pdf)