INRC Forum May 30th: Jason Eshraghian & Ruijie Zhu
Tuesday, May 30, 2023 @ 8:00-9:00am PT / 17:00-18:00 CET you are invited to attend an INRC Forum talk from Jason Eshraghian & Ruijie Zhu, University of California, Santa Cruz.
Scaling up SNNs with SpikeGPT
Abstract: If we had a dollar for every time we heard "It will never scale!", then neuromorphic engineers would be billionaires. This presentation will be centered on SpikeGPT, the first large-scale language model (LLM) using spiking neural nets (SNNs), and possibly the largest SNN that has been trained using error backpropagation.
The need for lightweight language models is more pressing than ever, especially now that we are becoming increasingly reliant on them from word processors and search engines, to code troubleshooting and academic grant writing. Our dependence on a single LLM means that every user is potentially pooling sensitive data into a singular database, which leads to significant security risks if breached.
SpikeGPT was built to move towards addressing the privacy and energy consumption challenges we presently run into using Transformer blocks. Our approach decomposes self-attention down into a recurrent form that is compatible with spiking neurons, along with dynamical weight matrices where the dynamics are learnable, rather than the parameters as with conventional deep learning.
We will provide an overview of what SpikeGPT does, how it works, and what it took to train it successfully. We will also provide a demo on how users can download pre-trained models available on HuggingFace so that listeners are able to experiment with them.
Bio(s):
Dr. Jason Eshraghian is an assistant professor of Electrical and Computer Engineering at UC Santa Cruz. He is the developer of snnTorch, a widely adopted Python library used to train and model brain-inspired spiking neural networks. He was awarded the IEEE TCAS-I Darlington'23, IEEE TVLSI'19, and IEEE AICAS'19 best paper awards, and the best live demonstration award at IEEE ICECS'20. He was the recipient of the Fulbright Future Fellowship (Australian-America Fulbright Commission), the Forrest Research Fellowship (Forrest Research Foundation), and the Endeavour Fellowship (Australian Government). He leads the UCSC Neuromorphic Computing Group which focuses on porting principles from neuroscience into building more effective learning algorithms in software and hardware. Dr. Eshraghian is the Secretary of the IEEE Neural Systems and Applications Committee and an Associate Editor with APL Machine Learning.
Ruijie Zhu is commencing his Ph.D. in Electrical and Computer Engineering at UC Santa Cruz in the Fall of 2023. He recently completed his Bachelor Degree in Computer Science at the University of Electronic Science and Technology of China, where he became a regular contributor to open-source neuromorphic projects, including snnTorch, SpikingJelly, and led the development of SpikeGPT, the first spiking neural network generative language model. He was elected as the chair of the 2020 Students Open-Source Conference (SOSConf), which attracted over 3,000 online participants. His research focus is on enabling the development of large-scale spiking neural networks.
Recording & Slides:
For details on past and future INRC talks, see the full INRC Forum Summer 2023 Schedule (accessible only to INRC Affiliates and Engaged Members).
If you are interested in becoming a member, here is the information about joining the INRC.