Distinguished Lecture Series: Sepp Hochreiter

February 7, 2024, 11:30 a.m. (CET)

Memory Concepts for Large Language Models

Time: February 7, 2024, 11:30 a.m. (CET)
  Universitätstraße 32.101, Campus Vaihingen of the University of Stuttgart. The talk will take place in person.
Download as iCal:

The Stuttgart ELLIS Unit is pleased to announce the upcoming ELLIS Distinguished Lecture Series talk by Sepp Hochreiter (Johannes Kepler Universität Linz).

No reservation required - we are looking forward to seeing you there!

With any questions, or to request a meeting with Prof. Hochreiter, please contact the new coordinator of the Stuttgart ELLIS Unit, Katrin Fauß, at ellis-office@uni-stuttgart.de

Title: Memory Concepts for Large Language Models

Abstract:

Currently, the most successful Deep Learning architecture for large language models is the transformer. The attention mechanism of the transformer is equivalent to modern Hopfield networks, therefore is an associative memory. However, this associative memory has disadvantages like its quadratic complexity with the sequence length when mutually associating sequences elements, its restriction to pairwise associations, its limitations in modifying the memory, its insufficient abstraction capabilities. The memory grows with growing context. In contrast, recurrent neural networks (RNNs) like LSTMs have linear complexity, associate sequence elements with a representation of all previous elements, can directly modify memory content, and have high abstraction capabilities. The memory is fixed independent of the context. However, RNNs cannot store sequence elements that were rare in the training data, since RNNs have to learn to store. Transformer can store rare or even new sequence elements, which is one of the main reasons besides their high parallelization why they outperformed RNNs in language modelling. I think that future successful Deep Learning architectures should comprise both of these memories: attention for implementing episodic memories and RNNs for implementing short-term memories and abstraction.

Bio:

Sepp Hochreiter leitet das Institut für Machine Learning, das LIT AI Lab und das Audi.JKU Deep Learning Center der Johannes Kepler Universität in Linz. Er forscht auf dem Gebiet des maschinellen Lernens und ist ein Pionier des boomenden Forschungsfeldes Deep Learning, das gerade die künstliche Intelligenz revolutioniert. Bekannt wurde Prof. Hochreiter durch die Entdeckung und Entwicklung von “Long Short Term Memory” (LSTM) in seiner Diplomarbeit im Jahre 1991, welche später 1997 publiziert wurde. In jüngster Zeit hat sich LSTM zur besten Methode für Sprach- und Textverarbeitung entwickelt, wo es neue Rekorde aufstellte. Seit 2012 wird LSTM in Google’s Android Spracherkenner verwendet, seit 2015 in Google’s Voice Transcription, seit 2016 in Google’S Allo und auch seit 2016 in Apple’s iOS 10. Zurzeit treibt Prof. Hochreiter die theoretischen Grundlagen von Deep Learning voran, indem er den Gradientenfluss durch neuronale Netze analysiert, das Konzept von “Self-Normalizing Networks” entwickelt und “Generative Adversarial Networks” (GANs) sowie “Reinforcement”-Algorithmen verbessert. Derzeitige Projekte umfassen Deep Learning für Medikamentenentwicklung, für Text- und Sprachanalyse, für Bildverarbeitung und im speziellen für autonomes Fahren.

ellis events 

Sepp Hochreiter
[Picture: Barbara Klaczak]

List of all events


December 2024

October 2024

November 2024

October 2024

September 2024

August 2024

July 2024

June 2024

July 2024

June 2024

April 2024

May 2024

April 2024

March 2024

February 2024

January 2024

December 2023

November 2023

October 2023

November 2023

October 2023

August 2023

July 2023

June 2023

July 2023

June 2023

May 2023

June 2023

May 2023

March 2023

February 2023

January 2023

December 2022

January 2023

December 2022

November 2022

October 2022

September 2022

August 2022

July 2022

June 2022

May 2022

April 2022

March 2022

February 2022

January 2022

February 2022

January 2022

December 2021

October 2021

September 2021

August 2021

July 2021

June 2021

May 2021

April 2021

March 2021

February 2021

January 2021

December 2020

November 2020

October 2020

September 2020

August 2020

July 2020

June 2020

May 2020

April 2020

March 2020

February 2020

January 2020

December 2019

November 2019

October 2019

July 2019

June 2019

July 2019

June 2019

May 2019

October 2020

April 2019

May 2019

April 2019

March 2019

February 2019

January 2019

December 2018

November 2018

September 2018

To the top of the page