TTT Models: A New Frontier in Data Processing Efficiency | Generative ai use Cases pdf | Generative ai Benefits for Business pdf | Generative ai Tools for Images | Turtles AI

Highlights:

• TTT models and their energy efficiency compared to transformers.
• Replacement of the hidden state in transformers with an internal machine learning model.
• TTT models’ ability to process large amounts of data while maintaining constant size.
• Development and adoption of transformer alternatives like SSM models.

TTT models, developed by teams from prestigious universities, promise to overcome the limitations of transformers, offering unprecedented efficiency in data processing with significantly reduced resource consumption.

In recent years, transformers have been at the forefront of AI, supporting advanced models like OpenAI’s Sora for video generation, Anthropic’s Claude, Google’s Gemini for text generation, and GPT-4o. However, these models are beginning to encounter technical obstacles, particularly related to computation. Their inefficiency in processing and analyzing large amounts of data, especially with standard hardware, is leading to an exponential and potentially unsustainable increase in energy demand as companies build and expand infrastructure to meet transformers’ requirements.

A promising solution that has recently emerged is the "test-time training" (TTT) architecture, developed by a team of researchers from Stanford, UC San Diego, UC Berkeley, and Meta over the course of a year and a half. These TTT models not only can process far more data than transformers but do so with significantly lower computational resource consumption. A fundamental component of transformers is the "hidden state," a long list of data that the model uses to "remember" what it has just processed. For example, if the model is reading a book, the hidden state values will be representations of words or parts of words. The hidden state, while being one of the elements that make transformers so powerful, is also their Achilles’ heel. To generate a single word about a book it has just read, the model has to scan through the entire lookup table, a computationally demanding task akin to rereading the entire book.

To address this issue, the team led by Yu Sun, a post-doc at Stanford and co-contributor in the TTT research, devised an internal machine learning model to replace the transformers’ hidden state. This internal model, unlike the lookup table of transformers, does not grow as it processes more data. Instead, it encodes the data into representative variables called "weights," maintaining high performance. No matter how much data a TTT model processes, the size of its internal model remains constant.

Sun and his team believe that future TTT models could efficiently handle billions of pieces of data, from text to images to videos. This is far beyond the capabilities of current models. "Our system can generate words about a book without the computational complexity of rereading the book each time," says Sun. Large video models based on transformers, like Sora, can only process ten seconds of video because they have a limited "brain" constrained to a lookup table. The team’s ultimate goal is to develop a system capable of processing long videos, approaching the visual experience of a human life.

However, doubts remain about TTT models. While promising, they are not an immediate replacement for transformers. The researchers have only developed two small models for study, making it difficult to compare them to larger transformer implementations. Mike Cook, a senior lecturer at King’s College London, notes that while the innovation is interesting, it is too early to determine if it is superior to existing architectures.

Nevertheless, the rapidly developing research into transformer alternatives signals a growing awareness of the need for a breakthrough. This week, AI startup Mistral released a model, Codestral Mamba, based on another alternative to transformers called "state space models" (SSMs). These also appear to be more computationally efficient and scalable to larger amounts of data. AI21 Labs and Cartesia are also exploring SSMs, with the latter having developed some of the first SSMs, including Codestral Mamba and Mamba-2.

Should these efforts succeed, they could make generative AI even more accessible and widespread than it is now, with all the positive and negative implications that entails.