What is it about?
The Transformer architecture, which powers AI like ChatGPT, is powerful but struggles with long data, becoming slow and memory-intensive. Our framework, Spectraformer, is designed to solve this efficiency problem. It systematically tests different mathematical "recipes"—called random features—to optimize the core of the Transformer. This has led us to discover novel combinations that make these powerful AI models significantly faster and less resource-hungry, while maintaining their high performance.
Featured Image
Photo by Igor Omilaev on Unsplash
Why is it important?
This work is important because it closes a key performance gap, establishing a new state-of-the-art for a class of efficient AI models called random feature-based Transformers. What's unique is our Spectraformer framework. For the first time, it unifies the previously fragmented research on using "random features" to make Transformers more efficient. This provides a systematic way to compare different techniques and discover novel, more powerful combinations that were previously overlooked.
Perspectives
I am proud that this work builds a bridge between modern Transformers and the rich history of kernel methods, allowing a wealth of established knowledge to be reused in today's AI applications. This is also an attempt to generalize the Transformer's fundamental attention mechanism. By viewing attention through the lens of kernels, we show that the standard "softmax" function is just one of many possibilities. The framework opens the door to discovering or even learning better attention functions for specific tasks , and I hope my open-source toolkit encourages the community to explore this flexible new approach.
Duke Nguyen
University of New South Wales
Read the Original
This page is a summary of: Spectraformer: A Unified Random Feature Framework for Transformer, ACM Transactions on Intelligent Systems and Technology, September 2025, ACM (Association for Computing Machinery),
DOI: 10.1145/3768161.
You can read the full text:
Resources
Contributors
The following have contributed to this page
