FMS 2024: Neo Semiconductor Developed 3D X-AI Chip Targeted to Replace Existing HBM Chips and Solve Data Bus Bottlenecks

NEO Semiconductor announced the development of its 3D X-AI chip technology, targeted to replace the current DRAM chips inside high bandwidth memory (HBM) to solve data bus bottlenecks by enabling AI processing in 3D DRAM.

3D X-AI can reduce the huge amount of data transferred between HBM and GPUs during AI workloads. The company’s innovation is set to change the performance, power consumption, and cost of AI Chips for AI applications like GenAI.

The 3D X-AI Chip uses 3D memory to perform AI operations. It reduces huge amount of data transferred between HBM and GPU. Set to change the performance, power consumption, and cost of AI Chips for AI applications like GenAI.

AI chips with NEO’s 3D X-AI technology can achieve:

100x performance acceleration: contains 8,000 neuron circuits to perform AI processing in 3D memory.
99% power reduction: minimizes the requirement of transferring data to the GPU for calculation, reducing power consumption and heat generation by the data bus.
8X memory density: contains 300 memory layers, allowing HBM to store larger AI models.

“Current AI Chips waste significant amounts of performance and power due to architectural and technological inefficiencies,” said Andy Hsu, founder and CEO. “The current AI Chip architecture stores data in HBM and relies on a GPU to perform all calculations. This separated data storage and data processing architecture makes the data bus an unavoidable performance bottleneck. Transferring huge amounts of data through the data bus causes limited performance and very high power consumption. 3D X-AI can perform AI processing in each HBM chip. This can drastically reduce the data transferred between HBM and GPU to improve performance and reduce power consumption dramatically.”

Click to enlarge

A single 3D X-AI die includes 300 layers of 3D DRAM cells with 128Gb capacity and 1 layer of neural circuit with 8,000 neurons. According to NEO’s estimation, this can support up to 10TB/s of AI processing throughput per die. Using 12 3D X-AI dies stacked with HBM packaging can achieve 120TB/s processing throughput, resulting in a 100x performance increase.

“The application of 3D X-AI technology can accelerate the development of emerging AI use cases and promote the creation of new ones,” said Jay Kramer, president, Network Storage Advisors. “Harnessing 3D X-AI technology to create the next generation of optimized AI Chips will spark a new era of innovation for AI Apps.”

NEO Semiconductor is showcasing its technologies at FMS: the Future of Memory and Storage.

Hsu delivered a keynote address titled New 3D AI Chip Technology Accelerates Generative AI on August 6.