An emerging storage technology called high-bandwidth flash (HBF) could boost accelerator memory capacity from hundreds of gigabytes to terabytes, according to developers Sandisk and SK Hynix.
HBF uses stacked NAND flash instead of DRAM, aiming to provide SSD-like capacities at speeds comparable to high-bandwidth memory (HBM). Sandisk claims its first-generation modules could achieve read bandwidths up to 1.6 TB/s and capacities up to 512 GB per 16-high module — more than 14 times the capacity of HBM4 used in current AMD and Nvidia accelerators.
However, NAND flash has finite write endurance and access latencies measured in microseconds rather than nanoseconds. The companies propose using HBF to supplement HBM for AI inference, with HBM handling write-intensive tasks and HBF managing read-heavy decode phases. Sandisk's slides suggest configurations offering up to 3.12 TB of memory across two HBM stacks and six HBF stacks.
Sandisk expects first samples later this year, with the first HBF-based AI inference devices available early next year. Standardisation efforts are under way through the Open Compute Project.