The culprit behind rising memory prices: Who is HBM memory?

The rapid advancement of AI technology and application ecology has had a profound impact on people's work and life. At the same time, people also discovered that the price of memory was like a rocket, completely breaking the inherent law of "overcapacity, iterative price reduction, and becoming more affordable year by year" for many years. Then, mobile phones and laptops were reduced or prices were increased, and desktop DIY was instantly ignored. The culprit behind this was HBM memory. The full English name of HBM is High Bandwidth Memory. It is a high-end storage product specially designed for AI training and high-performance computing. From the underlying principles and application scenarios, it still belongs to the category of DRAM dynamic random access memory. It has the same origin as the DDR4 and DDR5 ordinary memories commonly used in our computers and mobile phones but in different forms. For example, NVIDIA H200 is equipped with HMB3e memory, which is a stack of six square HBM chips surrounding the GPU core and integrated side by side with the CPU on the same interposer substrate. From a principle point of view, traditional DDR memory uses a flat tiled packaging layout, with a limited number of chip pins. The channel width and transmission speed of data transmission are limited by the physical structure. It is difficult to improve capacity and performance, and can only meet the needs of ordinary computing and entry-level AI applications. HBM has innovated the packaging architecture, using 3D vertical stacking technology and TSV through-silicon via technology to stack dozens of DRAM chips vertically layer by layer (familiar?), and then through micron-level precision through-silicon vias to achieve high-speed interconnection between each layer of chips. The transmission distance is shorter and the signal loss is extremely low, which increases the data throughput capacity of the memory and prevents the GPU from being idle waiting for data. It is the core supporting resource for AI chips. From a performance perspective, the bus width of each stack of HBM can reach 1024 or even 2048 bits, while ordinary DDR5 memory has only 64 bits, which is equivalent to upgrading from a "country road" to a "highway". The amount of data that can be transmitted per unit time increases by orders of magnitude. In terms of bandwidth, the GDDR6 memory on the graphics card only has a bandwidth of about 0.064TB/s, while the HBM3e single stack