Heat management has become an essential requirement in the design and production of AI chips, presenters said at Inha University’s semiconductor industry-academia-research workshop at The Heaven Resort on Daebudo Island in Ansan, Gyeonggi Province, on Sept. 3. Inha University Professor Choi Ri-no, Hanyang University Professor Choi Chang-hwan, Korea University Professor Yoo Hyun-yong and Inha University Professor Cho Jae-hoon outlined technology trends and thermal obstacles for devices, materials and packaging.
AI chips pushed to thermal limits by higher integration
Choi Ri-no said increasing integration density is pushing conventional cooling methods to their limits. He noted that 2.5D structures that place GPUs and high-bandwidth memory (HBM) on an interposer are widely used today, and that the industry’s move toward 3D structures that stack chips vertically will make removing heat even more difficult.
Choi highlighted HBM as a particular challenge: according to his calculations, a GPU consuming 800 watts still has a margin of about 40 degrees before reaching its thermal limit, while HBM consumes about 50 watts but reaches its thermal limit earlier because its structure of multiple stacked DRAM dies makes heat difficult to dissipate. He added that DRAM can withstand temperatures about 20 degrees lower than GPUs, “meaning HBM effectively determines the maximum operating temperature of the entire AI chip,” he said.
He also cited an Nvidia roadmap showing power consumption per GPU rack projected to increase 15-fold in four years, from 40 kilowatts for the Hopper generation in 2023 to 600 kilowatts in 2027, and warned that the combination of rising power consumption and greater integration density is making it increasingly difficult for conventional cooling methods to remove heat.
“Cooling will shift from methods that cool chips from the outside toward approaches that move increasingly closer to the chip itself,” Choi said. He described current systems that circulate coolant through a metal plate placed on top of the chip and said he expects cooling methods to extend to microchannels formed inside chips, allowing coolant to flow directly through them.
Thermal interface materials and packaging trade-offs
Hanyang University’s Choi Chang-hwan emphasized the importance of thermal interface materials (TIMs), which fill gaps between chips and cooling plates to improve heat transfer. He warned that adding more thermally conductive material to increase thermal conductivity makes TIMs harder, and “if the material becomes too rigid, it cannot properly conform to the chip surface, creating gaps that instead impede heat transfer.”
“Even with a TIM that has high thermal conductivity, it will not be effective if interface control fails,” Choi Chang-hwan said. Rather than simply adding conductive filler, his team is studying ways to create directional paths for heat dissipation, for example by aligning highly thermally conductive materials such as graphene in a specific direction.
Choi Chang-hwan also discussed cooling challenges as more DRAM dies are stacked in HBM: the bumps connecting the dies and the polymer material filling surrounding spaces impede heat transfer. He said hybrid bonding, which eliminates bumps and directly bonds copper to copper, could offer an alternative, but noted that the insulating layer surrounding the copper does not conduct heat well. His team is studying replacing the insulating layer with highly thermally conductive materials such as aluminum nitride (AlN) or boron nitride (BN).
According to simulation results presented by Choi, an eight-layer DRAM stack reached 95 degrees Celsius using conventional microbumps, compared with 70 degrees with hybrid bonding. He cautioned that “hybrid bonding will not arrive easily,” predicting that it will not be used in HBM4 and will likely be adopted around the early 2030s.
Device- and interconnect-level design must include thermal planning
Korea University Professor Yoo Hyun-yong said thermal issues in logic chips need to be addressed starting at the device and interconnect design stages. He explained that as transistors evolved from planar structures to FinFETs and gate-all-around (GAA) architectures, the paths through which heat escapes have changed. In GAA structures, in particular, the current-carrying channel is separated from the silicon substrate, “making it difficult to dissipate heat downward.”
“As scaling progresses, power density increases while the paths available for heat dissipation become narrower,” Yoo said. He called for thermal designs that use metal interconnects to spread heat concentrated in one location to surrounding areas and urged a shift from a PPA focus to “a PPAT approach that includes thermal considerations,” adding that “devices, interconnects, circuits, packages and systems must be designed together by considering not only power, performance and area but also actual operating temperatures.”
Materials research to improve substrate durability
Research was also presented on improving the durability of heat-dissipation substrates for power semiconductors. Inha University Professor Cho Jae-hoon said his team is adding reinforcing materials to highly thermally conductive AlN to address its tendency to fracture easily. “Our goal is to double its resistance to fracture while maintaining the same thermal conductivity,” Cho said.
