Skip to content

THE COURSE / M3 DESIGN TO MASK / M3.1

Architecture

After this lesson you can:

  • Tell a CPU, a GPU, a custom AI chip, an NPU and an FPGA apart by what each is built for.
  • Say what an architecture decides for a GPU-style accelerator: its blocks and its budgets for area, power and precision.
  • Explain, from a measured case, why more memory bandwidth can beat a bigger compute array.

Loading the lesson…

Sources

  1. NVIDIA, "NVIDIA Hopper Architecture In-Depth," NVIDIA Technical Blog, 22 March 2022 (read 10 November 2026). The full GH100 is made on "the TSMC 4N process customized for NVIDIA, with 80 billion transistors, a die size of 814 mm²": 8 GPCs, 144 SMs, 4 fourth-generation Tensor Cores per SM, 60 MB L2, 6 HBM3 or HBM2e stacks. The H100 SXM5 enables 132 SMs (528 Tensor Cores), 80 GB of HBM3 in 5 stacks on 10 512-bit memory controllers, and a 50 MB L2. Tensor Cores are specialised cores for matrix multiply-accumulate; FP8 "halves data storage requirements and doubles throughput compared to FP16 or BF16." The post's performance table is marked preliminary and is not cited; the shipping figures come from the product page.
  2. NVIDIA, H100 Tensor Core GPU, product specifications (read 10 November 2026). H100 SXM: FP16 and BF16 Tensor Core 1,979 teraFLOPS and FP8 3,958 teraFLOPS, both "with sparsity" (the dense rates are half); 80 GB of GPU memory at 3.35 TB/s; up to 700 W maximum thermal design power; NVLink at 900 GB/s.
  3. Google Cloud, "Introduction to Cloud TPU" (documentation, read 10 November 2026). TPUs are "application-specific integrated circuits (ASICs) used to accelerate machine learning workloads," hardware "designed for performing large matrix operations." CPUs suit quick prototyping needing "maximum flexibility" and small models; GPUs medium-to-large models and custom operations; TPUs models "dominated by matrix computations." The MXU is "a 128x128 systolic array."
  4. N. P. Jouppi, C. Young, N. Patil, D. Patterson and 71 others (Google), "In-Datacenter Performance Analysis of a Tensor Processing Unit," ISCA 2017 (arXiv 1704.04760) (read 10 November 2026). A custom ASIC deployed in Google's datacenters since 2015 for neural-network inference, designed, verified, built and deployed "in just 15 months". "The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory": a 256x256 array of 8-bit multiply-adds, a 24 MiB Unified Buffer and 4 MiB of accumulators. 28 nm, 700 MHz, TDP 75 W, 40 W busy; 34 GB/s from its 8 GiB DDR3 weight memory; its die is at most half of a 662 mm² Haswell die. In its floor plan the Unified Buffer is almost a third of the die and the matrix unit a quarter, so the datapath is nearly two-thirds. No caches, branch prediction or out-of-order execution: deterministic, for the 99th-percentile response time. "Eight-bit integer multiplies can be 6X less energy and 6X less area than IEEE 754 16-bit floating-point multiplies." On average 15-30 times faster than its contemporary CPU or GPU, with 30-80 times the TOPS per watt. Its roofline's ridge is at about 1,350 operations per byte of weights; four of its six production applications were memory-bandwidth limited. In the team's performance model, 4x memory bandwidth gave 3x on average, a faster clock little, and a 512x512 matrix unit was slightly slower on average than 256x256 (a 600x600 matrix takes 9 steps, 18 µs, on the smaller unit and 4 steps, 32 µs, on the larger); GDDR5 instead of DDR3 would have raised its geometric mean 2.6x.
  5. J. Hennessy and D. Patterson, "A New Golden Age for Computer Architecture," ACM Turing Lecture, ISCA 2018 (lecture page, read 10 November 2026). "The ending of Dennard Scaling and Moore's law also slowed this path; single core performance improved only 3% last year!" New advances could create compilers and domain-specific architectures that "deliver tenfold or more jumps in this new Golden Age." The full CACM article (February 2019, DOI 10.1145/3282307) could not be opened here and is not cited.
  6. Y. Han, H. Xu, M. Lu, H. Wang, J. Huang, Y. Wang, Y. Wang, F. Min, Q. Liu, M. Liu and N. Sun, "The Big Chip: Challenge, model and architecture," Fundamental Research 4, 1431–1441 (2024). The largest single die, 858 mm² (26 × 33 mm), set by the reticle and optics; the Poisson model tends to underestimate large chips' yield; the negative binomial model is widely used; a multi-chiplet system's yield set by its chiplets' yields and the bonding yield, which falls as chiplets are added.
  7. K. Tran (SK hynix), "The Era of High Bandwidth Memory," Hot Chips 28 tutorial, August 2016. TSVs through each DRAM die over a base die with the PHY, the TSV area, the DFT test logic and direct-access balls; 1,024 data I/Os from the base die to the host through the interposer, up to 256 GB/s a stack (HBM2, 2 Gb/s a pin); micro-bumps 25 µm across at 55 µm pitch; a molded known-good stacked die, 0.72 mm tall (Gen2); the test flow: wafer test, wafer burn-in, stacking, a logic test through IEEE 1500, hot and cold tests and repair, speed test; after assembly, repair of cells and of TSVs and micro-bumps by lane remapping through IEEE 1500; thermal dummy bumps for heat.
  8. SK hynix, "SK hynix Begins Volume Production of the World's First 12-Layer HBM3E," 26 September 2024. 36 GB in 12 layers; each DRAM chip made 40% thinner, so the stack keeps the eight-layer product's thickness; 9.6 Gbps; TSVs link upper and lower chips through thousands of fine holes; Advanced MR-MUF injects liquid protective material between the chips, controls warpage, reduces the pressure on the stacked chips, and dissipates 10% more heat than the previous generation.
  9. TSMC, "TSMC and Broadcom Enhance the CoWoS Platform with World's First 2X Reticle Size Interposer" (press release, 3 March 2020, as carried by Design & Reuse). Chip-on-Wafer-on-Substrate, first introduced in 2012; an interposer of about 1,700 mm², twice the reticle, by a mask-stitching process; several logic dies and up to six HBM cubes, 96 GB; up to 2.7 TB/s, 2.7 times its 2016 CoWoS; ready for N5.