https://winterstar67.github.io/Recent content on WinterstarHugoenTue, 07 Apr 2026 21:08:18 +0900https://winterstar67.github.io/posts/gqa/Sun, 29 Mar 2026 00:00:00 +0000https://winterstar67.github.io/posts/gqa/<h2 id="the-point-that-nanochat-use-gqa">The point that nanochat use GQA</h2> <ul> <li>GQA is in the <code>CausalSelfAttention.forward()</code> in <code>gpt.py</code> file</li> <li>The reasong of applying GQA is to make training and inference faster than MHA</li> </ul> <h2 id="paper-info">Paper info</h2> <ul> <li><strong>Title</strong>: <em>GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints</em></li> <li><strong>Authors</strong>: Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai</li> <li><strong>Venue</strong>: EMNLP 2023</li> <li><strong>URL</strong>: <a href="https://arxiv.org/pdf/2305.13245">https://arxiv.org/pdf/2305.13245</a></li> <li><strong>Length</strong>: 7 pages</li> </ul> <h2 id="0-background-knowledge-to-know">0. Background Knowledge to know</h2> <ol> <li>Concept of GPU memory bandwidth. <ul> <li>What&rsquo;s the effect of a large memory width and small memory bandwidth</li> </ul> </li> <li>Computational complexity of Attention calculation</li> <li>KV-cache</li> <li>Matrix multiplication with expand</li> <li>Computational efficient()</li> </ol> <h2 id="1-motivation">1. Motivation</h2> <p>The memory bandwidth bottleneck has quite huge adverse-effect on autoregressive decoding process like GPT rather than the encoder like BERT.</p>