Date of Award
2026
Document Type
Open Access Dissertation
Degree Name
Doctor of Philosophy in Computer Science (PhD)
Administrative Home Department
Department of Computer Science
Advisor 1
Jianhui Yue
Committee Member 1
Zhenlin Wang
Committee Member 2
Soner Onder
Committee Member 3
Kaichen Yang
Abstract
Large language model inference is limited by data movement rather than arithmetic: attention re-reads the key–value (KV) cache for every generated token, feed-forward weights are re-fetched just as often, and inference-time search grows that cache into a widely shared tree. Processing-in-memory (PIM) supplies the needed bandwidth only when data layout and access order match how DRAM works. This dissertation presents three systems, one per traffic source.
Biserial Sparse Attention targets attention. Predictor-free progressive filtering loads the first bit-plane of every cached token before pruning, then reads each admitted value at full precision. We add inexpensive attention score distribution guided block-level KV-cache pruning ahead of the token-level interval filter, so token-level traffic is bounded by a block budget, not by context length, and guide that budget from the score distribution of the previous decoding step, removing per-model calibration. A gap-driven allocator widens the per-token decision from read-or-skip to how many bits to read. Across the 7B models it holds plateau quality at 3.8–10.1% of dense attention traffic, against 20–27% for the prior accelerator.
Balance-and-Skip targets sparse feed-forward networks. Running them on near-bank PIM in lockstep wastes most of what sparsity saves: the up-projection suffers bank-level load imbalance, while the down-projection has almost nothing to skip at whole-row granularity. We balance the first with independent per-bank row activation, calibrated neuron placement, and cross-bank striping, and attack the second by gating inactive chunks inside open rows and packing active neurons together. On four sparse 7B models the projections run 1.7–2.4× faster than lockstep execution.
The third system targets the KV cache under test-time scaling. Tree search makes reasoning steps differ sharply in how many branches read them, so no single device serves all well; existing systems place every step uniformly. We place each step among near-bank PIM, a logic-die near-data processor, and the host GPU. Each device needs its own KV layout, so the choice must be made when a step starts, before its reuse is observable; a compact model predicts it from signals the serving stack already computes, and one permitted migration corrects what only observation can reach. At beam widths 64 and 128, attention runs 4.4–6.3× faster than all-GPU execution, worth up to 1.37× end to end, at 20–28× less energy.
Recommended Citation
Shen, Jiangqiu, "Inference in Motion: Memory-Centric Co-Design for Large Language Models", Open Access Dissertation, Michigan Technological University, 2026.
https://digitalcommons.mtu.edu/etdr/2174