The rapid emergence of large language models (LLMs) has transformed modern information processing, enabling unprecedented capabilities in reasoning, content generation, and human–machine interaction. However, their deployment is fundamentally constrained by the “memory wall” — the excessive cost of data movement between memory and compute units in conventional von Neumann architectures. As LLMs continue to scale in both model size and context length, this bottleneck increasingly dominates system performance, energy consumption, and scalability. Computing-in-memory (CIM) has long been viewed as a promising paradigm to address this challenge by performing computation directly within memory arrays, thereby minimizing data movement. Yet, despite its potential, CIM has traditionally been considered ill-suited for LLMs due to the inherent mismatch between the massive parameter size of LLMs and the limited capacity, precision, and reliability of emerging memory devices. This has led to a prevailing belief that CIM cannot effectively support modern generative AI workloads. This special session challenges this conventional wisdom and presents a new cross-layer perspective on how CIM can not only support but fundamentally reshape LLM system design. The key insight unifying this session is that CIM should not be viewed as a monolithic replacement for digital accelerators, but rather as a specialized substrate for selectively accelerating memory-intensive components of LLM workloads. By rethinking the mapping of LLM execution — offloading retrieval, context storage, and stochastic transformations into CIM fabrics — recent advances demonstrate that the perceived limitations of CIM can be transformed into unique advantages. The session begins with a broad architectural perspective on machine learning in memory, establishing the foundation for memory-centric computing paradigms. It then presents concrete system realizations showing how memristor-based CIM can efficiently support retrieval-augmented generation and long-context modeling in LLMs. Finally, the session highlights a deeper algorithm–hardware co-design insight: intrinsic device-level randomness, traditionally viewed as a drawback, can be directly exploited to implement advanced LLM compression techniques such as the recent breakthrough TurboQuant. Together, these talks provide a coherent and forward-looking vision in which CIM evolves from a niche accelerator into a key enabler for scalable, energy-efficient, and deployable generative AI systems. By bridging the gap between device-level characteristics and system-level design, this session aims to inspire new directions in EDA, architecture, and AI hardware research.
