ReplaySSM: Cache SSM Inputs, Not State
State-space models (SSMs) maintain a fixed-size recurrent state, but reading and writing it at every decoding step can bottleneck hybrid models. This eager state management causes …
Optimizing Compute Core Assignment for Dynamic Batch Inference in AI Inference Accelerator
Modern AI inference accelerators offer high-performance and power-efficient computations for machine learning models. Most accelerators employ static inference to enhance …