LLM Inference Optimization
Generation is bandwidth bound, not compute bound. Almost every optimization follows from that one fact.
-
-
Mixture of Experts Models
Every token activating every parameter is a choice, not a law. MoE unmakes it.
Generation is bandwidth bound, not compute bound. Almost every optimization follows from that one fact.
Every token activating every parameter is a choice, not a law. MoE unmakes it.