RT Y Combinator：本周Paper Club的主题是推理。对于每花费一个FLOP来训练一个模型，每年大约有六个用于服务它——并且...
RT Y Combinator: This week’s Paper Club is all about inference. For every FLOP spent training a model, roughly six go to serving it each year — and ...
Garry TanAI2026-10-24
RT Y Combinator 这周的Paper Club聚焦于推理。对于每花费一FLOP（浮点运算次数）来训练一个模型，每年大约有六个FLOP用于服务该模型——在过去十年中，这部分正确实施产生了超过2200亿美元的市场价值。但推理远不止调用model.forward。每个标记都必须通过复杂的路由、缓存、调度、内核、内存和硬件堆栈，同时平衡速度、成本和质量。我们邀请了来自@togethercompute、@ollama、@luminal_ai和@sgl_project & @radixark的研究人员和构建者，从端到端带我们了解这个堆栈。我们将看到标记是如何从一个API请求到GPU再到返回的，为什么同一个开源模型在不同服务者那里表现不同，以及新的编译器和缓存方法如何使推理更快、更高效。00:00 - 推理为何比训练更重要08:24 - 三种推理创新类型12:14 - 丹·富（@realDanFu）：推理请求的生命周期42:18 - 杰夫·摩根（@jmorgan）：并非所有开源模型标记都相同1:04:31 - 乔·费奥蒂（@joefioti）：用搜索而非启发式编译模型1:30:30 - 邢亮生（@lsyincs）：SGLang和代理时代的内存
原文
RT Y CombinatorThis week’s Paper Club is all about inference. For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade.But inference is much more than calling model.forward. Every token has to move through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality.We brought together researchers and builders from @togethercompute, @ollama, @luminal_ai, and @sgl_project & @radixark to take us through that stack from end to end. We’ll see how a token gets from an API request to a GPU and back, why the same open model can perform differently depending on who serves it, and how new approaches to compilers and caching are making inference faster and more efficient.00:00 - Why inference matters more than training08:24 - Three kinds of inference innovation12:14 - Dan Fu (@realDanFu): the lifecycle of an inference request42:18 - Jeff Morgan (@jmorgan): not all open model tokens are the same1:04:31 - Joe Fioti (@joefioti): compiling models with search, not heuristics1:30:30 - Liangsheng Yin (@lsyincs): SGLang and memory for the agent era