My Weird Prompts
Daniel Rosehill
Podcast
Episodes
Listen, download, subscribe
Same GPU, Same Weights, Different Answers
Two servers, same GPU, same weights, same prompt — different output. This episode digs into Daniel's thought experiment: three providers running vLLM, SGLang, and TensorRT-LLM on identical hardware, where the customer can't choose the engine. Does the inference engine actually change speed, cost, and the answers themselves? We walk through the architecture of all three — PagedAttention, RadixAttention, and the TensorRT-LLM backend overhaul that removed TensorRT — then look at the benchmark data, where the fastest engine changes with the load, and a 2026 CISPA paper showing that swapping only the backend moved GSM8K scores by up to 16.6 percentage points on the same weights. If you rent endpoints and never see the middle layer, this one's for you. Key sources: • Thinking Machines Lab (Horace He), 2025-09-10 — https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ • SGLang Team, 2025-09-22 (updated 2025-09-24) — https://www.lmsys.org/blog/2025-09-22-sglang-deterministic/ • vLLM repo — https://github.com/vllm-project/vllm Episode 5899 — myweirdprompts.com/n/5899 (permanent short link: myweirdprompts.com/616043)
My Weird Prompts RSS Feed
