https://vaibhav016.github.io/blog/2023/10/01/ssh-keys-cluster-setup-demystified/ 2023-10-01T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/03/10/distributed-training-part-1-batch-size-hierarchy/ 2026-03-10T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/03/10/distributed-training-part-2-ddp-fsdp-and-data-sharding/ 2026-03-10T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/03/10/distributed-training-part-3-surviving-frontier/ 2026-03-10T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/03/22/attention-residuals-kimi-fixing-transformers-blind-spot/ 2026-03-22T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/05/29/flow-based-llm-part-1-generation-as-transport/ 2026-05-29T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/05/30/flow-based-llm-part-2-tokens-on-the-simplex/ 2026-05-30T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/05/30/flow-based-llm-part-3-jumps-not-velocities/ 2026-05-30T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/05/30/flow-based-llm-part-4-one-process-all-along/ 2026-05-30T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/05/30/flow-based-llm-part-5-how-fast-how-far/ 2026-05-30T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/06/interrogating-the-transformer-part-1-one-pass-through-the-block/ 2026-09-06T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/06/interrogating-the-transformer-part-2-attention-and-the-square-root/ 2026-09-06T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/06/interrogating-the-transformer-part-3-residuals-and-the-norm/ 2026-09-06T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/06/interrogating-the-transformer-part-4-where-am-i/ 2026-09-06T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/06/interrogating-the-transformer-part-5-what-the-ffn-knows/ 2026-09-06T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/06/interrogating-the-transformer-part-6-what-changed-since-2017/ 2026-09-06T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/07/interrogating-the-gpu-part-1-where-the-time-goes/ 2026-09-07T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/07/interrogating-the-gpu-part-2-the-roofline/ 2026-09-07T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/07/interrogating-the-gpu-part-3-inside-the-memory-hierarchy/ 2026-09-07T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/07/interrogating-the-gpu-part-4-where-the-bytes-go/ 2026-09-07T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/07/interrogating-the-gpu-part-5-flashattention/ 2026-09-07T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/07/interrogating-the-gpu-part-6-the-2026-toolkit/ 2026-09-07T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/07/interrogating-the-gpu-part-7-building-at-scale/ 2026-09-07T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-moe-part-1-why-make-a-model-sparse/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-moe-part-2-how-a-token-picks-its-experts/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-moe-part-3-keeping-the-experts-busy/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-moe-part-4-the-bill-for-sparsity/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-moe-part-5-what-is-unsolved/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-moe-part-6-the-question-set/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-parallelism-part-1-why-one-gpu-cannot-do-it/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-parallelism-part-2-splitting-the-batch/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-parallelism-part-3-splitting-a-matrix-multiply/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-parallelism-part-4-splitting-the-layers/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-parallelism-part-5-long-sequences-and-sparse-models/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-parallelism-part-6-putting-them-together/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-parallelism-part-7-the-question-set/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-the-kv-cache-part-1-why-the-cache-exists/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-the-kv-cache-part-2-how-many-bytes/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-the-kv-cache-part-3-making-it-smaller/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-the-kv-cache-part-4-how-a-server-keeps-it/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-the-kv-cache-part-5-what-it-does-to-throughput/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/blog/2026/09/09/interrogating-the-kv-cache-part-6-the-question-set/ 2026-09-09T00:00:00+00:00 https://vaibhav016.github.io/pages/blog/ https://vaibhav016.github.io/pages/cv/ https://vaibhav016.github.io/ https://vaibhav016.github.io/pages/journey/ https://vaibhav016.github.io/pages/ml-interview-2026/ https://vaibhav016.github.io/pages/projects/ https://vaibhav016.github.io/pages/publications/ https://vaibhav016.github.io/googlebdc6f52a5d55c856.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/attnres-visuals/01-prenorm-dilution.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/attnres-visuals/02-standard-vs-attnres.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/attnres-visuals/03-block-attnres.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/attnres-visuals/04-vertical-attention-family.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/attnres-visuals/vertical_attention.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/blog-visuals/01-ddp-overview.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/blog-visuals/02-fsdp-layer-lifecycle.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/blog-visuals/03-ring-vs-tree-allreduce.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/blog-visuals/04-data-sharding.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/blog-visuals/05-communication-stack.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/blog-visuals/06-multinode-launch.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/blog-visuals/07-full-optimizer-step-trace.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/flow-llm-visuals/01-transport-paths.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/flow-llm-visuals/02-marginal-vs-conditional.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/flow-llm-visuals/03-live-training.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/flow-llm-visuals/p2-simplex.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/flow-llm-visuals/p3-ctmc.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/flow-llm-visuals/p4-generator.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/flow-llm-visuals/p5-race.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/gpu-visuals/g1-four-clocks.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/gpu-visuals/g2-roofline.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/gpu-visuals/g3-hierarchy.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/gpu-visuals/g4-memory-budget.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/gpu-visuals/g5-flashattention.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/gpu-visuals/g6-toolkit.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/gpu-visuals/g7-at-scale.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/kv-visuals/k1-why-the-cache-exists.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/kv-visuals/k2-the-byte-budget.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/kv-visuals/k3-shrinking-the-cache.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/kv-visuals/k4-paged-and-shared.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/kv-visuals/k5-throughput.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m1-comparison-preview.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m1-intuition-preview.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m1-parameter-map.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m1-plate-preview.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m1-primer.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m1-total-vs-active.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m1c-block-flow.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m2-routing.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m3-load-balancing.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m4-systems.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/moe-visuals/m5-frontier.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/parallel-visuals/p1-two-budgets.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/parallel-visuals/p2-data-parallel.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/parallel-visuals/p3-tensor-parallel.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/parallel-visuals/p4-pipeline-bubble.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/parallel-visuals/p5-context-and-experts.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/parallel-visuals/p6-topology.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part1-blog-visuals/01-gpu-hierarchy.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part1-blog-visuals/02-batch-size-hierarchy.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part1-blog-visuals/03-auto-compute-scaling.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part1-blog-visuals/04-progress-bar-mystery.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part1-blog-visuals/05-token-accounting.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part3-blog-visuals/01-nvme-checkpoint-sidecar.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part3-blog-visuals/02-self-resubmitting-jobs.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part3-blog-visuals/03-node-health-checks.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part3-blog-visuals/04-mfu-calculator.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/part3-blog-visuals/05-offline-env-wandb.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p1-forward-pass.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p2-scaled-dot-product.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p3-residual-gradients.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p4a-bag-of-tokens.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p4b-sinusoid-clocks.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p4c-add-vs-rotate.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p4d-rope.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p4e-stretch.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p4f-causal-leak.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p5-ffn-memory.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/images/blog/transformer-visuals/p6-modern-diff.html 2026-10-06T01:16:15+00:00 https://vaibhav016.github.io/src/docs/CV_2026.pdf 2026-10-06T01:16:16+00:00