Vraj Patel BATCHER

you are here · 00 / Top

ML systems · LLM inference · GPU kernels

Vraj Patel

I build fast LLM inference: GPU kernels, paged KV-cache serving, and quantization for GPUs and edge NPUs.

CS at UMass Dartmouth, graduating May 2027. Seeking ML systems, inference and kernel roles. Based in Massachusetts.

Everything here is built from scratch — CUDA and Triton kernels, FlashAttention-2 forward and backward, INT8 quantization — and benchmarked against the production baseline it's meant to replace, including where mine loses.

See the work GitHub LinkedIn Email Résumé

Measured results

1,030 tok/sPaged serving, 32 concurrent requests on GPT-2 small, up from 82 at 1
3.5xHugging Face with KV cache, at 32 concurrent requests
46.4%INT8 static memory cut vs bitsandbytes 29.3%
60.3%Of FP32 peak, Triton matmul at N=2048
Request queue 0

pinned

running · click to read

    waiting

      continuous · padding waste 0% · simulation

      Decode iterations iter 0
      prefill decode, one tick per token static-batch padding

      Each request is one of my projects. Click a bar or a request to read it.

      Toy numbers: the gauges illustrate the scheduler, not a benchmark. Real measurements are in each project, on an RTX 5080 Laptop GPU (Blackwell, sm_120 · 16 GB · CUDA 12.9).

      Live gauges simulation

      batch size

      0/ 4 slots

      KV cache usage

      0%of 64 blocks

      tokens / iteration

      0tok

      static batching would pad

      0%of slot-iters

      Continuous batching pads none: a free slot is refilled the next iteration.

      Who I am

      01 · About me

      I'm Vraj, a computer science student at UMass Dartmouth graduating in May 2027. I got hooked on the question of why a model that works in a notebook is slow in production, and that pulled me down the stack: from PyTorch into Triton and CUDA, from FLOPs into memory bandwidth, from a single request into scheduling hundreds.

      I like building the thing from scratch first, then benchmarking it honestly against the production baseline, including where mine loses. Right now I'm working on a senior capstone with General Dynamics Mission Systems and climbing a CUDA SGEMM ladder on my RTX 5080. Off the keyboard, I play tennis.

      • Based in Massachusetts
      • Graduating May 2027
      • Seeking ML systems, inference and kernel roles

      Now · Oct 2026

      • CUDAFinishing the blackwell-gemm CUDA ladder: kernel 03, 1D thread tiling.
      • CapstoneGDMS senior capstone: hybrid transformer + state space models vs an EKF.
      • Open sourcevLLM PR #61901 in review.

      Projects, newest focus first

      02 · Work

      Six projects, each built from scratch and measured. Open any card for the full write-up, tables and links.

      Paged inference engine

      paged-inference-engine · Aug 2026 · Python, Triton

      A mini vLLM-style server built without referencing vLLM source: block-based paged KV cache, continuous batching scheduler, a FlashAttention-2 Triton kernel that reads and writes through a block table, and INT8 KV cache quantization. Serves GPT-2 small, checked against Hugging Face with a parity harness.

      • Throughput, 1 → 32 concurrent requests (GPT-2 small)82 → 1,030 tok/s
      • vs Hugging Face with KV cache, 32 requests3.5x
      • Prefill vs Hugging Face, log-probwithin ~0.005
      • Honest result: vLLM 0.21.0 is faster, at 32 requests~23x
      Repo: paged-inference-engine Write-up: paged inference engine

      GPU inference stack

      cpp-gpu-inference · Triton, PyTorch

      FlashAttention-2 forward and backward in Triton, LLM.int8()-style INT8 post-training quantization, and a tiled Triton matmul profiled with Nsight Compute. SDPA is still faster at long sequences: causal block-skipping isn't implemented yet.

      • INT8 static memory cut (bitsandbytes 29.3%)46.4%
      • Tiled matmul, N=204860.3% FP32 peak
      • FA-2 peak memory, seq 8192 (matches SDPA)~288 MB
      • L1TEX uncoalesced gap, by tiling43.13% → 3.36%
      Repo: cpp-gpu-inference Write-up: GPU inference stack

      Edge quantization on Snapdragon

      edge-quantization-snapdragon · Qualcomm AI Hub

      A from-scratch 48M-param decoder (6 layers, d_model 512) running 100% on the Galaxy S26 Hexagon NPU. W8A8 is 1.8x faster with 3x less memory, and breaks the model; the sensitivity sweep shows why.

      • FP16 · 260 MB2.5 ms · 99.5% agree
      • W8A16 · 93 MB2.4 ms · 60.9% agree
      • W8A8 · 88 MB1.4 ms · 24.6% agree
      • Perplexity: + LayerNorm outputs in INT8+129%
      Repo: edge-quantization-snapdragon

      Blackwell SGEMM ladder

      blackwell-gemm · Oct 2026 · CUDA · in progress

      A CUDA SGEMM ladder from scratch on the RTX 5080, 10+ kernels from naive to tensor-core mma.sync and CuTe, each profiled with Nsight Compute against cuBLAS. Now on kernel 03: shared-memory / 1D thread tiling.

      • Done01 naive · 02 coalescing
      • Current03 1D thread tiling
      • Per-kernel GFLOP/smeasuring
      Repo: blackwell-gemm

      Modern LLM from scratch

      modern-llm-from-scratch · PyTorch

      The building blocks of a modern LLM, each implemented and tested on its own: RMSNorm, SwiGLU, RoPE, grouped-query attention and mixture-of-experts.

      Repo: modern-llm-from-scratch

      EN–DE transformer

      en-de-transformer · PyTorch

      A 6+6 encoder-decoder (8 heads, d_model 512) trained on 200K Tatoeba EN–DE pairs. It became the substrate for the FlashAttention-2 and INT8 work.

      Write-up: transformer shape trace

      Request log · every project and PR as a request

      IDProjectHeadline metricStatus

      Newest first

      03 · Experience

      Robotics, ocean science and simulation, before inference.

      1. Sep 2026–present

        Senior Capstone, General Dynamics Mission Systems (GDMS)

        Hybrid state estimation models (transformers and state space models) benchmarked against an Extended Kalman Filter (EKF).

      2. Summer 2026

        ML Engineer Intern, Fleet Robotics

        Fixed sim-to-real heading drift in a MuJoCo RL policy and retrained the SAC policy; built AWS/Snowflake telemetry pipelines and Streamlit dashboards.

        LinkedIn post about ML Engineer Intern, Fleet Robotics (opens in a new tab)

      3. Mar 2026

        Blue Tech externship, 2026

        CMVS simulated autonomous UUV sonar mine detection (MATLAB/Simulink); led a cohort workshop on AI-assisted development with Claude.

        LinkedIn post about Blue Tech externship, 2026 (opens in a new tab)

      4. Sep–Nov 2025

        Research Software Engineering Intern, SMAST

        Parallelized NOAA offshore-wind spatial simulations 2x.

        LinkedIn post about Research Software Engineering Intern, SMAST (opens in a new tab)

      5. Jun–Aug 2025

        Data Science Intern, SMAST

        38 years of estuary data; report turnaround cut from 3 weeks to 10 minutes.

        LinkedIn post about Data Science Intern, SMAST (opens in a new tab)

      6. 2025

        Blue Tech externship, 2025
        1st place, 2025 cohortLinkedIn post about Blue Tech externship, 2025 (opens in a new tab)

      Upstream contributions

      04 · Open source

      Two pull requests, neither merged yet.

      vLLM · #61901 · Oct 2026

      [Bugfix][LoRA] Validate dtype at config time for Triton LoRA kernels

      In review View PR #61901 on GitHub

      CobraML2 · #39 · Oct 2026

      Fused-attention toggle + GPT-2 benchmark (1.43x faster)

      Open, in review View PR #39 on GitHub

      Long-form write-ups

      05 · Writing

      Send a request · the queue is short

      07 · Contact

      vrajpatel.jobs@gmail.com

      Email me GitHub LinkedIn Résumé

      Study notes · PDF

      08 · Notes

      Notes I’ve written while working through this material. Each opens as a PDF in a new tab.

      Deep learning

      • Loading notes…