GPU Kernel Engineering
An agent skill for CUDA kernels is a packaged set of GPU-architecture knowledge, integration patterns, and code templates that a coding agent like Claude or Codex reads before writing a kernel turning “write me an RMSNorm kernel” into a buildable, benchmarked PyTorch extension in one pass. Hugging Face shipped exactly this inside its kernels library in February 2026, and the results are a genuinely useful data point for anyone who spends their days chasing milliseconds out of transformer inference. This post walks through how the workflow works, what the benchmarks actually show, and because reading about kernels is very different from debugging one at 1 a.m. what it looks like to hand-write and integrate a custom kernel yourself against a production inference engine like vLLM.
Why custom kernels still matter when everyone is talking about compilers?
It would be reasonable to assume that torch.compile, kernel fusion, and CUDA graphs have made hand-written CUDA mostly obsolete for inference work. That is not quite true. Compilers are excellent at fusing sequences of standard elementwise and reduction operations automatically, but they still launch a large number of small kernels for a single forward pass through a transformer, and each launch carries real dispatch overhead. Purpose-built kernels for operations like RMSNorm, RoPE, GEGLU, and fused attention scoring can bypass that overhead entirely by exploiting vectorized memory access, warp-level shuffle reductions, and tensor-core paths that a general-purpose compiler will not always find on its own. That is why frameworks like vLLM, SGLang, and TensorRT-LLM all ship a layer of custom CUDA and Triton kernels underneath their higher-level scheduling logic rather than relying on eager PyTorch or torch.compile alone.
The catch is that writing those kernels correctly is a genuinely narrow skill. You need to know the shared memory size and compute capability of the target GPU generation, understand the specific module hierarchy and normalization conventions of the library you are integrating with, get the PyTorch C++ bindings right so torch.compile can still trace through your op, and manage a build matrix that spans multiple CUDA, PyTorch, and Python versions. Most of that knowledge lives in scattered documentation, GPU architecture whitepapers, and hard-won Stack Overflow threads rather than in any one place which is precisely the kind of narrow, high-stakes, well-documented-but-scattered domain that turns out to be a good fit for an agent skill.
What the agent skill actually packages?
Hugging Face’s approach separates two concerns that used to be bundled together: writing a kernel and distributing it. The Kernel Hub already solved distribution you can pull a pre-compiled kernel with a single get_kernel call and skip the build step entirely. What was still missing was help for the person who has to write the kernel in the first place, and that is the gap the cuda-kernels skill fills.
The skill itself is compact: roughly 550 tokens of structured instructions in a SKILL.md file, backed by a set of reference documents and working examples that the agent greps and globs on demand rather than loading all at once. It covers architecture-aware optimization guidance for H100, A100, and T4 GPUs, including their differing compute capabilities and shared memory budgets; integration patterns and known pitfalls specific to the transformers and diffusers libraries; vectorized kernel templates for BF16, FP16, and FP32; benchmarking scaffolding for both isolated micro-benchmarks and end-to-end pipeline comparisons; and a path into the Kernel Hub via get_kernel. Installing it is a single command that drops the skill into .claude/skills/cuda-kernels/, where Claude Code and similar agents pick it up automatically, with equivalent flags for Codex and OpenCode.
Once installed, a prompt as simple as asking for a vectorized RMSNorm kernel for a specific model and GPU target is enough for the agent to read the skill, select the right architecture parameters, write the CUDA source, generate the PyTorch bindings, configure the build file, and produce a benchmark script end to end, without the developer hand-holding each step.
The workflow, visually
Diagram showing the agent-skill kernel development workflow: developer prompt, agent reads the skill, agent generates the kernel project, build and benchmark, publish to Kernel Hub
Does it actually work? The benchmark numbers
Hugging Face tested this against two real targets rather than toy examples: a video generation pipeline from diffusers (LTX-Video) and a large language model from transformers (Qwen3-8B, which has 65 RMSNorm modules spread across 32 layers). Both were benchmarked on an H100 80GB card in BF16.
For Qwen3-8B, the agent-written RMSNorm kernel beat the PyTorch baseline at every sequence length tested, and the margin widened as sequence length grew from roughly 1.58x at 128 tokens up to 2.47x at 8,192 tokens, for an average of about 1.94x and a measured bandwidth efficiency of roughly 22% of the H100’s theoretical peak.
Line chart showing custom RMSNorm kernel speedup scaling from 1.58x at 128 tokens to 2.47x at 8192 tokens on Qwen3-8B, H100
That scaling pattern makes intuitive sense: RMSNorm is a memory-bandwidth-bound operation, so as sequence length grows, a well-vectorized kernel gets proportionally more benefit from reduced memory traffic relative to the fixed overhead of a generic PyTorch kernel launch. For long-context inference specifically, that is a meaningful chunk of latency recovered from a single normalization layer.
The LTX-Video case is a good reminder to keep isolated kernel wins in perspective. The custom RMSNorm kernel alone was about 1.88x faster than PyTorch in isolation, but RMSNorm only accounts for roughly 5% of total compute in that pipeline most of the time is spent in attention, linear projections, and VAE decode. So the end-to-end speedup from that single kernel type was a more modest 6%, and it composed cleanly with torch.compile rather than fighting it.
Bar chart showing end-to-end LTX-Video generation speedup: 1.00x baseline, 1.06x with generated kernels, 1.34x with torch.compile, 1.43x combining both
The practical takeaway is one that any inference engineer will recognize: a fast kernel for one operation and a compiler that fuses everything around it are not competing techniques, they compound. The agent-generated kernel and torch.compile stacked to 1.43x end to end, ahead of either technique alone.
A practitioner’s view: hand-writing a custom RMSNorm kernel for vLLM
Reading someone else’s agent-generated benchmark table is one thing. Getting a custom kernel to actually integrate cleanly with a real serving engine is a different, messier exercise, and it is worth walking through because the failure modes are rarely about the CUDA code itself.
I recently went through this directly, writing a custom RMSNorm kernel targeting vLLM with Qwen2.5-0.5B as the test model deliberately a small model, since the point was to validate the integration path on limited compute before worrying about scale. The kernel math itself was the easy part. The harder part was getting vLLM to actually dispatch to my custom operator instead of quietly falling back to its own implementation. vLLM routes operator calls through an internal IR dispatch system, and a monkey-patch that looks correct at the Python level can still get silently overridden by that dispatch priority configuration at runtime, so the kernel appears to load successfully while the benchmark numbers tell you nothing has actually changed. Tracking that down meant stepping through vLLM’s dispatch internals rather than the kernel code, which is not the kind of thing that shows up in a CUDA programming guide.
The second snag was more mundane but just as easy to lose an afternoon to: a chat template mismatch when testing against the base (non-instruction-tuned) variant of the model, which produced outputs that looked plausible enough to mask the fact that the input formatting was wrong in the first place. Between the two issues, the actual kernel-writing time was a small fraction of the total project time which tracks with what the Hugging Face team found and packaged into their skill in the first place: the CUDA source is rarely the bottleneck, the surrounding integration is. That is also the strongest case for a well-scoped agent skill: it will not save you from vLLM’s dispatch internals, but it does mean the boilerplate around bindings, build configuration, and benchmark scaffolding doesn’t have to be rebuilt by hand every time.
From experiment to something you can actually ship
Once a kernel works, publishing it so it is reusable is a separate, well-defined path. The agent-generated project already follows the layout expected by kernel-builder: CUDA source under kernel_src/, C++ bindings and a torch.ops registration under torch-ext/, and a build.toml describing the target CUDA capability. From there, kernel-builder’s Nix flake builds every required PyTorch and CUDA variant in one pass, the result gets pushed to a model repo on the Hugging Face Hub, and anyone downstream can load it with a single get_kernel call no local build, no CUDA toolkit required on their end, and the correct pre-compiled binary is selected automatically based on their Python, PyTorch, and CUDA versions.
That split the skill for development, the Hub for distribution is the more interesting structural idea underneath this whole workflow. It turns kernel writing from a one-off, per-team, per-model exercise into something closer to a shared library ecosystem, which is exactly what a broad set of concurrent 2026 research efforts on agentic kernel generation and benchmarking (from CUDA-focused RL agents to multi-agent kernel optimization systems) are converging on from different directions.
Where this gets hard for most engineering teams
The gap between a working benchmark script on a researcher’s H100 and a kernel that survives contact with a production serving stack, a fleet of mixed GPU generations, and an on-call rotation is real, and it is usually not a modeling problem it is a platform and infrastructure problem. Teams that have deep AI/ML research talent do not always have equally deep GPU systems talent sitting next to it, and the two skill sets rarely overlap as much as org charts assume.
This is the kind of gap that a product engineering partner is actually useful for, rather than a nice-to-have. 47Billion works with enterprise teams on exactly this handoff taking AI and ML capabilities that work in a research notebook and building the surrounding product engineering, data infrastructure, and deployment pipeline needed to run them reliably in production, through its AI/ML and end-to-end product development practice. For a team that has validated a custom kernel or inference optimization approach internally but does not want to build and maintain a dedicated GPU systems function to operationalize it, that is a more realistic path than either hiring a standalone kernel team or shipping the optimization as a permanent research artifact that never reaches production.
Closing thought
The interesting part of this shift is not that agents can write CUDA it is that a genuinely narrow, high-friction skill (GPU kernel engineering) is now accessible to teams that previously had no realistic path to it, provided the domain knowledge is packaged well. The remaining hard part, based on hands-on experience with vLLM specifically, is not the kernel. It is everything wrapped around it dispatch systems, build matrices, and the unglamorous debugging that never makes it into a benchmark table.
