What Stormlog Is, How It Works, and Why We’re Building It

A training job can look healthy for twenty minutes. Loss goes down, throughput holds, the GPU shows as busy. Meanwhile the memory footprint is quietly changing. Maybe a list somewhere keeps a reference to every step’s output. Maybe one rank in an eight-GPU job starts drifting while the other seven stay flat. Then the job dies with an out-of-memory error, and the only evidence you have is the stack trace from the moment of death.

Inference has its own version of this. An endpoint can hit its latency target with one request in flight and behave very differently with thirty-two. The quick test you ran before deploying doesn’t tell you where the curve bends, or why.

In both cases you’re debugging from a symptom. The cause happened earlier, in a place nobody was recording. Stormlog is our attempt to make that earlier behavior inspectable and reproducible: measure it while it happens, keep the evidence after the process exits, and make it possible to look at the same evidence twice.

I work on Stormlog with Silas Asamoah and Derrick Dwamena. It’s open source (MIT), written in Python, and it covers PyTorch, TensorFlow, JAX, and OpenAI-compatible inference endpoints. This page is the explanation I wanted to be able to link to: what it is, how the pieces fit, and where its edges are.

The page is long, and it’s built so you can stop early. The first sections explain the problem and the mental model with very little code. The middle covers how Stormlog is put together. The end has runnable examples, an honest list of what Stormlog isn’t, and the roadmap. If you’d rather start with commands, go to Try it.

You don’t need to know CUDA to follow the rest of this page. A handful of ideas cover most of it.

None of these has to look wrong until the last step. Reserved memory is allowed to stay high. Allocated memory can creep for hours. A job fails when one request for one block can’t be satisfied, and that depends on the peak and on how the free space is arranged, not on how the run looked a minute earlier.

Step through five situations below. The percentages are illustrative. They’re there to show the idea, not to describe a particular GPU.

What Stormlog records here, for the runtimes that expose it, is the shape of the climb instead of only the final error. What each backend exposes differs, and the section on what Stormlog can see gets specific.

Stormlog is an open-source toolkit for profiling GPU memory and inference behavior. It measures a workload, writes what it saw to local files, and gives you several ways to read those files again: Python APIs, command-line tools, and a terminal interface.

That sentence covers a lot of surface, so here is what the v0.4.0 release contains, with the caveat that matters for each part.

People open Stormlog for different reasons. These are the ones we design for. Each card ends with a fragment of the workflow that person would run.

Stormlog gives you two kinds of instrument, and choosing between them is mostly a matter of which question you’re asking. A profiler answers a bounded question: what happened inside this operation? A tracker answers a time-based one: what happened over time?

In practice you often start with a tracker, because you don’t yet know where to look. Once the timeline shows when something changes, a profiler around the suspect step tells you more about that step. The two are different code paths in the package, not one tool with two modes.

On a machine without a supported GPU, GPUMemoryProfiler and MemoryTracker raise a RuntimeError. Swap in CPUMemoryProfiler or CPUMemoryTracker to check your instrumentation on a laptop. They offer the same methods (profile_function, get_summary, phase), but the CPU profile result names the function name rather than function_name.

Every Stormlog workflow follows one path. A workload produces measurements. A profiler, tracker, or collector reads them. They’re normalized into events that carry an identity. Those events are written as artifacts. Analysis, plots, and the terminal interface read the artifacts.

There are two lanes because inference measures different things. A training job reports memory counters from inside the process. An inference run mostly observes a server from the outside, as a client, and records requests. The lanes share the ideas that matter (local files, identity, analysis later) but not the record format. We keep that distinction visible on purpose: inference artifacts are infer.* JSONL records, not TelemetryEventV4.

Two things the diagram can’t show. The TUI isn’t a separate analysis engine: it reuses tracker sessions for live data and the same event model for artifacts. And nothing in the path needs a hosted service.

The code is organized as four packages, and the boundaries follow what you’d guess from the names.

• stormlog holds the PyTorch profiler and tracker, the CPU fallbacks, telemetry normalization and the session contract, the analyzers, the visualizer, the device collectors, local query, the TUI, and inference.
• stormlog.tensorflow holds the TensorFlow profiler, tracker, analyzer, visualizer, and runtime diagnostics. It has no TUI of its own.
• stormlog.jax holds the JAX profiler, tracker, diagnostics, analyzer, visualizer, and runtime helpers.
• stormlog.infer profiles OpenAI-compatible endpoints. It’s separate from the framework tools because the endpoint might be backed by PyTorch, vLLM, SGLang, TensorRT-LLM, MLX-LM, or a hosted gateway.

The stormlog command opens a Textual app. It adapts live tracker data through a tracker session and loads saved artifacts as TelemetryEventV4 records, the same records the CLI reads. Plot export reuses the visualizer. That’s why a diagnose bundle written by a CI job can be opened in the TUI Diagnostics tab by someone else later, and why a bug fix in an analyzer shows up in both places.

On the PyTorch side, device memory comes from a collector, and every collector answers three questions: what is the current sample, what can you measure, and what are you called. CUDA, ROCm, and MPS each have one. The tracker checks every sample against the collector’s declaration. A populated field the collector said it can’t provide counts as a collector failure.

Declaring capabilities lets Stormlog leave a gap as a gap. The easy move for a profiler is to fill missing counters with zeros so charts keep drawing, but a zero is a claim: it says the allocator holds nothing. Stormlog has moved away from that more than once. Always-on tracking exports health events instead of synthetic zero samples (0.3.0). JAX statistics the runtime can’t provide are marked unavailable instead of reported as zero (0.3.8). Device-only tracking never fabricates allocator, fragmentation, history, or attribution findings (0.3.10).

“Memory” is several measurements at different layers, and a backend may expose some of them and not others. It helps to keep the layers apart.

That table is why “supports CUDA, ROCm, MPS, TensorFlow, and JAX” is an incomplete sentence. Support means different things on each. TensorFlow and JAX sit outside the PyTorch collector contract and read their own runtimes:

• TensorFlow runtimes can be CUDA, ROCm, Metal, or CPU. Counters can depend on the runtime on Metal. tfmemprof info prints build and runtime diagnostics.
• JAX device memory is read through jax.Device.memory_stats() after an XLA sync. It appears only when the runtime reports bytes_in_use. Process memory is reported separately.

The point of the layering isn’t to apologize for gaps. It’s that an investigation can use the layers that exist. A device-only runtime still gives you a device-used timeline, sessions, and query. It just can’t give you fragmentation, and it says so.

A Stormlog investigation tends to have five stages. They aren’t rigid, and you’ll often loop between the middle ones.

The thing to notice is the line between stages two and three. Until the process exits, the evidence is a set of numbers in memory. After, it’s files.

Those commands run on a CPU-only host too, using the CPU fallback, which makes them a reasonable first check of your setup before you point them at a GPU.

Telemetry is only useful across tools if they agree on what an event is. Stormlog has one canonical event and one session contract, and the tracker, the CLI, diagnose bundles, OOM bundles, and the TUI all use them.

The current event is TelemetryEventV4. Version 2, version 3, and recognized legacy records are upgraded to it on load, so older artifacts still open.

If you reuse one sink directory across runs, captures stay separate by session_id. Analysis defaults to the newest completed session, then the newest interrupted one, then the newest incomplete one, and you can pick another with --session-id.

Artifacts are what make the rest of this useful after the job is gone. In plain terms, an artifact is a file or directory holding what Stormlog saw, in a form something else can read.

That gives a failure five properties it usually lacks:

• Reloadable. gpumemprof analyze ./live_sink reads a whole directory, not just the last run.
• Shareable. Send the directory. The other person sees the same events.
• Comparable. stormlog query summary groups the same metric by session or by rank.
• Suitable for CI. A fixed exit-code table and a report.json verdict, so a job can branch without parsing text. See the report contract.
• Reviewable after the process exits. An OOM bundle holds the events that led to the failure, not only the failure.

Session identity is what holds this together. Without it, a directory with five runs in it is one pile of events. With it, a diagnose bundle’s manifest names its session, an OOM bundle points back at the run that produced it, and the TUI can switch between captures instead of merging them. The artifacts article goes through the file layouts in detail, so I won’t repeat them here.

Everything in the core workflow works on files you choose, in directories you choose. stormlog query reads them with no database. The CLI and the TUI run locally. Nothing in the path above sends profiling data to a hosted service.

Some things do leave the machine, and they’re opt-in. Weights & Biases and MLflow exports exist as optional extras. Inference profiling sends requests to the endpoint you point it at, which is the point of it. Stormlog doesn’t record the API key for an inference run, and it stores a cache-reset URL without credentials or query string.

Here’s a loop where the bug is one line that looks like bookkeeping.

The fix is usually small, such as appending output.detach().cpu(), or not keeping the output at all. The hard part is noticing. A tracker running next to this loop records allocated memory on every interval, and the timeline shows the difference between the two runs immediately.

gpumemprof analyze looks at that timeline for growth. Its leak heuristic reports only consistently positive growth, which is why a healthy run that climbs and then levels off isn’t flagged.

Stormlog can tell you that memory is climbing and when it started. It can’t tell you which line holds the reference. For allocator-level evidence on CUDA there’s an opt-in native history mode that records allocator history and attaches best-effort pointer-to-tensor attribution to a diagnose or OOM bundle. The leak walkthrough runs a measured investigation end to end, with the hardware and numbers.

One GPU number for a whole job is a sum or an average, and both can hide the interesting rank. If one of four ranks drifts, the average moves by a quarter of what that rank did. The chart below makes the point visible: toggle the average and compare it with rank 2.

Four pieces make this investigation possible.

• Rank identity. Telemetry carries job_id, rank, local_rank, and world_size. They’re inferred from common launcher environment variables, or set with options such as --job-id and --rank on gpumemprof track.
• Aligned telemetry. gpumemprof analyze merges per-rank timelines. With --visualization it writes a cross-rank timeline plot.
• Loading all ranks at once. The TUI Diagnostics tab loads rank artifacts, keeps ranks separate, and renders per-rank timelines and a rank table.
• First-cause suspects. The analyzer ranks which rank and phase spiked first. These are ranked heuristics. When phases overlap across threads, Stormlog marks the attribution ambiguous instead of guessing.

The distributed diagnostics article walks through the rank-aware workflow in detail.

Stormlog started with training memory, and it isn’t only that anymore. Inference raises a different class of questions, and most of them are about behavior under load rather than a single peak.

• Time to first token (TTFT) and end-to-end latency
• Throughput, and how it changes with concurrency
• Token counts, and whether they came from the server or an estimate
• Failures, timeouts, and rejected requests
• Workload shape: arrival pattern, prompt lengths, shared prefixes
• Cache state, and whether the cache was cold
• Device memory while all of that happens
• What the serving engine itself was doing

The tool for it is stormlog infer. It sends controlled traffic to any endpoint that accepts an OpenAI-style Chat Completions request, which covers PyTorch servers, vLLM, SGLang, TensorRT-LLM, MLX-LM, and hosted gateways, and it records what a client can observe.

These are in v0.4.0 and documented in the inference guide:

• A workload matrix over concurrency, input length, and output length, streaming or not, with warmup requests recorded but excluded from analysis.
• Controlled arrivals: closed loop, fixed rate, seeded Poisson, bursts, and replay of a recorded trace. Open-loop runs also report latency from each request’s intended arrival, so time spent waiting for a slot stays visible.
• Prompt control: repeat, unique, or shared-prefix prompts, and a way to ask for a cold prefix cache by calling a reset URL before each case.
• A workload record with a seed and a digest, so the same workload sent to two engine configurations can be recognized as the same.
• Client-observed latency percentiles, TTFT for streaming responses, throughput, failure rate, and token counts with their source recorded on every request.
• An optional collector, stormlog infer collect-server, that runs on the serving host and records server process memory and whole-device or MIG memory. The analyzer joins it to a case only when server identity and clocks line up, and it never sums values from different collectors.
• A fixed exit-code table, so a CI job can tell “no request succeeded” apart from a usage error.

A client can see that latency rose. It can’t see why. The active roadmap (#210) is about closing that gap by joining five kinds of evidence:

Two caveats keep this honest. A batch is shared by many requests, so Stormlog’s correlation records represent shared execution through membership. They keep measured batch duration separate from any estimated per-request cost, and don’t assign a shared kernel to one request. And an aggregate metrics scrape stays aggregate evidence. Exact request-to-iteration attribution needs worker instrumentation, which is a separate piece of work.

What exists beyond v0.4.0 is on the development branch and listed under “Unreleased” in the changelog: scraping vLLM’s Prometheus metrics during a run, receiving vLLM request spans, and importing a vLLM execution hook’s log into request, iteration, and membership records. That’s vLLM only, and it isn’t in a release yet.

Native tools are good, and you should keep using them. Different tools answer different layers of the problem.

What Stormlog adds is mostly about everything around the measurement: a common workflow across frameworks, normalized telemetry, artifacts that survive the process, session-aware investigation, Python, CLI, and TUI paths to the same data, automated diagnostics, repeatable runs, CI-friendly output, and inference workload profiling. That’s a different job from reading a kernel timeline.

• A replacement for every vendor GPU profiler.
• A hosted observability platform.
• A root-cause oracle. Its analyzers are heuristics that point at evidence, and findings carry confidence and limits.
• A guarantee that every backend exposes the same counters.
• A reason to skip the native tools for your framework or runtime.
• An LLM that guesses what went wrong. The measurements and labels don’t depend on a model.

It’s easy to blur “works today”, “being built”, and “we’d like to find out”. Here they’re kept apart. The three columns are the same split the project uses, and the inference roadmap itself says its milestones describe outcomes, not release versions or dates. I’m following that: nothing below has a ship date.

The contract for relating a request to shared server iterations and GPU activity shipped in 0.3.10 (#211). It reports iteration elapsed time, summed activity duration, and merged GPU interval time as three different numbers. The rule behind it: shared execution isn’t falsely assigned to one request. Only the collector side is still being built, and the first target is vLLM.

#227 describes a local UI for investigating a slow or failed inference run through its raw requests and metrics, with every displayed value traceable to a field or a calculation. It’s designed to sit on Stormlog’s existing artifacts and report contracts behind a small service boundary, not to reimplement analysis in the frontend. A prototype exists (PR #228) with explicitly synthetic fixtures. It’s a design and data-contract exercise, not something you can install.

#118 asks whether an optional native collector can provide evidence, or lower collection cost, that the existing PyTorch and Nsight route can’t. Candidates include CUPTI activity collection, USDT or eBPF probes, and programmable GPU probes. None is a dependency of Stormlog, none is committed, and a negative result is a valid way for that issue to end.

Install the package, then add extras for the runtime you use.

Then a ten-second capture, an analysis, a diagnose bundle, and a look at what was saved. This runs on a CPU-only laptop too.

For inference, point the profiler at any OpenAI-compatible endpoint you’re allowed to send traffic to:

To see everything in one place, pip install "stormlog[tui,torch]" and run stormlog. Docs, source, and the package page:

• Documentation, including the production cookbook and the CPU compatibility guide
• GitHub, where issues and the roadmap live
• stormlog on PyPI
• Getting started, the step-by-step version of this section

This page is the map. These are the places it points to, and what each one teaches.