Skip to content

See GPU memory before it breaks your training.

Stormlog is an open-source profiler that gives PyTorch, TensorFlow, and JAX teams real-time GPU memory visibility, leak detection, diagnostics, and exportable timelines across CLI, Python API, and Textual TUI workflows — now with inference endpoint profiling.

Stormlog overview
Works withPyTorchTensorFlowJAXOpenAI-compatible inferenceCLIPython APITextual TUIJSON exportCSV exportHTML reports

A product surface built around real debugging pressure.

The goal is not just to collect numbers. Stormlog helps teams see GPU memory as it shifts, isolate signals worth acting on, and move from guesswork to repeatable workflow.

Watch memory shift while training is still running.

Track allocation, peak usage, and reserved memory in one place instead of stitching together shell commands and printouts.

Real-time monitoring

Follow GPU allocation as it changes mid-epoch, not after the crash report lands.

Threshold alerts

Apply warning and critical limits so risky runs surface immediately instead of after hours of wasted compute.

Interactive TUI

Inspect platform info, live tracking, exports, and diagnostics without opening a browser.

Pinpoint growth patterns before they become OOM crashes.

Move from vague symptoms to concrete signals you can act on, including suspicious allocation growth and distributed anomalies.

Leak detection

Identify suspicious growth patterns and isolate where memory starts drifting run over run.

Artifact diagnostics

Load exported snapshots and compare them later to trace distributed or intermittent issues with context intact.

Timeline views

Generate timeline plots and HTML artifacts to show how memory behaved across the full workload.

Fit Stormlog into the stack you already have.

Adopt the profiler incrementally — from quick CLI sessions and deeper Python instrumentation in training code, to load-driven profiling of OpenAI-compatible inference endpoints.

CLI automation

Start monitoring or diagnostics sessions from the terminal without reworking your whole training loop.

Python hooks

Use decorators, context managers, and programmatic sessions when you need tighter profiling control.

CPU-compatible workflows

Prepare and test profiling routines before moving them onto production GPU infrastructure.

Inference endpoint profiling

Drive controlled load against any OpenAI-compatible Chat Completions endpoint and capture latency, throughput, and device-memory results.

A terminal-native workspace that still feels like a product.

Monitoring controls, visualization exports, diagnostics, and CLI-driven actions in a single interface.

Quick startOverview
Overview

Overview

Orient new users with platform details, keyboard shortcuts, and a fast path into every Stormlog surface.

Profile what your serving stack actually delivers.

The stormlog infer command group drives controlled load against OpenAI-compatible Chat Completions endpoints and reports the numbers you need to size, tune, and compare deployments.

vLLMSGLangTensorRT-LLMMLX-LMHosted gateways
  • Latency percentiles

    End-to-end latency and TTFT percentiles for both streaming and non-streaming responses.

  • Throughput under load

    Requests/sec, output tokens/sec, and total tokens/sec under configurable concurrency.

  • Accurate token accounting

    Server usage metadata with a tokenizer fallback so token counts stay trustworthy across providers.

  • Peak device memory

    Peak sampled device memory captured when system telemetry is available alongside latency and throughput.

Read the inference docs
stormlog infer · OpenAI-compatible
bash
stormlog infer profile \
  --base-url http://localhost:8000/v1 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --concurrency 1,4,8 \
  --input-tokens 512,2048 \
  --requests 50 \
  --output artifacts/infer.jsonl

Instrument, observe, diagnose, export, optimize.

Integrate Stormlog, watch a run live, capture useful evidence, and apply fixes before the next training cycle wastes more GPU time.

01

Instrument

Add Stormlog to the workload you care about, from lightweight decorators to deeper session-based profiling.

step 01
from stormlog import profile

@profile(track_tensors=True, detect_leaks=True)
def train_epoch(model, dataloader):
    for batch in dataloader:
        loss = model(batch)
        loss.backward()
02

Observe

Launch the TUI or a CLI session to watch allocation, peak memory, and alerts while the training run is alive.

step 02
$ stormlog monitor --pid 12345
┌─ Live GPU Memory ──────────────────────┐
│ Allocated  16.2 / 24.5 GB              │
│ Peak       19.8 / 24.5 GB              │
│ Alerts     None                        │
└────────────────────────────────────────┘
03

Diagnose

Inspect spikes, suspicious growth, and anomaly indicators before the next restart cycle begins.

step 03
[WARN] suspicious growth detected
tensor: grad_cache
change: +128MB over 50 iterations
signal: growth beyond threshold
04

Export

Ship artifacts into CI, review threads, or follow-up debugging sessions instead of relying on memory alone.

step 04
$ stormlog export --format json --output run.json
$ stormlog export --format html --output run.html

✓ timeline written
✓ diagnostics artifact saved
05

Optimize

Use the evidence to fix leaks, restore the intended batch size, and avoid repeat OOM failures in future runs.

step 05
Before: OOM at batch_size=64
After: batch_size=64 stable again
Peak allocated: 2.04 GB → 0.09 GB

✓ 50 epochs completed
✓ zero OOM interruptions

Reactive debugging vs. instrumented visibility.

Drag the divider to compare guesswork against a workflow with live monitoring, anomaly signals, and exported evidence.

With Stormlogillustrative session

$ stormlog monitor --pid 12345

Allocated 16.2 / 24.5 GiB

Peak 19.8 / 24.5 GiB

✓ live alerts enabled

[WARN] suspicious growth detected

signal: grad_cache +128MB

reason: repeated growth over threshold

✓ export diagnostics artifact

After fixing the leak

batch_size = 64 ✓ stable again

peak allocated: 2.04 GiB → 0.09 GiB

zero OOM interruptions across 50 epochs

Without Stormlogillustrative session

$ python train.py

Epoch 9/50... training

Epoch 10/50... training

RuntimeError: CUDA out of memory while allocating 2.4 GiB

$ nvidia-smi

| 23476 MiB / 24564 MiB |

Which tensor grew? Which step spiked?

Fallback strategy

batch_size = 64 → OOM

batch_size = 32 → unstable

batch_size = 16 → slow but survives

What's new in Stormlog

v0.3.9Sep 1, 2026

MLflow integration for experiment tracking and model registry integration, mirroring Weights & Biases exporter capabilities for unified ML workflow observability.

MLflow experiment tracking exporter

Stormlog now exports memory profiles and profiling metadata to MLflow, enabling integrated tracking of GPU memory alongside model metrics, parameters, and artifacts in your MLflow experiments.

  • Log memory traces and summary statistics directly to MLflow runs
  • Automatic model registry integration for memory profiles
  • Feature parity with Weights & Biases exporter for experiment tracking
  • Tag and filter runs by memory profiles in MLflow UI
python
from stormlog.exporters import MLflowExporter
from stormlog.pytorch import GPUMemoryProfiler
import mlflow

mlflow.start_run()

profiler = GPUMemoryProfiler()
exporter = MLflowExporter()

with profiler.profile_context("training"):
    # Your training loop
    for batch in dataloader:
        model(batch).backward()

exporter.export(profiler.trace(), "gpu_memory_profile")
mlflow.end_run()
Read the docs

Credibility comes from the repo, the docs, and the people shipping it.

Stormlog's proof is the public codebase, the published package, the documentation footprint, and the maintainers who keep the project moving.

Maintainers

Core maintainers who set direction, review changes, and keep Stormlog production-ready.

Contributors

Everyone who has shipped code to the repository, synced live from GitHub.

Questions, ideas, or a workload we should profile?

Open a discussion on GitHub so the answer stays searchable for the next person with the same question. For anything that does not belong in public, email a maintainer directly.

Trace memory, profile inference, and keep training runs stable.

Use the docs to get started, inspect the repository, or install Stormlog from PyPI for your next training run or inference benchmark.