ezpz benchmark⚓︎
Run all ezpz examples sequentially and generate a markdown report with timing, metrics, and environment information.
Usage⚓︎
# Run all examples
ezpz benchmark
# Run specific examples
ezpz benchmark --examples test fsdp vit
# Override model size (preset is passed through to each example)
ezpz benchmark --model small
ezpz benchmark --model xl # production-scale presets also work
ezpz benchmark --model extra-large # ...as do long-form aliases
# Custom output directory
ezpz benchmark --outdir outputs/my-benchmark
Options⚓︎
| Flag | Description |
|---|---|
--examples |
Space-separated list of examples to run (default: all) |
--model |
Model size preset passed through to each example (default: small). Each example resolves the preset against its own MODEL_PRESETS; sizes range debug → xxxl with long-form aliases (xlarge / extra-large, etc.). See each example's docs for per-file parameter scales. Note: agpt-2b and agpt-20b are fsdp_tp-only presets; passing them via ezpz benchmark --model agpt-2b will fail on the other four examples (test, fsdp, vit, diffusion), which don't define those keys. |
--outdir |
Output directory (default: outputs/benchmarks/{timestamp}) |
Available Examples⚓︎
test, fsdp, vit, fsdp_tp, diffusion, hf, hf_trainer
Output⚓︎
Each run creates a timestamped directory with:
outputs/benchmarks/20260330_091310/
env.json # Environment metadata (git, nodes, torch version, ...)
timings.csv # Per-example wall time and exit code
report.md # Markdown report with results table
test.log # Captured stdout/stderr for each example
fsdp.log
...
Example Output⚓︎
Running 7 example(s): test, fsdp, vit, fsdp_tp, diffusion, hf, hf_trainer
════════════════════════════════════════════════════════════════
[1/7] Running: test
ETA: ~5m 30s remaining
cmd: ezpz launch python3 -m ezpz.examples.test --model small
log: outputs/benchmarks/.../test.log
════════════════════════════════════════════════════════════════
✓ test completed in 1m 06s
...
════════════════════════════════════════════════════════════════
7/7 passed in 13m 51s
────────────────────────────────────────────────────────────────
✓ test 1m 06s
✓ fsdp 31s
✓ vit 42s
...
════════════════════════════════════════════════════════════════