Perlmutter: the first non-ALCF system⚓︎

Everything in the failover stack was built and validated on ALCF hardware — PBS, Intel XPU, PALS. Perlmutter (NERSC) is none of those: SLURM, NVIDIA A100, no PALS. That makes it the first real test of whether any of it generalizes, and the answer is mostly yes, with one bug that only a different scheduler could have exposed.

For first-time setup (allocation, modules, install), see ezpz on Perlmutter; this page is about what was verified there and what the numbers are.

What was validated⚓︎


`get_machine()` → `Perlmutter`, scheduler → SLURM	✅ already mapped
Bad-node provenance on SLURM `nid` names	✅
Node-identity matching (`-hsnN`/domain stripping)	✅ degrades correctly
`failover/topology.py` on real output	✅
268 failover tests	✅

The topology parser was written from Aurora XPU logs and read NVIDIA SLURM output with no changes:

['nid001329'][device='cuda'][node=0/1][local_rank=0/3][rank=0/7]
['nid002813'][device='cuda'][node=1/1][local_rank=0/3][rank=4/7]

hosts: ['nid001329', 'nid002813']   world: 8
host0 owns: range(0, 4)   host1 owns: range(4, 8)
rank 8 -> None            # out of range, not a guess

The missing-patterns warning also fired here on a real third machine, auto-detected — previously it had only ever run against an explicit machine="polaris" in a test:

WARNING no bad-node patterns registered for machine 'perlmutter' — every
failover on this machine will be a BLIND rotation (the sick node may stay
in the allocation). Registered: aurora, sunspot

That is worth reading as a real limitation, not a nicety: failover on Perlmutter is currently always blind. Nothing here has the shepherd died from signal 9 / rank N died from signal 9 signatures the ALCF machines emit, so the scraper cannot name a culprit.

The bug only SLURM could show⚓︎

--auto-retry swaps a bad node out by rewriting the active hostfile. On PBS that works because mpiexec is handed the file. On SLURM the hostfile was used only to count lines for -N:

before swap: srun -u --verbose -N2 -n8 --gpus-per-node=4
after  swap: srun -u --verbose -N2 -n8 --gpus-per-node=4
IDENTICAL despite different hosts? True

So the loop detected the bad node, picked a spare, rewrote the hostfile, recorded provenance in bad_nodes.txt — and relaunched with a command that said nothing about which nodes to use. SLURM re-picked from the same allocation and could hand back the sick node, while every artifact claimed a swap had happened. Worse than unsupported: the postmortem actively misleads.

Fixed in #235 by passing --nodelist=<hostfile> — the file path, not an inlined list, because AutoRetryConfig assembles the command once and mutates the hostfile in place between attempts. srun documents --nodelist={<node_name_list>|<filename>}. Verified in a live launch:

--nodelist=/pscratch/.../nodefile-57520568
srun: jobid 57520568: nodes(2):`nid[001329,002813]'
srun: launching StepId=57520568.0 on host nid001329, 4 tasks: [0-3]
srun: launching StepId=57520568.0 on host nid002813, 4 tasks: [4-7]

exclude_hosts → -x came with it, because -w is a floor rather than a fence: its man page says the job "will contain all of these hosts and possibly additional hosts as needed".

AmSC benchmarks⚓︎

agpt-2b, tp=1, bs=1, seq 2048, 20 iterations, torch 2.13.0.

	tok/s	tok/s/GPU	MFU	TFLOPS
1 node / 4 A100	40,363	10,091	30.3%	94.6
2 node / 8 A100	51,244	6,406	19.2%	60.1

Rows are in experiments/perlmutter/results/runs.csv, generated by ezpz export-amsc, which self-labeled system as Perlmutter from run provenance with no override.

Scaling 1→2 nodes is 1.27× against an ideal 2×, i.e. 64% parallel efficiency. That is the honest headline: a 2B model at seq 2048 with bs=1 gives each GPU very little work per step, so the FSDP all-gather between nodes is a large fraction of step time. Larger batches or sequences would scale better; this config is chosen to match the Aurora/Sunspot series, not to flatter the interconnect.

Three earlier rows here were wrong by 8×, and nothing errored

The first numbers published on this page — 2.3% MFU at bs=1, and the conclusion that "batch size is the lever and compile is noise" — were all measured with NCCL silently falling back to TCP. Same config, the plugin the only difference:

	tok/s	MFU
2 node, no OFI plugin	6,175	2.31%
2 node, OFI plugin	51,244	19.2%

8.3×. With TCP, adding a second node made the job 6.5× slower than one node. Every conclusion drawn from those runs was an artifact:

"MFU under 5% because a 2B model at bs≤2 cannot fill an A100" — one node reaches 30% at bs=1. The GPUs were waiting on the network, not starved for work.
"bs 1→2 is 1.93×, compile is 1.04×" — both measured entirely inside the degraded regime, and neither ratio survives.
The Sunspot comparison, which implied Perlmutter was ~6× slower per GPU.

The invalid rows are kept in runs.csv marked -NO-OFI-PLUGIN with an error note rather than deleted, since the failure mode is more useful than the absence.

Failover on SLURM⚓︎

--auto-retry works here, but only after two SLURM-specific bugs were fixed. Validated on job 57541913: 4 nodes split 2 active + 2 spare, ranks SIGKILLed on active[1].

bad_nodes.txt:   nid001544  scraped  attempt=1
active.hostfile: nid001304
                 nid002821
attempts: 2

The victim is named with evidence (scraped, not a blind guess), removed from the active set, replaced by a spare, and the job relaunched.

The kill deliberately targets active[1], never active[0] — a blind rotation always evicts active[0], so killing it would let a guess pass as identification. That design is what exposed three stacked bugs on PBS (node-kill postmortem).

The two bugs⚓︎

A killed node read as WALLTIME (#238). srun reports a SIGKILLed rank as rc=143 — the same code as a clean walltime expiry — and emits none of the PALS signatures the classifier uses to tell them apart. Every killed node on SLURM therefore collapsed to "the clock ran out" and nothing failed over. The fix matches what srun actually prints:

srun: error: nid001321: tasks 4-7: Killed       <- the victim
srun: error: nid001320: tasks 0-3: Terminated   <- the cascade

Killed only, never Terminated — the latter is what every rank gets at normal teardown, so matching it would retire a node on every expiring job.

The rewritten hostfile never reached srun (#235) — see above.

Still worth knowing⚓︎

src/ezpz/bin/failover.sh has no SLURM branches at all. The bash failover library is PBS-only, so Python's --auto-retry is the only route on Perlmutter.

SLURM discovery functions⚓︎

Every ezpz.slurm helper is unit-tested against mocked subprocess output. Job 57540535 ran them against a real scheduler for the first time, and cross-checked the nodelist against scontrol:

OK  get_slurm_jobid_of_active_job:     57540535
OK  get_nodelist_from_slurm_jobid:     ['nid001288','nid001568','nid002657','nid003548']
OK  get_slurm_running_jobs:            ['57540534','57540535']
OK  get_slurm_nodefile_of_active_job:  .../nodefile-57540535
OK  build_launch_cmd:                  srun -u --verbose -N4 -n16 --gpus-per-node=4
nodelist matches scontrol: True (4 hosts)

LoRA⚓︎

First validation on anything other than Intel XPU. agpt-2b, 2 nodes / 8 A100, bs=1 seq2048, 20 iters:

config	tok/s	MFU	TFLOPS
full fine-tune (rank 0)	52,060	19.56%	61.0
LoRA r16, attn only	90,878	23.50%	73.3
LoRA r32, attn+mlp	82,639	21.72%	67.8
LoRA r64, attn+mlp	82,633	22.12%	69.0
LoRA r8 / r16, attn+mlp	—	—	—

Small ranks with attn,mlp hang

r8 and r16 with --lora-target attn,mlp hang at the first forward pass — 7 seconds in, then silence until killed. Zero iterations, no traceback: the signature of a collective deadlock. r32 and r64 use the same targets and train fine, so it is the rank that matters, not the targeting.

Reproduced in 3 jobs each, including one with the config alone in its allocation, which rules out batch position and neighbours. Tracked in #239.

1.75× and 1.59× over the full fine-tune — the expected shape, since frozen base weights mean no optimizer state and a much smaller backward. The baseline's 19.56% independently matches the 19.25% from a separate 2-node run, so the sweep produces comparable numbers.

Environment notes⚓︎

Three Perlmutter-specific hazards, each of which cost a job — or, in the first case, five jobs and a set of published numbers — before being understood.

Load nccl/2.24.3, or NCCL silently uses TCP

The single most important line in any multi-node job here:

module load nccl/2.24.3

It sets NCCL_NET="AWS Libfabric" and puts libnccl-net on LD_LIBRARY_PATH. Without it NCCL finds no network plugin and falls back to TCP over the host stack — 8.3× slower inter-node, and no error, no warning, nothing in the log. It just runs, slowly.

It also breaks things that are not obviously about throughput: the hf_trainer benchmark hung at step 0 and died with TCPStore recvValue failed between nodes, which reads as a dataloader or rendezvous problem.

Assert it rather than trusting it, the way experiments/perlmutter/*.sbatch now do:

echo "=== NCCL_NET=${NCCL_NET:-<unset>} ==="
case ":${LD_LIBRARY_PATH:-}:" in
    *nccl*plugin*) echo "plugin: yes" ;;
    *) echo "WARNING: no nccl plugin — expect TCP fallback" ;;
esac

--compile needs torch ≥ 2.13

On NERSC's pytorch/2.8.0, --compile aborts every rank before iteration 1:

torch._dynamo.exc.Unsupported: Attempted to call function marked as skipped
  qualname: disable
from user code: .../torch/_dynamo/external_utils.py:70, in inner

external_utils.inner is the autograd-hook trampoline — Dynamo tracing into the FSDP2 backward hook and refusing. Distinct from the _MaskPartial caveat --compile documents, and this ran --tp 1 anyway. Tracked as #236; fixed by pytorch/2.13.0, where the signature disappears entirely.

pytorch/2.13.0 needs MPICH_GPU_SUPPORT_ENABLED=0

The 2.13.0 module exports MPICH_GPU_SUPPORT_ENABLED=1 while its mpi4py is not linked against the GTL library, so every rank segfaults during init:

MPICH ERROR [nid008192] - Abort(-1): MPIDI_CRAY_init:
GPU_SUPPORT_ENABLED is requested, but GTL library is not linked
srun: error: nid008201: tasks 4-5,7: Segmentation fault

This presents as rc=139 with zero iterations and looks like a compile failure if you only read the exit code. The 2.8.0 build does not hit it. Disabling GPU-aware MPI costs nothing here — collectives go over NCCL and MPI is used only for rank discovery.

Reproducing⚓︎

# torch 2.13.0 venv, built on the NERSC module so CUDA and the
# Slingshot stack stay matched (a pip wheel would not be):
python -m venv --system-site-packages .venv-213     # from pytorch/2.13.0
.venv-213/bin/pip install -e . --no-deps            # keep the module's torch

sbatch --export=ALL,BS=2,COMPILE=yes,VENV=.venv-213 \
    experiments/perlmutter/amsc_bench.sbatch

The job echoes its resolved config and torch version into the log:

=== config: model=agpt-2b bs=2 seq=2048 iters=20 compile=yes ===
=== venv=.venv-213 torch=2.13.0+cu130 ===

That is not decoration. COMPILE="${COMPILE:---compile}" substitutes on empty as well as unset, so --export=ALL,COMPILE= silently kept compile on and one job "testing the uncompiled path" ran compiled. The echo is what caught it.

AmSC llm-finetuning: measured, deliberately not published⚓︎

The AmSC training/llm-finetuning benchmark runs on Perlmutter — Llama-3.2-1B, 2×4 A100, block_size 8192: 21.82 s, 60,060 tok/s, 20/20 steps.

That number is not in the AmSC repo, and should not be until the benchmark's own specification is runnable. Getting it required two departures from the command as written:

--fsdp_config '{"fsdp_version": 1}'. FSDP2 refuses to put a shared parameter in two wrap groups, and Llama-3.2-1B ties embed_tokens to lm_head, so every rank dies before step 1 (#237). Neither hf_trainer nor the benchmark's own torchrun reference passes a wrap policy, so the spec as written cannot run this model under FSDP2 on torch 2.13.
A cached, non-streaming dataset. --streaming fetches per batch over the network; on compute nodes it hangs at step 0 until the walltime kills it. The spec says --streaming.

A row produced by a modified command would read as "this benchmark was run" when it was not. The right order is to raise both with the benchmark's maintainers and publish once the spec is fixed.

When it does land it belongs at benchmarks/training/llm-finetuning/results/Perlmutter/, matching the sibling vit-weather benchmark — and it will need column translation, not just a different path. vit-weather publishes three columns (configuration_name, samples_per_sec, iters_per_sec); ezpz export-amsc emits eleven with different names. See the warning in ezpz export-amsc for what that repo actually contains today.