Train ViT with FSDP on MNIST⚓︎
See:
Help⚓︎
--help
$ python3 -m ezpz.examples.vit --help
usage: ezpz.examples.vit [-h] [--img_size IMG_SIZE] [--batch_size BATCH_SIZE]
[--num_heads NUM_HEADS] [--head_dim HEAD_DIM]
[--hidden-dim HIDDEN_DIM] [--mlp-dim MLP_DIM]
[--dropout DROPOUT]
[--attention-dropout ATTENTION_DROPOUT]
[--num_classes NUM_CLASSES] [--dataset {fake,mnist}]
[--depth DEPTH] [--patch_size PATCH_SIZE]
[--dtype DTYPE] [--compile]
[--num_workers NUM_WORKERS] [--max_iters MAX_ITERS]
[--warmup WARMUP] [--attn_type {native,sdpa}]
[--cuda_sdpa_backend {flash_sdp,mem_efficient_sdp,math_sdp,cudnn_sdp,all}]
[--fsdp]
Train a simple ViT
options:
-h, --help show this help message and exit
--img_size IMG_SIZE, --img-size IMG_SIZE
Image size
--batch_size BATCH_SIZE, --batch-size BATCH_SIZE
Batch size
--num_heads NUM_HEADS, --num-heads NUM_HEADS
Number of heads
--head_dim HEAD_DIM, --head-dim HEAD_DIM
Hidden Dimension
--hidden-dim HIDDEN_DIM, --hidden_dim HIDDEN_DIM
Hidden Dimension
--mlp-dim MLP_DIM, --mlp_dim MLP_DIM
MLP Dimension
--dropout DROPOUT Dropout rate
--attention-dropout ATTENTION_DROPOUT, --attention_dropout ATTENTION_DROPOUT
Attention Dropout rate
--num_classes NUM_CLASSES, --num-classes NUM_CLASSES
Number of classes
--dataset {fake,mnist}
Dataset to use
--depth DEPTH Depth
--patch_size PATCH_SIZE, --patch-size PATCH_SIZE
Patch size
--dtype DTYPE Data type
--compile Compile model
--num_workers NUM_WORKERS, --num-workers NUM_WORKERS
Number of workers
--max_iters MAX_ITERS, --max-iters MAX_ITERS
Maximum iterations
--warmup WARMUP Warmup iterations (or fraction) before starting to
collect metrics.
--attn_type {native,sdpa}, --attn-type {native,sdpa}
Attention function to use.
--cuda_sdpa_backend {flash_sdp,mem_efficient_sdp,math_sdp,cudnn_sdp,all}, --cuda-sdpa-backend {flash_sdp,mem_efficient_sdp,math_sdp,cudnn_sdp,all}
CUDA SDPA backend to use.
--fsdp Use FSDP
Output⚓︎
Output on Sunspot
$ ezpz launch python3 -m ezpz.examples.vit
[2025-12-31 12:13:01,324304][I][ezpz/launch:396:launch] ----[🍋 ezpz.launch][started][2025-12-31-121301]----
[2025-12-31 12:13:02,176169][I][ezpz/launch:416:launch] Job ID: 12458339
[2025-12-31 12:13:02,176953][I][ezpz/launch:417:launch] nodelist: ['x1921c0s3b0n0', 'x1921c0s7b0n0']
[2025-12-31 12:13:02,177350][I][ezpz/launch:418:launch] hostfile: /var/spool/pbs/aux/12458339.sunspot-pbs-0001.head.cm.sunspot.alcf.anl.gov
[2025-12-31 12:13:02,178010][I][ezpz/pbs:264:get_pbs_launch_cmd] ✅ Using [24/24] GPUs [2 hosts] x [12 GPU/host]
[2025-12-31 12:13:02,178699][I][ezpz/launch:367:build_executable] Building command to execute by piecing together:
[2025-12-31 12:13:02,179082][I][ezpz/launch:368:build_executable] (1.) launch_cmd: mpiexec --envall --np=24 --ppn=12 --hostfile=/var/spool/pbs/aux/12458339.sunspot-pbs-0001.head.cm.sunspot.alcf.anl.gov --no-vni --cpu-bind=verbose,list:2-4:10-12:18-20:26-28:34-36:42-44:54-56:62-64:70-72:78-80:86-88:94-96
[2025-12-31 12:13:02,179891][I][ezpz/launch:369:build_executable] (2.) cmd_to_launch: python3 -m ezpz.examples.vit
[2025-12-31 12:13:02,180622][I][ezpz/launch:433:launch] Took: 1.46 seconds to build command.
[2025-12-31 12:13:02,180965][I][ezpz/launch:436:launch] Executing:
mpiexec
--envall
--np=24
--ppn=12
--hostfile=/var/spool/pbs/aux/12458339.sunspot-pbs-0001.head.cm.sunspot.alcf.anl.gov
--no-vni
--cpu-bind=verbose,list:2-4:10-12:18-20:26-28:34-36:42-44:54-56:62-64:70-72:78-80:86-88:94-96
python3
-m
ezpz.examples.vit
[2025-12-31 12:13:02,182157][I][ezpz/launch:443:launch] Execution started @ 2025-12-31-121302...
[2025-12-31 12:13:02,182600][I][ezpz/launch:139:run_command] Running command:
mpiexec --envall --np=24 --ppn=12 --hostfile=/var/spool/pbs/aux/12458339.sunspot-pbs-0001.head.cm.sunspot.alcf.anl.gov --no-vni --cpu-bind=verbose,list:2-4:10-12:18-20:26-28:34-36:42-44:54-56:62-64:70-72:78-80:86-88:94-96 python3 -m ezpz.examples.vit
cpubind:list x1921c0s7b0n0 pid 108722 rank 12 0: mask 0x1c
cpubind:list x1921c0s7b0n0 pid 108723 rank 13 1: mask 0x1c00
cpubind:list x1921c0s7b0n0 pid 108724 rank 14 2: mask 0x1c0000
cpubind:list x1921c0s7b0n0 pid 108725 rank 15 3: mask 0x1c000000
cpubind:list x1921c0s7b0n0 pid 108726 rank 16 4: mask 0x1c00000000
cpubind:list x1921c0s7b0n0 pid 108727 rank 17 5: mask 0x1c0000000000
cpubind:list x1921c0s7b0n0 pid 108728 rank 18 6: mask 0x1c0000000000000
cpubind:list x1921c0s7b0n0 pid 108729 rank 19 7: mask 0x1c000000000000000
cpubind:list x1921c0s7b0n0 pid 108730 rank 20 8: mask 0x1c00000000000000000
cpubind:list x1921c0s7b0n0 pid 108731 rank 21 9: mask 0x1c0000000000000000000
cpubind:list x1921c0s7b0n0 pid 108732 rank 22 10: mask 0x1c000000000000000000000
cpubind:list x1921c0s7b0n0 pid 108733 rank 23 11: mask 0x1c00000000000000000000000
cpubind:list x1921c0s3b0n0 pid 105486 rank 0 0: mask 0x1c
cpubind:list x1921c0s3b0n0 pid 105487 rank 1 1: mask 0x1c00
cpubind:list x1921c0s3b0n0 pid 105488 rank 2 2: mask 0x1c0000
cpubind:list x1921c0s3b0n0 pid 105489 rank 3 3: mask 0x1c000000
cpubind:list x1921c0s3b0n0 pid 105490 rank 4 4: mask 0x1c00000000
cpubind:list x1921c0s3b0n0 pid 105491 rank 5 5: mask 0x1c0000000000
cpubind:list x1921c0s3b0n0 pid 105492 rank 6 6: mask 0x1c0000000000000
cpubind:list x1921c0s3b0n0 pid 105493 rank 7 7: mask 0x1c000000000000000
cpubind:list x1921c0s3b0n0 pid 105494 rank 8 8: mask 0x1c00000000000000000
cpubind:list x1921c0s3b0n0 pid 105495 rank 9 9: mask 0x1c0000000000000000000
cpubind:list x1921c0s3b0n0 pid 105496 rank 10 10: mask 0x1c000000000000000000000
cpubind:list x1921c0s3b0n0 pid 105497 rank 11 11: mask 0x1c00000000000000000000000
[2025-12-31 12:13:08,706913][I][ezpz/dist:1501:setup_torch_distributed] Using torch_{device,backend}= {xpu, xccl}
[2025-12-31 12:13:08,709436][I][ezpz/dist:1366:setup_torch_DDP] Caught MASTER_PORT=45161 from environment!
[2025-12-31 12:13:08,710117][I][ezpz/dist:1382:setup_torch_DDP] Using torch.distributed.init_process_group with
- master_addr='x1921c0s3b0n0'
- master_port='45161'
- world_size=24
- rank=0
- local_rank=0
- timeout=datetime.timedelta(seconds=3600)
- backend='xccl'
[2025-12-31 12:13:08,711400][I][ezpz/dist:1014:init_process_group] Calling torch.distributed.init_process_group_with: rank=0 world_size=24 backend=xccl
[2025-12-31 12:13:09,470261][I][ezpz/dist:1727:setup_torch] Using device='xpu' with backend='xccl' + 'xccl' for distributed training.
[2025-12-31 12:13:09,471063][W][ezpz/dist:544:print_dist_setup] Using [24 / 24] available "xpu" devices !!
[2025-12-31 12:13:09,471499][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=0/1][rank=00/23][local_rank=00/11]
[2025-12-31 12:13:09,470671][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=0/1][rank=10/23][local_rank=10/11]
[2025-12-31 12:13:09,470709][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=1/1][rank=01/23][local_rank=01/11]
[2025-12-31 12:13:09,470724][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=0/1][rank=02/23][local_rank=02/11]
[2025-12-31 12:13:09,470717][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=1/1][rank=03/23][local_rank=03/11]
[2025-12-31 12:13:09,470725][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=0/1][rank=04/23][local_rank=04/11]
[2025-12-31 12:13:09,470729][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=1/1][rank=05/23][local_rank=05/11]
[2025-12-31 12:13:09,470727][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=0/1][rank=06/23][local_rank=06/11]
[2025-12-31 12:13:09,470702][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=1/1][rank=07/23][local_rank=07/11]
[2025-12-31 12:13:09,470697][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=0/1][rank=08/23][local_rank=08/11]
[2025-12-31 12:13:09,470703][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=1/1][rank=09/23][local_rank=09/11]
[2025-12-31 12:13:09,470729][I][ezpz/dist:1774:setup_torch] ['x1921c0s3b0n0'][device='xpu'][node=1/1][rank=11/23][local_rank=11/11]
[2025-12-31 12:13:09,474499][I][ezpz/dist:2039:setup_wandb] Setting up wandb from rank=0
[2025-12-31 12:13:09,474926][I][ezpz/dist:2040:setup_wandb] Using WB_PROJECT=ezpz.examples.vit
[2025-12-31 12:13:09,470772][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=0/1][rank=12/23][local_rank=00/11]
[2025-12-31 12:13:09,470811][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=1/1][rank=13/23][local_rank=01/11]
[2025-12-31 12:13:09,470827][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=0/1][rank=14/23][local_rank=02/11]
[2025-12-31 12:13:09,470866][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=1/1][rank=15/23][local_rank=03/11]
[2025-12-31 12:13:09,470869][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=0/1][rank=16/23][local_rank=04/11]
[2025-12-31 12:13:09,470813][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=1/1][rank=17/23][local_rank=05/11]
[2025-12-31 12:13:09,470869][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=0/1][rank=18/23][local_rank=06/11]
[2025-12-31 12:13:09,470871][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=1/1][rank=19/23][local_rank=07/11]
[2025-12-31 12:13:09,470827][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=0/1][rank=20/23][local_rank=08/11]
[2025-12-31 12:13:09,470825][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=1/1][rank=21/23][local_rank=09/11]
[2025-12-31 12:13:09,470874][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=0/1][rank=22/23][local_rank=10/11]
[2025-12-31 12:13:09,470870][I][ezpz/dist:1774:setup_torch] ['x1921c0s7b0n0'][device='xpu'][node=1/1][rank=23/23][local_rank=11/11]
wandb: Currently logged in as: foremans (aurora_gpt) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.23.1
wandb: Run data is saved locally in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/wandb/run-20251231_121309-g19jy6bl
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run snowy-hill-239
wandb: View project at https://wandb.ai/aurora_gpt/ezpz.examples.vit
wandb: View run at https://wandb.ai/aurora_gpt/ezpz.examples.vit/runs/g19jy6bl
[2025-12-31 12:13:10,974322][I][ezpz/dist:2069:setup_wandb] wandb.run=[snowy-hill-239](https://wandb.ai/aurora_gpt/ezpz.examples.vit/runs/g19jy6bl)
[2025-12-31 12:13:10,980450][I][ezpz/dist:2112:setup_wandb] Running on machine='SunSpot'
[2025-12-31 12:13:10,983391][I][examples/vit:509:main] Using native for SDPA backend
[2025-12-31 12:13:10,984013][I][examples/vit:535:main] Using AttentionBlock Attention with args.compile=False
[2025-12-31 12:13:10,984652][I][examples/vit:287:train_fn] asdict(config)={'img_size': 224, 'batch_size': 128, 'num_heads': 16, 'head_dim': 64, 'depth': 24, 'patch_size': 16, 'hidden_dim': 1024, 'mlp_dim': 4096, 'dropout': 0.0, 'attention_dropout': 0.0, 'num_classes': 1000}
[2025-12-31 12:14:34,029080][I][examples/vit:354:train_fn]
==========================================================================================
Layer (type:depth-idx) Output Shape Param #
==========================================================================================
VisionTransformer [128, 1000] 200,704
├─PatchEmbed: 1-1 [128, 196, 1024] 787,456
├─Dropout: 1-2 [128, 196, 1024] --
├─Identity: 1-3 [128, 196, 1024] --
├─Identity: 1-4 [128, 196, 1024] --
├─Sequential: 1-5 [128, 196, 1024] 201,547,776
├─Identity: 1-6 [128, 196, 1024] --
├─LayerNorm: 1-7 [128, 1024] 2,048
├─Dropout: 1-8 [128, 1024] --
├─Linear: 1-9 [128, 1000] 1,025,000
==========================================================================================
Total params: 203,562,984
Trainable params: 203,562,984
Non-trainable params: 0
Total mult-adds (G): 45.69
==========================================================================================
Input size (MB): 77.07
Forward/backward pass size (MB): 49532.61
Params size (MB): 813.45
Estimated Total Size (MB): 50423.13
==========================================================================================
[2025-12-31 12:14:34,032988][I][examples/vit:355:train_fn] Model size: nparams=0.91 B
[2025-12-31 12:14:34,038818][I][ezpz/dist:685:wrap_model] Wrapping model with: ddp
2025:12:31-12:14:34:(105486) |CCL_WARN| value of CCL_OP_SYNC changed to be 1 (default:0)
2025:12:31-12:14:34:(105486) |CCL_WARN| value of CCL_PROCESS_LAUNCHER changed to be pmix (default:hydra)
[2025-12-31 12:14:47,099101][I][ezpz/dist:685:wrap_model] Wrapping model with: ddp
[2025-12-31 12:14:47,296113][I][ezpz/history:220:__init__] Using History with distributed_history=True
[2025-12-31 12:14:47,312293][I][examples/vit:408:train_fn] Training with 24 x xpu (s), using torch_dtype=torch.bfloat16
[2025-12-31 12:15:32,154227][I][examples/vit:445:train_fn] iter=10 loss=7.111572 dt=0.744444 dtd=0.003141 dtf=0.021592 dto=0.698994 dtb=0.020717 loss/mean=7.036184 loss/max=7.148438 loss/min=6.926270 loss/std=0.051675 dt/mean=0.744634 dt/max=0.745104 dt/min=0.744115 dt/std=0.000000 dtd/mean=0.003989 dtd/max=0.005223 dtd/min=0.003121 dtd/std=0.000751 dtf/mean=0.021139 dtf/max=0.021592 dtf/min=0.020827 dtf/std=0.000188 dto/mean=0.698727 dto/max=0.699881 dto/min=0.697217 dto/std=0.000961 dtb/mean=0.020778 dtb/max=0.021557 dtb/min=0.020275 dtb/std=0.000320
[2025-12-31 12:15:33,015905][I][examples/vit:445:train_fn] iter=11 loss=7.032715 dt=0.702058 dtd=0.001719 dtf=0.022411 dto=0.657195 dtb=0.020732 loss/mean=7.011394 loss/max=7.086426 loss/min=6.919678 loss/std=0.041570 dt/mean=0.735905 dt/max=0.771187 dt/min=0.697317 dt/std=0.022530 dtd/mean=0.001865 dtd/max=0.002308 dtd/min=0.001698 dtd/std=0.000162 dtf/mean=0.021999 dtf/max=0.023057 dtf/min=0.021225 dtf/std=0.000596 dto/mean=0.691333 dto/max=0.726180 dto/min=0.652583 dto/std=0.022813 dtb/mean=0.020708 dtb/max=0.021126 dtb/min=0.020190 dtb/std=0.000228
[2025-12-31 12:15:33,747367][I][examples/vit:445:train_fn] iter=12 loss=6.982422 dt=0.717778 dtd=0.001721 dtf=0.023286 dto=0.671980 dtb=0.020791 loss/mean=7.006734 loss/max=7.068848 loss/min=6.895752 loss/std=0.042117 dt/mean=0.724329 dt/max=0.732337 dt/min=0.713690 dt/std=0.004574 dtd/mean=0.003245 dtd/max=0.005390 dtd/min=0.001609 dtd/std=0.001376 dtf/mean=0.023302 dtf/max=0.024685 dtf/min=0.021369 dtf/std=0.000678 dto/mean=0.677032 dto/max=0.684881 dto/min=0.669961 dto/std=0.003860 dtb/mean=0.020750 dtb/max=0.021848 dtb/min=0.020185 dtb/std=0.000450
[2025-12-31 12:15:34,556097][I][examples/vit:445:train_fn] iter=13 loss=7.114746 dt=0.742885 dtd=0.001825 dtf=0.023194 dto=0.697149 dtb=0.020716 loss/mean=7.035777 loss/max=7.121094 loss/min=6.938965 loss/std=0.043673 dt/mean=0.743002 dt/max=0.746045 dt/min=0.739701 dt/std=0.002211 dtd/mean=0.004097 dtd/max=0.005476 dtd/min=0.001825 dtd/std=0.001211 dtf/mean=0.022114 dtf/max=0.023244 dtf/min=0.021464 dtf/std=0.000590 dto/mean=0.696058 dto/max=0.699107 dto/min=0.692656 dto/std=0.002238 dtb/mean=0.020733 dtb/max=0.021334 dtb/min=0.020149 dtb/std=0.000345
[2025-12-31 12:15:35,310688][I][examples/vit:445:train_fn] iter=14 loss=7.011475 dt=0.720220 dtd=0.001705 dtf=0.022270 dto=0.675471 dtb=0.020774 loss/mean=7.039348 loss/max=7.114502 loss/min=6.937744 loss/std=0.041890 dt/mean=0.750122 dt/max=0.777821 dt/min=0.720220 dt/std=0.018705 dtd/mean=0.001850 dtd/max=0.002233 dtd/min=0.001625 dtd/std=0.000194 dtf/mean=0.022314 dtf/max=0.024492 dtf/min=0.021074 dtf/std=0.000813 dto/mean=0.705263 dto/max=0.730777 dto/min=0.675471 dto/std=0.018371 dtb/mean=0.020695 dtb/max=0.021149 dtb/min=0.020171 dtb/std=0.000279
[2025-12-31 12:15:36,066943][I][examples/vit:445:train_fn] iter=15 loss=7.011230 dt=0.735597 dtd=0.001686 dtf=0.022349 dto=0.690830 dtb=0.020732 loss/mean=7.028381 loss/max=7.112061 loss/min=6.949463 loss/std=0.039451 dt/mean=0.751743 dt/max=0.761701 dt/min=0.735432 dt/std=0.008930 dtd/mean=0.002519 dtd/max=0.004533 dtd/min=0.001567 dtd/std=0.000977 dtf/mean=0.022890 dtf/max=0.024537 dtf/min=0.021450 dtf/std=0.000851 dto/mean=0.705617 dto/max=0.715304 dto/min=0.690162 dto/std=0.008198 dtb/mean=0.020716 dtb/max=0.021369 dtb/min=0.020218 dtb/std=0.000304
[2025-12-31 12:15:36,829552][I][examples/vit:445:train_fn] iter=16 loss=7.066895 dt=0.728091 dtd=0.001674 dtf=0.023043 dto=0.682651 dtb=0.020723 loss/mean=7.036306 loss/max=7.118652 loss/min=6.950928 loss/std=0.040736 dt/mean=0.735283 dt/max=0.739360 dt/min=0.728091 dt/std=0.003266 dtd/mean=0.003385 dtd/max=0.005231 dtd/min=0.001674 dtd/std=0.001218 dtf/mean=0.022158 dtf/max=0.023252 dtf/min=0.021710 dtf/std=0.000435 dto/mean=0.688973 dto/max=0.694191 dto/min=0.682651 dto/std=0.003334 dtb/mean=0.020767 dtb/max=0.021937 dtb/min=0.020278 dtb/std=0.000388
[2025-12-31 12:15:37,624977][I][examples/vit:445:train_fn] iter=17 loss=7.056885 dt=0.739866 dtd=0.001641 dtf=0.022526 dto=0.694913 dtb=0.020786 loss/mean=7.031006 loss/max=7.094238 loss/min=6.946289 loss/std=0.042612 dt/mean=0.747859 dt/max=0.765443 dt/min=0.731697 dt/std=0.012958 dtd/mean=0.002078 dtd/max=0.003025 dtd/min=0.001545 dtd/std=0.000425 dtf/mean=0.022239 dtf/max=0.023510 dtf/min=0.021278 dtf/std=0.000711 dto/mean=0.702795 dto/max=0.719565 dto/min=0.686761 dto/std=0.012629 dtb/mean=0.020747 dtb/max=0.021336 dtb/min=0.020204 dtb/std=0.000284
[2025-12-31 12:15:38,414813][I][examples/vit:445:train_fn] iter=18 loss=7.021973 dt=0.730126 dtd=0.001684 dtf=0.022472 dto=0.685149 dtb=0.020821 loss/mean=7.009206 loss/max=7.126221 loss/min=6.934570 loss/std=0.041062 dt/mean=0.757921 dt/max=0.776361 dt/min=0.729369 dt/std=0.015862 dtd/mean=0.002343 dtd/max=0.004546 dtd/min=0.001629 dtd/std=0.000985 dtf/mean=0.022413 dtf/max=0.023448 dtf/min=0.021651 dtf/std=0.000472 dto/mean=0.712472 dto/max=0.730281 dto/min=0.685116 dto/std=0.015107 dtb/mean=0.020693 dtb/max=0.021261 dtb/min=0.020149 dtb/std=0.000297
[2025-12-31 12:15:39,209452][I][examples/vit:445:train_fn] iter=19 loss=7.029053 dt=0.733423 dtd=0.001668 dtf=0.022388 dto=0.688487 dtb=0.020880 loss/mean=7.010173 loss/max=7.071777 loss/min=6.963379 loss/std=0.031614 dt/mean=0.757083 dt/max=0.784014 dt/min=0.728249 dt/std=0.018796 dtd/mean=0.001842 dtd/max=0.002232 dtd/min=0.001557 dtd/std=0.000210 dtf/mean=0.022281 dtf/max=0.023509 dtf/min=0.021485 dtf/std=0.000466 dto/mean=0.712357 dto/max=0.738712 dto/min=0.684110 dto/std=0.018800 dtb/mean=0.020603 dtb/max=0.021067 dtb/min=0.020249 dtb/std=0.000253
[2025-12-31 12:15:39,959071][I][examples/vit:445:train_fn] iter=20 loss=6.989258 dt=0.729999 dtd=0.001656 dtf=0.023220 dto=0.684263 dtb=0.020860 loss/mean=6.997142 loss/max=7.051025 loss/min=6.924805 loss/std=0.037109 dt/mean=0.736245 dt/max=0.744330 dt/min=0.723712 dt/std=0.006234 dtd/mean=0.002480 dtd/max=0.003943 dtd/min=0.001558 dtd/std=0.000659 dtf/mean=0.023104 dtf/max=0.024808 dtf/min=0.021568 dtf/std=0.000875 dto/mean=0.689984 dto/max=0.697561 dto/min=0.679231 dto/std=0.005511 dtb/mean=0.020676 dtb/max=0.021082 dtb/min=0.020291 dtb/std=0.000217
[2025-12-31 12:15:40,714336][I][examples/vit:445:train_fn] iter=21 loss=6.949463 dt=0.738463 dtd=0.001676 dtf=0.022789 dto=0.693238 dtb=0.020760 loss/mean=7.004436 loss/max=7.096680 loss/min=6.928467 loss/std=0.038073 dt/mean=0.744181 dt/max=0.747653 dt/min=0.738463 dt/std=0.002405 dtd/mean=0.003571 dtd/max=0.005418 dtd/min=0.001676 dtd/std=0.001278 dtf/mean=0.022411 dtf/max=0.023831 dtf/min=0.021547 dtf/std=0.000525 dto/mean=0.697513 dto/max=0.700923 dto/min=0.693238 dto/std=0.001938 dtb/mean=0.020686 dtb/max=0.021187 dtb/min=0.020283 dtb/std=0.000298
[2025-12-31 12:15:41,522736][I][examples/vit:445:train_fn] iter=22 loss=7.014893 dt=0.755023 dtd=0.001776 dtf=0.025431 dto=0.706979 dtb=0.020837 loss/mean=7.004486 loss/max=7.110352 loss/min=6.913818 loss/std=0.038965 dt/mean=0.748414 dt/max=0.766116 dt/min=0.732838 dt/std=0.012672 dtd/mean=0.002078 dtd/max=0.003655 dtd/min=0.001651 dtd/std=0.000579 dtf/mean=0.022676 dtf/max=0.025431 dtf/min=0.021047 dtf/std=0.001224 dto/mean=0.702984 dto/max=0.719493 dto/min=0.687241 dto/std=0.012053 dtb/mean=0.020675 dtb/max=0.021195 dtb/min=0.020129 dtb/std=0.000270
[2025-12-31 12:15:42,268381][I][examples/vit:445:train_fn] iter=23 loss=7.009033 dt=0.732504 dtd=0.001736 dtf=0.022508 dto=0.687491 dtb=0.020769 loss/mean=7.007579 loss/max=7.079834 loss/min=6.947510 loss/std=0.039306 dt/mean=0.757627 dt/max=0.778756 dt/min=0.731312 dt/std=0.016388 dtd/mean=0.002094 dtd/max=0.003084 dtd/min=0.001562 dtd/std=0.000526 dtf/mean=0.022488 dtf/max=0.023932 dtf/min=0.021376 dtf/std=0.000707 dto/mean=0.712397 dto/max=0.732798 dto/min=0.685533 dto/std=0.016056 dtb/mean=0.020648 dtb/max=0.021215 dtb/min=0.020275 dtb/std=0.000233
[2025-12-31 12:15:43,132486][I][examples/vit:445:train_fn] iter=24 loss=7.014648 dt=0.791617 dtd=0.001653 dtf=0.025728 dto=0.743375 dtb=0.020861 loss/mean=7.015188 loss/max=7.098389 loss/min=6.912109 loss/std=0.045387 dt/mean=0.763487 dt/max=0.791617 dt/min=0.728050 dt/std=0.020098 dtd/mean=0.001822 dtd/max=0.002207 dtd/min=0.001626 dtd/std=0.000180 dtf/mean=0.022584 dtf/max=0.025728 dtf/min=0.021624 dtf/std=0.000843 dto/mean=0.718427 dto/max=0.746834 dto/min=0.683279 dto/std=0.019673 dtb/mean=0.020654 dtb/max=0.021240 dtb/min=0.020345 dtb/std=0.000230
[2025-12-31 12:15:43,923470][I][examples/vit:445:train_fn] iter=25 loss=6.988525 dt=0.730475 dtd=0.005105 dtf=0.022559 dto=0.682103 dtb=0.020708 loss/mean=6.990112 loss/max=7.058594 loss/min=6.914795 loss/std=0.035373 dt/mean=0.745873 dt/max=0.765539 dt/min=0.722147 dt/std=0.014330 dtd/mean=0.002260 dtd/max=0.005105 dtd/min=0.001588 dtd/std=0.000836 dtf/mean=0.023081 dtf/max=0.025434 dtf/min=0.021319 dtf/std=0.001245 dto/mean=0.699837 dto/max=0.717573 dto/min=0.677176 dto/std=0.013419 dtb/mean=0.020695 dtb/max=0.021258 dtb/min=0.020226 dtb/std=0.000287
[2025-12-31 12:15:44,671377][I][examples/vit:445:train_fn] iter=26 loss=6.978271 dt=0.728732 dtd=0.001722 dtf=0.027302 dto=0.678941 dtb=0.020767 loss/mean=7.000173 loss/max=7.060303 loss/min=6.930664 loss/std=0.038023 dt/mean=0.751072 dt/max=0.780739 dt/min=0.718407 dt/std=0.018831 dtd/mean=0.001792 dtd/max=0.002193 dtd/min=0.001632 dtd/std=0.000166 dtf/mean=0.022540 dtf/max=0.027302 dtf/min=0.021307 dtf/std=0.001100 dto/mean=0.706061 dto/max=0.735676 dto/min=0.673876 dto/std=0.019266 dtb/mean=0.020680 dtb/max=0.021233 dtb/min=0.020343 dtb/std=0.000255
[2025-12-31 12:15:45,426093][I][examples/vit:445:train_fn] iter=27 loss=6.985840 dt=0.735329 dtd=0.001691 dtf=0.022562 dto=0.690344 dtb=0.020733 loss/mean=6.988373 loss/max=7.067383 loss/min=6.901123 loss/std=0.034054 dt/mean=0.742626 dt/max=0.748739 dt/min=0.735329 dt/std=0.003427 dtd/mean=0.003301 dtd/max=0.005561 dtd/min=0.001586 dtd/std=0.001388 dtf/mean=0.022639 dtf/max=0.024640 dtf/min=0.021827 dtf/std=0.000775 dto/mean=0.695893 dto/max=0.701744 dto/min=0.690344 dto/std=0.003605 dtb/mean=0.020792 dtb/max=0.021600 dtb/min=0.020282 dtb/std=0.000347
[2025-12-31 12:15:46,187495][I][examples/vit:445:train_fn] iter=28 loss=6.958496 dt=0.743998 dtd=0.001662 dtf=0.022803 dto=0.698705 dtb=0.020828 loss/mean=7.000173 loss/max=7.078369 loss/min=6.931885 loss/std=0.038867 dt/mean=0.746971 dt/max=0.760301 dt/min=0.736719 dt/std=0.009229 dtd/mean=0.002349 dtd/max=0.003210 dtd/min=0.001617 dtd/std=0.000579 dtf/mean=0.022645 dtf/max=0.024866 dtf/min=0.021274 dtf/std=0.001089 dto/mean=0.701287 dto/max=0.713176 dto/min=0.691328 dto/std=0.008573 dtb/mean=0.020691 dtb/max=0.021143 dtb/min=0.020316 dtb/std=0.000245
[2025-12-31 12:15:46,977725][I][examples/vit:445:train_fn] iter=29 loss=6.984131 dt=0.770773 dtd=0.001760 dtf=0.024091 dto=0.724120 dtb=0.020802 loss/mean=6.992188 loss/max=7.054688 loss/min=6.922852 loss/std=0.032682 dt/mean=0.756948 dt/max=0.776455 dt/min=0.732494 dt/std=0.015026 dtd/mean=0.001839 dtd/max=0.002294 dtd/min=0.001539 dtd/std=0.000199 dtf/mean=0.022533 dtf/max=0.024091 dtf/min=0.021525 dtf/std=0.000700 dto/mean=0.711877 dto/max=0.731058 dto/min=0.687968 dto/std=0.014665 dtb/mean=0.020698 dtb/max=0.021198 dtb/min=0.020312 dtb/std=0.000234
[2025-12-31 12:15:47,731430][I][examples/vit:445:train_fn] iter=30 loss=6.968750 dt=0.728727 dtd=0.001725 dtf=0.022213 dto=0.683936 dtb=0.020853 loss/mean=6.986908 loss/max=7.042236 loss/min=6.934814 loss/std=0.030321 dt/mean=0.738177 dt/max=0.744469 dt/min=0.728727 dt/std=0.003020 dtd/mean=0.003073 dtd/max=0.004729 dtd/min=0.001595 dtd/std=0.000956 dtf/mean=0.022537 dtf/max=0.024266 dtf/min=0.021735 dtf/std=0.000633 dto/mean=0.691848 dto/max=0.697793 dto/min=0.683936 dto/std=0.002686 dtb/mean=0.020719 dtb/max=0.021254 dtb/min=0.020223 dtb/std=0.000282
[2025-12-31 12:15:48,481466][I][examples/vit:445:train_fn] iter=31 loss=7.029053 dt=0.736371 dtd=0.001709 dtf=0.022263 dto=0.691483 dtb=0.020916 loss/mean=6.982574 loss/max=7.067383 loss/min=6.832764 loss/std=0.049333 dt/mean=0.745070 dt/max=0.751480 dt/min=0.736371 dt/std=0.003513 dtd/mean=0.003256 dtd/max=0.005504 dtd/min=0.001579 dtd/std=0.001344 dtf/mean=0.022521 dtf/max=0.023703 dtf/min=0.021750 dtf/std=0.000556 dto/mean=0.698558 dto/max=0.705201 dto/min=0.691483 dto/std=0.003774 dtb/mean=0.020735 dtb/max=0.021304 dtb/min=0.020201 dtb/std=0.000310
[2025-12-31 12:15:49,251607][I][examples/vit:445:train_fn] iter=32 loss=7.045166 dt=0.749095 dtd=0.001659 dtf=0.023011 dto=0.703600 dtb=0.020825 loss/mean=7.009898 loss/max=7.066162 loss/min=6.954834 loss/std=0.026349 dt/mean=0.745355 dt/max=0.756215 dt/min=0.736894 dt/std=0.007296 dtd/mean=0.002465 dtd/max=0.004041 dtd/min=0.001549 dtd/std=0.000795 dtf/mean=0.022334 dtf/max=0.023693 dtf/min=0.021286 dtf/std=0.000763 dto/mean=0.699811 dto/max=0.710788 dto/min=0.691312 dto/std=0.007118 dtb/mean=0.020746 dtb/max=0.021371 dtb/min=0.020353 dtb/std=0.000265
[2025-12-31 12:15:49,997199][I][examples/vit:445:train_fn] iter=33 loss=7.020264 dt=0.727653 dtd=0.001750 dtf=0.022298 dto=0.682872 dtb=0.020734 loss/mean=6.983205 loss/max=7.062500 loss/min=6.895508 loss/std=0.035641 dt/mean=0.738254 dt/max=0.745029 dt/min=0.727653 dt/std=0.003339 dtd/mean=0.003172 dtd/max=0.004648 dtd/min=0.001657 dtd/std=0.000887 dtf/mean=0.022837 dtf/max=0.024299 dtf/min=0.022065 dtf/std=0.000576 dto/mean=0.691531 dto/max=0.697682 dto/min=0.682872 dto/std=0.002794 dtb/mean=0.020714 dtb/max=0.021391 dtb/min=0.020169 dtb/std=0.000318
[2025-12-31 12:15:50,759356][I][examples/vit:445:train_fn] iter=34 loss=6.914062 dt=0.741301 dtd=0.001698 dtf=0.024205 dto=0.694543 dtb=0.020855 loss/mean=6.980601 loss/max=7.046875 loss/min=6.911621 loss/std=0.036172 dt/mean=0.742607 dt/max=0.749077 dt/min=0.736232 dt/std=0.003409 dtd/mean=0.003402 dtd/max=0.005554 dtd/min=0.001676 dtd/std=0.001323 dtf/mean=0.022480 dtf/max=0.024390 dtf/min=0.021743 dtf/std=0.000859 dto/mean=0.695940 dto/max=0.702199 dto/min=0.691508 dto/std=0.003538 dtb/mean=0.020785 dtb/max=0.021545 dtb/min=0.020341 dtb/std=0.000307
[2025-12-31 12:15:51,512264][I][examples/vit:445:train_fn] iter=35 loss=6.983398 dt=0.734306 dtd=0.001671 dtf=0.022545 dto=0.689456 dtb=0.020635 loss/mean=6.997111 loss/max=7.062988 loss/min=6.936035 loss/std=0.028505 dt/mean=0.742718 dt/max=0.749095 dt/min=0.734306 dt/std=0.003453 dtd/mean=0.003206 dtd/max=0.005422 dtd/min=0.001573 dtd/std=0.001368 dtf/mean=0.022523 dtf/max=0.024025 dtf/min=0.021785 dtf/std=0.000639 dto/mean=0.696285 dto/max=0.702540 dto/min=0.689456 dto/std=0.003634 dtb/mean=0.020704 dtb/max=0.021270 dtb/min=0.020327 dtb/std=0.000209
[2025-12-31 12:15:52,257198][I][examples/vit:445:train_fn] iter=36 loss=6.991455 dt=0.735148 dtd=0.001708 dtf=0.022518 dto=0.690147 dtb=0.020774 loss/mean=6.995128 loss/max=7.067383 loss/min=6.942139 loss/std=0.032034 dt/mean=0.741631 dt/max=0.747561 dt/min=0.735148 dt/std=0.003155 dtd/mean=0.003153 dtd/max=0.005476 dtd/min=0.001570 dtd/std=0.001396 dtf/mean=0.022958 dtf/max=0.024346 dtf/min=0.021980 dtf/std=0.000610 dto/mean=0.694814 dto/max=0.700863 dto/min=0.690147 dto/std=0.003289 dtb/mean=0.020706 dtb/max=0.021249 dtb/min=0.020209 dtb/std=0.000304
[2025-12-31 12:15:53,082889][I][examples/vit:445:train_fn] iter=37 loss=7.010742 dt=0.754607 dtd=0.001597 dtf=0.024032 dto=0.708152 dtb=0.020825 loss/mean=6.982076 loss/max=7.057861 loss/min=6.918457 loss/std=0.034110 dt/mean=0.746617 dt/max=0.757576 dt/min=0.737717 dt/std=0.007612 dtd/mean=0.002758 dtd/max=0.005157 dtd/min=0.001536 dtd/std=0.001230 dtf/mean=0.022482 dtf/max=0.024220 dtf/min=0.021333 dtf/std=0.000915 dto/mean=0.700606 dto/max=0.711254 dto/min=0.690800 dto/std=0.007668 dtb/mean=0.020771 dtb/max=0.021395 dtb/min=0.020302 dtb/std=0.000313
[2025-12-31 12:15:53,822050][I][examples/vit:445:train_fn] iter=38 loss=6.975830 dt=0.725656 dtd=0.001658 dtf=0.025023 dto=0.678155 dtb=0.020820 loss/mean=6.994375 loss/max=7.053711 loss/min=6.947266 loss/std=0.024550 dt/mean=0.753547 dt/max=0.784940 dt/min=0.718371 dt/std=0.020758 dtd/mean=0.001801 dtd/max=0.002128 dtd/min=0.001635 dtd/std=0.000166 dtf/mean=0.022542 dtf/max=0.025023 dtf/min=0.021558 dtf/std=0.000725 dto/mean=0.708533 dto/max=0.739468 dto/min=0.674357 dto/std=0.020838 dtb/mean=0.020671 dtb/max=0.021199 dtb/min=0.020265 dtb/std=0.000210
[2025-12-31 12:15:54,657351][I][examples/vit:445:train_fn] iter=39 loss=6.992188 dt=0.764405 dtd=0.001675 dtf=0.025172 dto=0.716769 dtb=0.020789 loss/mean=6.970714 loss/max=7.027100 loss/min=6.914795 loss/std=0.034333 dt/mean=0.754408 dt/max=0.770382 dt/min=0.729280 dt/std=0.013428 dtd/mean=0.002184 dtd/max=0.003573 dtd/min=0.001631 dtd/std=0.000654 dtf/mean=0.022970 dtf/max=0.025172 dtf/min=0.021572 dtf/std=0.001008 dto/mean=0.708547 dto/max=0.723172 dto/min=0.684613 dto/std=0.012350 dtb/mean=0.020707 dtb/max=0.021262 dtb/min=0.020282 dtb/std=0.000285
[2025-12-31 12:15:55,440080][I][examples/vit:445:train_fn] iter=40 loss=7.038330 dt=0.733582 dtd=0.001704 dtf=0.024849 dto=0.686207 dtb=0.020821 loss/mean=6.987508 loss/max=7.041504 loss/min=6.928223 loss/std=0.026493 dt/mean=0.760830 dt/max=0.792707 dt/min=0.725636 dt/std=0.020782 dtd/mean=0.001795 dtd/max=0.002238 dtd/min=0.001635 dtd/std=0.000158 dtf/mean=0.022468 dtf/max=0.024849 dtf/min=0.021375 dtf/std=0.000751 dto/mean=0.715907 dto/max=0.747222 dto/min=0.682017 dto/std=0.020906 dtb/mean=0.020659 dtb/max=0.021324 dtb/min=0.020248 dtb/std=0.000278
[2025-12-31 12:15:56,186584][I][examples/vit:445:train_fn] iter=41 loss=6.945068 dt=0.727525 dtd=0.001735 dtf=0.022900 dto=0.682011 dtb=0.020879 loss/mean=6.969208 loss/max=7.058594 loss/min=6.916992 loss/std=0.032973 dt/mean=0.748452 dt/max=0.770154 dt/min=0.727149 dt/std=0.015412 dtd/mean=0.002192 dtd/max=0.003208 dtd/min=0.001573 dtd/std=0.000546 dtf/mean=0.022726 dtf/max=0.024180 dtf/min=0.021340 dtf/std=0.000864 dto/mean=0.702787 dto/max=0.723392 dto/min=0.681505 dto/std=0.014915 dtb/mean=0.020747 dtb/max=0.021484 dtb/min=0.020331 dtb/std=0.000298
[2025-12-31 12:15:57,002317][I][examples/vit:445:train_fn] iter=42 loss=7.038574 dt=0.776704 dtd=0.001668 dtf=0.022678 dto=0.731496 dtb=0.020862 loss/mean=6.976430 loss/max=7.039307 loss/min=6.919434 loss/std=0.034110 dt/mean=0.756732 dt/max=0.786670 dt/min=0.723782 dt/std=0.018553 dtd/mean=0.001802 dtd/max=0.002211 dtd/min=0.001626 dtd/std=0.000166 dtf/mean=0.022396 dtf/max=0.023093 dtf/min=0.021225 dtf/std=0.000533 dto/mean=0.711839 dto/max=0.741265 dto/min=0.678716 dto/std=0.018654 dtb/mean=0.020695 dtb/max=0.021359 dtb/min=0.020267 dtb/std=0.000279
[2025-12-31 12:15:57,815591][I][examples/vit:445:train_fn] iter=43 loss=7.035645 dt=0.758882 dtd=0.012152 dtf=0.022963 dto=0.702949 dtb=0.020818 loss/mean=6.980540 loss/max=7.037842 loss/min=6.929443 loss/std=0.032973 dt/mean=0.765730 dt/max=0.788675 dt/min=0.732798 dt/std=0.018196 dtd/mean=0.002373 dtd/max=0.012152 dtd/min=0.001639 dtd/std=0.002068 dtf/mean=0.022709 dtf/max=0.024687 dtf/min=0.021031 dtf/std=0.000978 dto/mean=0.719959 dto/max=0.741003 dto/min=0.687917 dto/std=0.017812 dtb/mean=0.020690 dtb/max=0.021201 dtb/min=0.020271 dtb/std=0.000228
[2025-12-31 12:15:58,555728][I][examples/vit:445:train_fn] iter=44 loss=6.953613 dt=0.719717 dtd=0.001672 dtf=0.022583 dto=0.674526 dtb=0.020936 loss/mean=6.987224 loss/max=7.059814 loss/min=6.906738 loss/std=0.037973 dt/mean=0.744013 dt/max=0.769423 dt/min=0.719717 dt/std=0.016983 dtd/mean=0.001891 dtd/max=0.002867 dtd/min=0.001532 dtd/std=0.000326 dtf/mean=0.022596 dtf/max=0.025096 dtf/min=0.021569 dtf/std=0.000788 dto/mean=0.698715 dto/max=0.723603 dto/min=0.674526 dto/std=0.016540 dtb/mean=0.020811 dtb/max=0.021485 dtb/min=0.020209 dtb/std=0.000297
[2025-12-31 12:15:59,320244][I][examples/vit:445:train_fn] iter=45 loss=6.965576 dt=0.736210 dtd=0.001692 dtf=0.024698 dto=0.689147 dtb=0.020674 loss/mean=6.996837 loss/max=7.056885 loss/min=6.898682 loss/std=0.037109 dt/mean=0.744331 dt/max=0.750168 dt/min=0.736210 dt/std=0.003461 dtd/mean=0.003146 dtd/max=0.005443 dtd/min=0.001592 dtd/std=0.001397 dtf/mean=0.022662 dtf/max=0.024698 dtf/min=0.021666 dtf/std=0.000771 dto/mean=0.697826 dto/max=0.704015 dto/min=0.689147 dto/std=0.003852 dtb/mean=0.020697 dtb/max=0.021296 dtb/min=0.020216 dtb/std=0.000292
[2025-12-31 12:16:00,081771][I][examples/vit:445:train_fn] iter=46 loss=6.968750 dt=0.741624 dtd=0.001717 dtf=0.023044 dto=0.696055 dtb=0.020807 loss/mean=6.971436 loss/max=7.033203 loss/min=6.863525 loss/std=0.034499 dt/mean=0.747135 dt/max=0.763320 dt/min=0.732857 dt/std=0.011744 dtd/mean=0.002163 dtd/max=0.003133 dtd/min=0.001573 dtd/std=0.000460 dtf/mean=0.022883 dtf/max=0.025204 dtf/min=0.021264 dtf/std=0.001195 dto/mean=0.701406 dto/max=0.716596 dto/min=0.688054 dto/std=0.011067 dtb/mean=0.020684 dtb/max=0.021180 dtb/min=0.020288 dtb/std=0.000252
[2025-12-31 12:16:00,841569][I][examples/vit:445:train_fn] iter=47 loss=6.917969 dt=0.734437 dtd=0.001652 dtf=0.023252 dto=0.688742 dtb=0.020790 loss/mean=6.979259 loss/max=7.023926 loss/min=6.917969 loss/std=0.027274 dt/mean=0.740831 dt/max=0.746801 dt/min=0.734437 dt/std=0.002858 dtd/mean=0.003643 dtd/max=0.005405 dtd/min=0.001531 dtd/std=0.001477 dtf/mean=0.022557 dtf/max=0.024191 dtf/min=0.021717 dtf/std=0.000603 dto/mean=0.693941 dto/max=0.701646 dto/min=0.688742 dto/std=0.002652 dtb/mean=0.020690 dtb/max=0.021128 dtb/min=0.020172 dtb/std=0.000248
[2025-12-31 12:16:01,595278][I][examples/vit:445:train_fn] iter=48 loss=6.997559 dt=0.732685 dtd=0.001657 dtf=0.022673 dto=0.687710 dtb=0.020645 loss/mean=6.980276 loss/max=7.027344 loss/min=6.907715 loss/std=0.035048 dt/mean=0.745125 dt/max=0.751038 dt/min=0.732685 dt/std=0.004158 dtd/mean=0.002830 dtd/max=0.004684 dtd/min=0.001657 dtd/std=0.001000 dtf/mean=0.022955 dtf/max=0.024705 dtf/min=0.022187 dtf/std=0.000669 dto/mean=0.698627 dto/max=0.705444 dto/min=0.687710 dto/std=0.004085 dtb/mean=0.020713 dtb/max=0.021169 dtb/min=0.020320 dtb/std=0.000278
[2025-12-31 12:16:02,343776][I][examples/vit:445:train_fn] iter=49 loss=6.997070 dt=0.729157 dtd=0.001664 dtf=0.023304 dto=0.683426 dtb=0.020763 loss/mean=6.977102 loss/max=7.046875 loss/min=6.920898 loss/std=0.028033 dt/mean=0.738032 dt/max=0.743651 dt/min=0.729157 dt/std=0.003392 dtd/mean=0.003257 dtd/max=0.005523 dtd/min=0.001553 dtd/std=0.001387 dtf/mean=0.022512 dtf/max=0.023798 dtf/min=0.021764 dtf/std=0.000579 dto/mean=0.691580 dto/max=0.697807 dto/min=0.683426 dto/std=0.003778 dtb/mean=0.020684 dtb/max=0.021177 dtb/min=0.020273 dtb/std=0.000231
[2025-12-31 12:16:03,087624][I][examples/vit:445:train_fn] iter=50 loss=6.925293 dt=0.730377 dtd=0.001676 dtf=0.022686 dto=0.684904 dtb=0.021110 loss/mean=6.973846 loss/max=7.045166 loss/min=6.856689 loss/std=0.040075 dt/mean=0.737219 dt/max=0.743345 dt/min=0.730377 dt/std=0.003285 dtd/mean=0.003267 dtd/max=0.005473 dtd/min=0.001551 dtd/std=0.001356 dtf/mean=0.022508 dtf/max=0.023485 dtf/min=0.021924 dtf/std=0.000446 dto/mean=0.690625 dto/max=0.697362 dto/min=0.684904 dto/std=0.003584 dtb/mean=0.020819 dtb/max=0.021817 dtb/min=0.020345 dtb/std=0.000324
[2025-12-31 12:16:03,908992][I][examples/vit:445:train_fn] iter=51 loss=7.015381 dt=0.749356 dtd=0.001659 dtf=0.024072 dto=0.702847 dtb=0.020778 loss/mean=6.979340 loss/max=7.034912 loss/min=6.907959 loss/std=0.035908 dt/mean=0.744960 dt/max=0.756240 dt/min=0.735700 dt/std=0.007744 dtd/mean=0.002572 dtd/max=0.004506 dtd/min=0.001551 dtd/std=0.000984 dtf/mean=0.022622 dtf/max=0.024409 dtf/min=0.021383 dtf/std=0.001038 dto/mean=0.699012 dto/max=0.709381 dto/min=0.689005 dto/std=0.007465 dtb/mean=0.020755 dtb/max=0.021618 dtb/min=0.020256 dtb/std=0.000345
[2025-12-31 12:16:04,680315][I][examples/vit:445:train_fn] iter=52 loss=6.945557 dt=0.718842 dtd=0.001774 dtf=0.022528 dto=0.673614 dtb=0.020926 loss/mean=6.967998 loss/max=7.070068 loss/min=6.917236 loss/std=0.039451 dt/mean=0.752761 dt/max=0.781505 dt/min=0.718842 dt/std=0.019747 dtd/mean=0.001806 dtd/max=0.002223 dtd/min=0.001580 dtd/std=0.000170 dtf/mean=0.022435 dtf/max=0.023659 dtf/min=0.021713 dtf/std=0.000526 dto/mean=0.707762 dto/max=0.735720 dto/min=0.673614 dto/std=0.019645 dtb/mean=0.020759 dtb/max=0.021476 dtb/min=0.020367 dtb/std=0.000264
[2025-12-31 12:16:05,486745][I][examples/vit:445:train_fn] iter=53 loss=7.022705 dt=0.726803 dtd=0.001651 dtf=0.022481 dto=0.681990 dtb=0.020681 loss/mean=6.981262 loss/max=7.050781 loss/min=6.899170 loss/std=0.040407 dt/mean=0.752933 dt/max=0.773121 dt/min=0.726373 dt/std=0.016582 dtd/mean=0.001963 dtd/max=0.003030 dtd/min=0.001545 dtd/std=0.000437 dtf/mean=0.022516 dtf/max=0.024074 dtf/min=0.021568 dtf/std=0.000693 dto/mean=0.707640 dto/max=0.726559 dto/min=0.680444 dto/std=0.016295 dtb/mean=0.020813 dtb/max=0.021907 dtb/min=0.020303 dtb/std=0.000374
[2025-12-31 12:16:06,253200][I][examples/vit:445:train_fn] iter=54 loss=6.986328 dt=0.714074 dtd=0.001667 dtf=0.022484 dto=0.669063 dtb=0.020860 loss/mean=6.987539 loss/max=7.055664 loss/min=6.936035 loss/std=0.028705 dt/mean=0.751958 dt/max=0.784207 dt/min=0.714074 dt/std=0.020769 dtd/mean=0.001780 dtd/max=0.002099 dtd/min=0.001626 dtd/std=0.000151 dtf/mean=0.022335 dtf/max=0.023637 dtf/min=0.021591 dtf/std=0.000496 dto/mean=0.707091 dto/max=0.738269 dto/min=0.669063 dto/std=0.020846 dtb/mean=0.020752 dtb/max=0.021352 dtb/min=0.020242 dtb/std=0.000278
[2025-12-31 12:16:07,036859][I][examples/vit:445:train_fn] iter=55 loss=7.020752 dt=0.726670 dtd=0.001671 dtf=0.023300 dto=0.680836 dtb=0.020863 loss/mean=6.975759 loss/max=7.020752 loss/min=6.936035 loss/std=0.021573 dt/mean=0.748641 dt/max=0.772063 dt/min=0.725118 dt/std=0.017210 dtd/mean=0.002003 dtd/max=0.003673 dtd/min=0.001546 dtd/std=0.000579 dtf/mean=0.022822 dtf/max=0.024702 dtf/min=0.021458 dtf/std=0.000904 dto/mean=0.703149 dto/max=0.725319 dto/min=0.678733 dto/std=0.017051 dtb/mean=0.020667 dtb/max=0.021340 dtb/min=0.020243 dtb/std=0.000276
[2025-12-31 12:16:07,827772][I][examples/vit:445:train_fn] iter=56 loss=6.957275 dt=0.730915 dtd=0.001995 dtf=0.022926 dto=0.685141 dtb=0.020853 loss/mean=6.961477 loss/max=7.016602 loss/min=6.924561 loss/std=0.020670 dt/mean=0.757991 dt/max=0.778522 dt/min=0.730217 dt/std=0.017030 dtd/mean=0.002200 dtd/max=0.003252 dtd/min=0.001647 dtd/std=0.000499 dtf/mean=0.022664 dtf/max=0.023615 dtf/min=0.021357 dtf/std=0.000580 dto/mean=0.712382 dto/max=0.732616 dto/min=0.684696 dto/std=0.016824 dtb/mean=0.020746 dtb/max=0.021194 dtb/min=0.020332 dtb/std=0.000239
[2025-12-31 12:16:08,579242][I][examples/vit:445:train_fn] iter=57 loss=7.012207 dt=0.737117 dtd=0.001701 dtf=0.023760 dto=0.690935 dtb=0.020722 loss/mean=6.988149 loss/max=7.083252 loss/min=6.920166 loss/std=0.034222 dt/mean=0.753169 dt/max=0.786585 dt/min=0.716214 dt/std=0.021419 dtd/mean=0.001782 dtd/max=0.002007 dtd/min=0.001624 dtd/std=0.000127 dtf/mean=0.022379 dtf/max=0.023760 dtf/min=0.021507 dtf/std=0.000557 dto/mean=0.708152 dto/max=0.740705 dto/min=0.672085 dto/std=0.021671 dtb/mean=0.020857 dtb/max=0.021699 dtb/min=0.020344 dtb/std=0.000374
[2025-12-31 12:16:09,362010][I][examples/vit:445:train_fn] iter=58 loss=6.983643 dt=0.763087 dtd=0.001641 dtf=0.024827 dto=0.715777 dtb=0.020842 loss/mean=6.971110 loss/max=7.022461 loss/min=6.918945 loss/std=0.027204 dt/mean=0.748961 dt/max=0.769213 dt/min=0.724905 dt/std=0.015968 dtd/mean=0.001924 dtd/max=0.002593 dtd/min=0.001557 dtd/std=0.000297 dtf/mean=0.022521 dtf/max=0.024827 dtf/min=0.021563 dtf/std=0.000732 dto/mean=0.703753 dto/max=0.723593 dto/min=0.680019 dto/std=0.015505 dtb/mean=0.020763 dtb/max=0.021321 dtb/min=0.020258 dtb/std=0.000261
[2025-12-31 12:16:10,113690][I][examples/vit:445:train_fn] iter=59 loss=6.993652 dt=0.732712 dtd=0.001718 dtf=0.022345 dto=0.687850 dtb=0.020799 loss/mean=6.973938 loss/max=7.031738 loss/min=6.913330 loss/std=0.033942 dt/mean=0.740580 dt/max=0.746455 dt/min=0.732712 dt/std=0.003285 dtd/mean=0.003164 dtd/max=0.005442 dtd/min=0.001579 dtd/std=0.001364 dtf/mean=0.022843 dtf/max=0.024134 dtf/min=0.021971 dtf/std=0.000731 dto/mean=0.693711 dto/max=0.699074 dto/min=0.687850 dto/std=0.003418 dtb/mean=0.020862 dtb/max=0.021546 dtb/min=0.020358 dtb/std=0.000337
[2025-12-31 12:16:10,896999][I][examples/vit:445:train_fn] iter=60 loss=6.969971 dt=0.747576 dtd=0.001661 dtf=0.023705 dto=0.701328 dtb=0.020880 loss/mean=6.962677 loss/max=7.035400 loss/min=6.872803 loss/std=0.038965 dt/mean=0.746533 dt/max=0.760261 dt/min=0.733879 dt/std=0.009983 dtd/mean=0.002699 dtd/max=0.004842 dtd/min=0.001575 dtd/std=0.001212 dtf/mean=0.022815 dtf/max=0.025205 dtf/min=0.021562 dtf/std=0.001125 dto/mean=0.700239 dto/max=0.713503 dto/min=0.687099 dto/std=0.009887 dtb/mean=0.020781 dtb/max=0.021246 dtb/min=0.020256 dtb/std=0.000274
[2025-12-31 12:16:11,665090][I][examples/vit:445:train_fn] iter=61 loss=6.933838 dt=0.748712 dtd=0.001657 dtf=0.023562 dto=0.702590 dtb=0.020903 loss/mean=6.976868 loss/max=7.038330 loss/min=6.912598 loss/std=0.033942 dt/mean=0.748078 dt/max=0.776145 dt/min=0.718618 dt/std=0.019553 dtd/mean=0.001892 dtd/max=0.002529 dtd/min=0.001623 dtd/std=0.000261 dtf/mean=0.022607 dtf/max=0.024323 dtf/min=0.021660 dtf/std=0.000795 dto/mean=0.702790 dto/max=0.728456 dto/min=0.674124 dto/std=0.019070 dtb/mean=0.020790 dtb/max=0.021661 dtb/min=0.020219 dtb/std=0.000346
[2025-12-31 12:16:12,412241][I][examples/vit:445:train_fn] iter=62 loss=6.958252 dt=0.732601 dtd=0.001710 dtf=0.022547 dto=0.687455 dtb=0.020889 loss/mean=6.965841 loss/max=7.051514 loss/min=6.913818 loss/std=0.035426 dt/mean=0.739734 dt/max=0.745430 dt/min=0.732601 dt/std=0.003257 dtd/mean=0.003280 dtd/max=0.005450 dtd/min=0.001559 dtd/std=0.001292 dtf/mean=0.022775 dtf/max=0.024696 dtf/min=0.021823 dtf/std=0.000722 dto/mean=0.692936 dto/max=0.698621 dto/min=0.687455 dto/std=0.003339 dtb/mean=0.020744 dtb/max=0.021340 dtb/min=0.020305 dtb/std=0.000279
[2025-12-31 12:16:13,174938][I][examples/vit:445:train_fn] iter=63 loss=7.009521 dt=0.740339 dtd=0.001691 dtf=0.023941 dto=0.693831 dtb=0.020876 loss/mean=6.973592 loss/max=7.028320 loss/min=6.924316 loss/std=0.029232 dt/mean=0.739645 dt/max=0.748818 dt/min=0.733537 dt/std=0.005658 dtd/mean=0.003032 dtd/max=0.005469 dtd/min=0.001573 dtd/std=0.001450 dtf/mean=0.022828 dtf/max=0.024976 dtf/min=0.021450 dtf/std=0.001211 dto/mean=0.692919 dto/max=0.702017 dto/min=0.685927 dto/std=0.005692 dtb/mean=0.020867 dtb/max=0.022359 dtb/min=0.020390 dtb/std=0.000440
[2025-12-31 12:16:13,919647][I][examples/vit:445:train_fn] iter=64 loss=6.896729 dt=0.731400 dtd=0.001672 dtf=0.022233 dto=0.686662 dtb=0.020833 loss/mean=6.962646 loss/max=7.011475 loss/min=6.896729 loss/std=0.027964 dt/mean=0.737714 dt/max=0.741256 dt/min=0.731400 dt/std=0.002549 dtd/mean=0.003417 dtd/max=0.005453 dtd/min=0.001575 dtd/std=0.001243 dtf/mean=0.022408 dtf/max=0.023186 dtf/min=0.021953 dtf/std=0.000294 dto/mean=0.691057 dto/max=0.695397 dto/min=0.686662 dto/std=0.002354 dtb/mean=0.020831 dtb/max=0.021627 dtb/min=0.020358 dtb/std=0.000299
[2025-12-31 12:16:14,724718][I][examples/vit:445:train_fn] iter=65 loss=6.983154 dt=0.740475 dtd=0.001657 dtf=0.026268 dto=0.691680 dtb=0.020869 loss/mean=6.979614 loss/max=7.029297 loss/min=6.906494 loss/std=0.025466 dt/mean=0.737195 dt/max=0.748038 dt/min=0.728805 dt/std=0.006987 dtd/mean=0.002650 dtd/max=0.005013 dtd/min=0.001557 dtd/std=0.001223 dtf/mean=0.022836 dtf/max=0.026268 dtf/min=0.021512 dtf/std=0.001202 dto/mean=0.690811 dto/max=0.701612 dto/min=0.682157 dto/std=0.007034 dtb/mean=0.020898 dtb/max=0.022102 dtb/min=0.020411 dtb/std=0.000431
[2025-12-31 12:16:15,460588][I][examples/vit:445:train_fn] iter=66 loss=6.993408 dt=0.717219 dtd=0.001660 dtf=0.022625 dto=0.672191 dtb=0.020743 loss/mean=6.968008 loss/max=7.035645 loss/min=6.893066 loss/std=0.032624 dt/mean=0.751308 dt/max=0.772520 dt/min=0.717219 dt/std=0.019154 dtd/mean=0.001972 dtd/max=0.002954 dtd/min=0.001552 dtd/std=0.000424 dtf/mean=0.022521 dtf/max=0.023821 dtf/min=0.021679 dtf/std=0.000597 dto/mean=0.705970 dto/max=0.727169 dto/min=0.672191 dto/std=0.018948 dtb/mean=0.020846 dtb/max=0.021620 dtb/min=0.020417 dtb/std=0.000325
[2025-12-31 12:16:16,284567][I][examples/vit:445:train_fn] iter=67 loss=6.943604 dt=0.745598 dtd=0.002759 dtf=0.023811 dto=0.698298 dtb=0.020731 loss/mean=6.967296 loss/max=7.018311 loss/min=6.920166 loss/std=0.025391 dt/mean=0.745446 dt/max=0.759088 dt/min=0.734187 dt/std=0.009741 dtd/mean=0.002868 dtd/max=0.005055 dtd/min=0.001623 dtd/std=0.001256 dtf/mean=0.022759 dtf/max=0.024636 dtf/min=0.021525 dtf/std=0.001030 dto/mean=0.699143 dto/max=0.711942 dto/min=0.687225 dto/std=0.009435 dtb/mean=0.020677 dtb/max=0.021094 dtb/min=0.020287 dtb/std=0.000258
[2025-12-31 12:16:17,011642][I][examples/vit:445:train_fn] iter=68 loss=6.976074 dt=0.713172 dtd=0.001813 dtf=0.023177 dto=0.667256 dtb=0.020926 loss/mean=6.972229 loss/max=7.019043 loss/min=6.924316 loss/std=0.023519 dt/mean=0.747308 dt/max=0.778938 dt/min=0.712412 dt/std=0.020759 dtd/mean=0.001799 dtd/max=0.002089 dtd/min=0.001641 dtd/std=0.000141 dtf/mean=0.022417 dtf/max=0.023780 dtf/min=0.021402 dtf/std=0.000554 dto/mean=0.702239 dto/max=0.732663 dto/min=0.667256 dto/std=0.020857 dtb/mean=0.020854 dtb/max=0.021991 dtb/min=0.020358 dtb/std=0.000379
[2025-12-31 12:16:17,757204][I][examples/vit:445:train_fn] iter=69 loss=7.019287 dt=0.736641 dtd=0.001659 dtf=0.023917 dto=0.690050 dtb=0.021014 loss/mean=6.978435 loss/max=7.033447 loss/min=6.893066 loss/std=0.031614 dt/mean=0.736665 dt/max=0.741711 dt/min=0.731619 dt/std=0.003202 dtd/mean=0.003064 dtd/max=0.005396 dtd/min=0.001570 dtd/std=0.001377 dtf/mean=0.022623 dtf/max=0.024116 dtf/min=0.021729 dtf/std=0.000645 dto/mean=0.690252 dto/max=0.696442 dto/min=0.686604 dto/std=0.003588 dtb/mean=0.020725 dtb/max=0.021321 dtb/min=0.020251 dtb/std=0.000325
[2025-12-31 12:16:18,541456][I][examples/vit:445:train_fn] iter=70 loss=6.986084 dt=0.760055 dtd=0.001617 dtf=0.025038 dto=0.712560 dtb=0.020840 loss/mean=6.975403 loss/max=7.017090 loss/min=6.896240 loss/std=0.027344 dt/mean=0.746176 dt/max=0.763127 dt/min=0.731291 dt/std=0.012082 dtd/mean=0.002768 dtd/max=0.005176 dtd/min=0.001572 dtd/std=0.001342 dtf/mean=0.022493 dtf/max=0.025038 dtf/min=0.021381 dtf/std=0.001002 dto/mean=0.700163 dto/max=0.716501 dto/min=0.684296 dto/std=0.012116 dtb/mean=0.020752 dtb/max=0.021390 dtb/min=0.020222 dtb/std=0.000294
[2025-12-31 12:16:19,334196][I][examples/vit:445:train_fn] iter=71 loss=6.953369 dt=0.771233 dtd=0.001641 dtf=0.022572 dto=0.726191 dtb=0.020829 loss/mean=6.974915 loss/max=7.038574 loss/min=6.913574 loss/std=0.029813 dt/mean=0.757112 dt/max=0.789940 dt/min=0.722059 dt/std=0.020186 dtd/mean=0.001796 dtd/max=0.002250 dtd/min=0.001634 dtd/std=0.000169 dtf/mean=0.022474 dtf/max=0.023432 dtf/min=0.021478 dtf/std=0.000466 dto/mean=0.712175 dto/max=0.744055 dto/min=0.677076 dto/std=0.020119 dtb/mean=0.020667 dtb/max=0.021133 dtb/min=0.020371 dtb/std=0.000207
[2025-12-31 12:16:20,077115][I][examples/vit:445:train_fn] iter=72 loss=7.032959 dt=0.734149 dtd=0.001679 dtf=0.022509 dto=0.688994 dtb=0.020967 loss/mean=6.976024 loss/max=7.038330 loss/min=6.899414 loss/std=0.038571 dt/mean=0.740182 dt/max=0.745929 dt/min=0.734149 dt/std=0.003146 dtd/mean=0.003266 dtd/max=0.005491 dtd/min=0.001601 dtd/std=0.001299 dtf/mean=0.022640 dtf/max=0.024450 dtf/min=0.021874 dtf/std=0.000701 dto/mean=0.693552 dto/max=0.699313 dto/min=0.688994 dto/std=0.002868 dtb/mean=0.020724 dtb/max=0.021125 dtb/min=0.020261 dtb/std=0.000253
[2025-12-31 12:16:20,836082][I][examples/vit:445:train_fn] iter=73 loss=7.007568 dt=0.745784 dtd=0.001600 dtf=0.024390 dto=0.699126 dtb=0.020668 loss/mean=6.979442 loss/max=7.045898 loss/min=6.938721 loss/std=0.027204 dt/mean=0.742050 dt/max=0.748410 dt/min=0.735671 dt/std=0.003312 dtd/mean=0.003212 dtd/max=0.005472 dtd/min=0.001600 dtd/std=0.001346 dtf/mean=0.022506 dtf/max=0.024390 dtf/min=0.021779 dtf/std=0.000684 dto/mean=0.695677 dto/max=0.702715 dto/min=0.691317 dto/std=0.003525 dtb/mean=0.020656 dtb/max=0.021041 dtb/min=0.020242 dtb/std=0.000207
[2025-12-31 12:16:21,664389][I][examples/vit:445:train_fn] iter=74 loss=6.933594 dt=0.749712 dtd=0.001732 dtf=0.024171 dto=0.702734 dtb=0.021074 loss/mean=6.961670 loss/max=7.008789 loss/min=6.928467 loss/std=0.021573 dt/mean=0.744975 dt/max=0.755956 dt/min=0.736076 dt/std=0.007805 dtd/mean=0.002906 dtd/max=0.004958 dtd/min=0.001630 dtd/std=0.001174 dtf/mean=0.022639 dtf/max=0.024171 dtf/min=0.021378 dtf/std=0.001003 dto/mean=0.698668 dto/max=0.708864 dto/min=0.689240 dto/std=0.007451 dtb/mean=0.020762 dtb/max=0.021149 dtb/min=0.020382 dtb/std=0.000228
[2025-12-31 12:16:22,453938][I][examples/vit:445:train_fn] iter=75 loss=6.971191 dt=0.717920 dtd=0.001719 dtf=0.022678 dto=0.672664 dtb=0.020860 loss/mean=6.967296 loss/max=7.068115 loss/min=6.936523 loss/std=0.028438 dt/mean=0.752582 dt/max=0.785606 dt/min=0.717920 dt/std=0.021208 dtd/mean=0.001796 dtd/max=0.002113 dtd/min=0.001629 dtd/std=0.000155 dtf/mean=0.022076 dtf/max=0.022856 dtf/min=0.021302 dtf/std=0.000369 dto/mean=0.707902 dto/max=0.740324 dto/min=0.672664 dto/std=0.021404 dtb/mean=0.020809 dtb/max=0.022220 dtb/min=0.020329 dtb/std=0.000479
[2025-12-31 12:16:23,201097][I][examples/vit:445:train_fn] iter=76 loss=6.988281 dt=0.727026 dtd=0.001768 dtf=0.022971 dto=0.681522 dtb=0.020766 loss/mean=6.961833 loss/max=7.000977 loss/min=6.909180 loss/std=0.025466 dt/mean=0.755362 dt/max=0.786134 dt/min=0.719971 dt/std=0.020749 dtd/mean=0.001817 dtd/max=0.002205 dtd/min=0.001637 dtd/std=0.000169 dtf/mean=0.022384 dtf/max=0.024022 dtf/min=0.021393 dtf/std=0.000706 dto/mean=0.710380 dto/max=0.739394 dto/min=0.674898 dto/std=0.020961 dtb/mean=0.020782 dtb/max=0.021310 dtb/min=0.020390 dtb/std=0.000236
[2025-12-31 12:16:23,947330][I][examples/vit:445:train_fn] iter=77 loss=6.943848 dt=0.730181 dtd=0.001688 dtf=0.022086 dto=0.685413 dtb=0.020994 loss/mean=6.956411 loss/max=7.021484 loss/min=6.871338 loss/std=0.025088 dt/mean=0.741112 dt/max=0.746444 dt/min=0.730181 dt/std=0.003829 dtd/mean=0.002945 dtd/max=0.005359 dtd/min=0.001569 dtd/std=0.001373 dtf/mean=0.022700 dtf/max=0.024477 dtf/min=0.021997 dtf/std=0.000720 dto/mean=0.694724 dto/max=0.701460 dto/min=0.685413 dto/std=0.003837 dtb/mean=0.020743 dtb/max=0.021349 dtb/min=0.020327 dtb/std=0.000262
[2025-12-31 12:16:24,773082][I][examples/vit:445:train_fn] iter=78 loss=6.978516 dt=0.761054 dtd=0.001700 dtf=0.024937 dto=0.713578 dtb=0.020839 loss/mean=6.965098 loss/max=7.022461 loss/min=6.865479 loss/std=0.034166 dt/mean=0.748827 dt/max=0.766382 dt/min=0.732463 dt/std=0.012634 dtd/mean=0.002365 dtd/max=0.004309 dtd/min=0.001603 dtd/std=0.000892 dtf/mean=0.022818 dtf/max=0.025502 dtf/min=0.021410 dtf/std=0.001281 dto/mean=0.702965 dto/max=0.718497 dto/min=0.686230 dto/std=0.012092 dtb/mean=0.020679 dtb/max=0.021142 dtb/min=0.020321 dtb/std=0.000249
[2025-12-31 12:16:25,518658][I][examples/vit:445:train_fn] iter=79 loss=6.922363 dt=0.727205 dtd=0.001661 dtf=0.023797 dto=0.680897 dtb=0.020849 loss/mean=6.969279 loss/max=7.024170 loss/min=6.917969 loss/std=0.028303 dt/mean=0.751349 dt/max=0.784285 dt/min=0.716415 dt/std=0.020322 dtd/mean=0.001798 dtd/max=0.002212 dtd/min=0.001578 dtd/std=0.000171 dtf/mean=0.022376 dtf/max=0.024469 dtf/min=0.021327 dtf/std=0.000698 dto/mean=0.706422 dto/max=0.737019 dto/min=0.671566 dto/std=0.020411 dtb/mean=0.020752 dtb/max=0.021284 dtb/min=0.020372 dtb/std=0.000268
[2025-12-31 12:16:26,272580][I][examples/vit:445:train_fn] iter=80 loss=6.928955 dt=0.734315 dtd=0.003959 dtf=0.022724 dto=0.686737 dtb=0.020895 loss/mean=6.962504 loss/max=7.028076 loss/min=6.925293 loss/std=0.027552 dt/mean=0.740758 dt/max=0.746343 dt/min=0.734315 dt/std=0.002784 dtd/mean=0.003410 dtd/max=0.005488 dtd/min=0.001590 dtd/std=0.001289 dtf/mean=0.022813 dtf/max=0.024931 dtf/min=0.021937 dtf/std=0.000724 dto/mean=0.693835 dto/max=0.700043 dto/min=0.686737 dto/std=0.003044 dtb/mean=0.020699 dtb/max=0.021340 dtb/min=0.020316 dtb/std=0.000263
[2025-12-31 12:16:27,084889][I][examples/vit:445:train_fn] iter=81 loss=6.980469 dt=0.746773 dtd=0.001703 dtf=0.023629 dto=0.700639 dtb=0.020802 loss/mean=6.972107 loss/max=7.020996 loss/min=6.912598 loss/std=0.028705 dt/mean=0.748201 dt/max=0.759930 dt/min=0.739375 dt/std=0.007907 dtd/mean=0.002643 dtd/max=0.004569 dtd/min=0.001562 dtd/std=0.000958 dtf/mean=0.022796 dtf/max=0.024607 dtf/min=0.021381 dtf/std=0.001134 dto/mean=0.702011 dto/max=0.712560 dto/min=0.692673 dto/std=0.007507 dtb/mean=0.020751 dtb/max=0.021455 dtb/min=0.020430 dtb/std=0.000278
[2025-12-31 12:16:27,843702][I][examples/vit:445:train_fn] iter=82 loss=6.937988 dt=0.734645 dtd=0.001703 dtf=0.023584 dto=0.688593 dtb=0.020765 loss/mean=6.963796 loss/max=7.047852 loss/min=6.843262 loss/std=0.039979 dt/mean=0.755872 dt/max=0.788405 dt/min=0.721559 dt/std=0.020939 dtd/mean=0.001804 dtd/max=0.002214 dtd/min=0.001632 dtd/std=0.000168 dtf/mean=0.022503 dtf/max=0.023584 dtf/min=0.021390 dtf/std=0.000611 dto/mean=0.710801 dto/max=0.742479 dto/min=0.676907 dto/std=0.021092 dtb/mean=0.020765 dtb/max=0.021568 dtb/min=0.020342 dtb/std=0.000294
[2025-12-31 12:16:28,646170][I][examples/vit:445:train_fn] iter=83 loss=6.970703 dt=0.768635 dtd=0.001662 dtf=0.023451 dto=0.722643 dtb=0.020879 loss/mean=6.966736 loss/max=7.012695 loss/min=6.913574 loss/std=0.026851 dt/mean=0.751228 dt/max=0.781235 dt/min=0.720570 dt/std=0.019909 dtd/mean=0.001883 dtd/max=0.002957 dtd/min=0.001620 dtd/std=0.000329 dtf/mean=0.022392 dtf/max=0.024167 dtf/min=0.021578 dtf/std=0.000560 dto/mean=0.706234 dto/max=0.735096 dto/min=0.675456 dto/std=0.019756 dtb/mean=0.020719 dtb/max=0.021357 dtb/min=0.020337 dtb/std=0.000256
[2025-12-31 12:16:29,459074][I][examples/vit:445:train_fn] iter=84 loss=6.983887 dt=0.747875 dtd=0.001695 dtf=0.024866 dto=0.700442 dtb=0.020872 loss/mean=6.968292 loss/max=7.023682 loss/min=6.903809 loss/std=0.031855 dt/mean=0.752166 dt/max=0.773170 dt/min=0.729233 dt/std=0.015648 dtd/mean=0.002163 dtd/max=0.003639 dtd/min=0.001540 dtd/std=0.000643 dtf/mean=0.022767 dtf/max=0.024866 dtf/min=0.021553 dtf/std=0.000826 dto/mean=0.706544 dto/max=0.726773 dto/min=0.683277 dto/std=0.015370 dtb/mean=0.020692 dtb/max=0.021045 dtb/min=0.020339 dtb/std=0.000232
[2025-12-31 12:16:30,208391][I][examples/vit:445:train_fn] iter=85 loss=6.955322 dt=0.724813 dtd=0.001680 dtf=0.023383 dto=0.678959 dtb=0.020791 loss/mean=6.962280 loss/max=7.042480 loss/min=6.893799 loss/std=0.035695 dt/mean=0.751438 dt/max=0.784257 dt/min=0.715721 dt/std=0.020661 dtd/mean=0.001808 dtd/max=0.002161 dtd/min=0.001590 dtd/std=0.000179 dtf/mean=0.022539 dtf/max=0.023851 dtf/min=0.021704 dtf/std=0.000611 dto/mean=0.706313 dto/max=0.738357 dto/min=0.671076 dto/std=0.020862 dtb/mean=0.020778 dtb/max=0.021399 dtb/min=0.020329 dtb/std=0.000294
[2025-12-31 12:16:30,995940][I][examples/vit:445:train_fn] iter=86 loss=7.006348 dt=0.746630 dtd=0.004681 dtf=0.023454 dto=0.697523 dtb=0.020972 loss/mean=6.955139 loss/max=7.006592 loss/min=6.906006 loss/std=0.033146 dt/mean=0.744196 dt/max=0.762628 dt/min=0.727688 dt/std=0.013330 dtd/mean=0.002605 dtd/max=0.004681 dtd/min=0.001628 dtd/std=0.001020 dtf/mean=0.022811 dtf/max=0.024531 dtf/min=0.021475 dtf/std=0.000946 dto/mean=0.697820 dto/max=0.715555 dto/min=0.680953 dto/std=0.012916 dtb/mean=0.020960 dtb/max=0.022034 dtb/min=0.020438 dtb/std=0.000390
[2025-12-31 12:16:31,768262][I][examples/vit:445:train_fn] iter=87 loss=6.986572 dt=0.753831 dtd=0.001649 dtf=0.023473 dto=0.707854 dtb=0.020854 loss/mean=6.973979 loss/max=7.022217 loss/min=6.937012 loss/std=0.024000 dt/mean=0.752923 dt/max=0.785754 dt/min=0.716823 dt/std=0.020745 dtd/mean=0.001765 dtd/max=0.002053 dtd/min=0.001533 dtd/std=0.000135 dtf/mean=0.022395 dtf/max=0.023473 dtf/min=0.021672 dtf/std=0.000459 dto/mean=0.708042 dto/max=0.740092 dto/min=0.672523 dto/std=0.020517 dtb/mean=0.020721 dtb/max=0.021177 dtb/min=0.020375 dtb/std=0.000236
[2025-12-31 12:16:32,573348][I][examples/vit:445:train_fn] iter=88 loss=6.975830 dt=0.744247 dtd=0.001681 dtf=0.024306 dto=0.697233 dtb=0.021027 loss/mean=6.963277 loss/max=7.026367 loss/min=6.892090 loss/std=0.030571 dt/mean=0.742532 dt/max=0.756320 dt/min=0.730731 dt/std=0.010125 dtd/mean=0.002749 dtd/max=0.004906 dtd/min=0.001625 dtd/std=0.001149 dtf/mean=0.022892 dtf/max=0.025352 dtf/min=0.021483 dtf/std=0.001251 dto/mean=0.696179 dto/max=0.709292 dto/min=0.684075 dto/std=0.009735 dtb/mean=0.020712 dtb/max=0.021224 dtb/min=0.020333 dtb/std=0.000239
[2025-12-31 12:16:33,343035][I][examples/vit:445:train_fn] iter=89 loss=6.995605 dt=0.719805 dtd=0.001665 dtf=0.022857 dto=0.674301 dtb=0.020983 loss/mean=6.958171 loss/max=7.008789 loss/min=6.910400 loss/std=0.030195 dt/mean=0.746791 dt/max=0.771666 dt/min=0.719805 dt/std=0.018258 dtd/mean=0.002107 dtd/max=0.003667 dtd/min=0.001627 dtd/std=0.000692 dtf/mean=0.022595 dtf/max=0.024857 dtf/min=0.021425 dtf/std=0.001067 dto/mean=0.701338 dto/max=0.722970 dto/min=0.674301 dto/std=0.017279 dtb/mean=0.020751 dtb/max=0.021155 dtb/min=0.020439 dtb/std=0.000192
[2025-12-31 12:16:34,121240][I][examples/vit:445:train_fn] iter=90 loss=6.995850 dt=0.726353 dtd=0.001734 dtf=0.022452 dto=0.681219 dtb=0.020947 loss/mean=6.958120 loss/max=7.021484 loss/min=6.881348 loss/std=0.034720 dt/mean=0.749065 dt/max=0.770316 dt/min=0.725386 dt/std=0.016169 dtd/mean=0.001976 dtd/max=0.002963 dtd/min=0.001595 dtd/std=0.000374 dtf/mean=0.022522 dtf/max=0.024208 dtf/min=0.021439 dtf/std=0.000793 dto/mean=0.703768 dto/max=0.723543 dto/min=0.680416 dto/std=0.015765 dtb/mean=0.020798 dtb/max=0.021766 dtb/min=0.020314 dtb/std=0.000326
[2025-12-31 12:16:34,889609][I][examples/vit:445:train_fn] iter=91 loss=6.919434 dt=0.725853 dtd=0.004117 dtf=0.022681 dto=0.678346 dtb=0.020710 loss/mean=6.969594 loss/max=7.016113 loss/min=6.892090 loss/std=0.027690 dt/mean=0.750679 dt/max=0.769203 dt/min=0.725032 dt/std=0.015449 dtd/mean=0.002085 dtd/max=0.004117 dtd/min=0.001563 dtd/std=0.000580 dtf/mean=0.022936 dtf/max=0.024109 dtf/min=0.021977 dtf/std=0.000661 dto/mean=0.704853 dto/max=0.723677 dto/min=0.678346 dto/std=0.015343 dtb/mean=0.020805 dtb/max=0.021383 dtb/min=0.020440 dtb/std=0.000236
[2025-12-31 12:16:35,646065][I][examples/vit:445:train_fn] iter=92 loss=6.972900 dt=0.743182 dtd=0.001629 dtf=0.022283 dto=0.698406 dtb=0.020864 loss/mean=6.962301 loss/max=7.013672 loss/min=6.867188 loss/std=0.034444 dt/mean=0.754227 dt/max=0.777483 dt/min=0.731005 dt/std=0.016673 dtd/mean=0.002124 dtd/max=0.003964 dtd/min=0.001629 dtd/std=0.000662 dtf/mean=0.022493 dtf/max=0.024020 dtf/min=0.021623 dtf/std=0.000811 dto/mean=0.708874 dto/max=0.731137 dto/min=0.686752 dto/std=0.015792 dtb/mean=0.020735 dtb/max=0.021157 dtb/min=0.020330 dtb/std=0.000213
[2025-12-31 12:16:36,405894][I][examples/vit:445:train_fn] iter=93 loss=6.955811 dt=0.737651 dtd=0.001657 dtf=0.027067 dto=0.688207 dtb=0.020721 loss/mean=6.956289 loss/max=7.032227 loss/min=6.893311 loss/std=0.029941 dt/mean=0.739291 dt/max=0.745214 dt/min=0.734239 dt/std=0.003010 dtd/mean=0.003118 dtd/max=0.005423 dtd/min=0.001554 dtd/std=0.001424 dtf/mean=0.022985 dtf/max=0.027067 dtf/min=0.022070 dtf/std=0.001058 dto/mean=0.692402 dto/max=0.699386 dto/min=0.688207 dto/std=0.003538 dtb/mean=0.020786 dtb/max=0.022114 dtb/min=0.020341 dtb/std=0.000411
[2025-12-31 12:16:37,197312][I][examples/vit:445:train_fn] iter=94 loss=6.941406 dt=0.748396 dtd=0.001713 dtf=0.022729 dto=0.703060 dtb=0.020894 loss/mean=6.956451 loss/max=7.019287 loss/min=6.906250 loss/std=0.030571 dt/mean=0.753825 dt/max=0.763172 dt/min=0.746897 dt/std=0.006138 dtd/mean=0.003277 dtd/max=0.005392 dtd/min=0.001634 dtd/std=0.001310 dtf/mean=0.022594 dtf/max=0.024555 dtf/min=0.021348 dtf/std=0.001013 dto/mean=0.707138 dto/max=0.716177 dto/min=0.699779 dto/std=0.006010 dtb/mean=0.020816 dtb/max=0.021603 dtb/min=0.020155 dtb/std=0.000312
[2025-12-31 12:16:37,943003][I][examples/vit:445:train_fn] iter=95 loss=6.995361 dt=0.725441 dtd=0.001654 dtf=0.023189 dto=0.679825 dtb=0.020773 loss/mean=6.960592 loss/max=7.019287 loss/min=6.904785 loss/std=0.028771 dt/mean=0.744709 dt/max=0.761425 dt/min=0.725441 dt/std=0.013316 dtd/mean=0.002854 dtd/max=0.004914 dtd/min=0.001627 dtd/std=0.001285 dtf/mean=0.022630 dtf/max=0.024223 dtf/min=0.021440 dtf/std=0.000929 dto/mean=0.698424 dto/max=0.715335 dto/min=0.679825 dto/std=0.012162 dtb/mean=0.020800 dtb/max=0.021414 dtb/min=0.020413 dtb/std=0.000289
[2025-12-31 12:16:38,685541][I][examples/vit:445:train_fn] iter=96 loss=6.924316 dt=0.730290 dtd=0.001679 dtf=0.022138 dto=0.685656 dtb=0.020817 loss/mean=6.952393 loss/max=7.033936 loss/min=6.894531 loss/std=0.028505 dt/mean=0.738270 dt/max=0.744446 dt/min=0.730290 dt/std=0.003330 dtd/mean=0.003186 dtd/max=0.005505 dtd/min=0.001642 dtd/std=0.001371 dtf/mean=0.022534 dtf/max=0.023826 dtf/min=0.021813 dtf/std=0.000534 dto/mean=0.691715 dto/max=0.697518 dto/min=0.685656 dto/std=0.003466 dtb/mean=0.020836 dtb/max=0.021774 dtb/min=0.020360 dtb/std=0.000338
[2025-12-31 12:16:39,454027][I][examples/vit:445:train_fn] iter=97 loss=6.977295 dt=0.747579 dtd=0.001707 dtf=0.026247 dto=0.698855 dtb=0.020769 loss/mean=6.967041 loss/max=7.055420 loss/min=6.875977 loss/std=0.035210 dt/mean=0.745013 dt/max=0.753496 dt/min=0.739475 dt/std=0.005213 dtd/mean=0.002956 dtd/max=0.004715 dtd/min=0.001561 dtd/std=0.000987 dtf/mean=0.022581 dtf/max=0.026247 dtf/min=0.021254 dtf/std=0.001179 dto/mean=0.698686 dto/max=0.706904 dto/min=0.692804 dto/std=0.004965 dtb/mean=0.020791 dtb/max=0.021900 dtb/min=0.020284 dtb/std=0.000342
[2025-12-31 12:16:40,195874][I][examples/vit:445:train_fn] iter=98 loss=6.967773 dt=0.732756 dtd=0.001660 dtf=0.025091 dto=0.685154 dtb=0.020851 loss/mean=6.942464 loss/max=7.022705 loss/min=6.878174 loss/std=0.035641 dt/mean=0.739319 dt/max=0.745367 dt/min=0.732756 dt/std=0.003418 dtd/mean=0.002662 dtd/max=0.003719 dtd/min=0.001660 dtd/std=0.000606 dtf/mean=0.022934 dtf/max=0.025091 dtf/min=0.021985 dtf/std=0.000785 dto/mean=0.692973 dto/max=0.697970 dto/min=0.685154 dto/std=0.002909 dtb/mean=0.020750 dtb/max=0.021213 dtb/min=0.020436 dtb/std=0.000199
[2025-12-31 12:16:40,950324][I][examples/vit:445:train_fn] iter=99 loss=6.977539 dt=0.745598 dtd=0.002238 dtf=0.023939 dto=0.698594 dtb=0.020827 loss/mean=6.956676 loss/max=7.008301 loss/min=6.899658 loss/std=0.022355 dt/mean=0.741473 dt/max=0.748397 dt/min=0.733044 dt/std=0.004455 dtd/mean=0.003068 dtd/max=0.005405 dtd/min=0.001547 dtd/std=0.001378 dtf/mean=0.023009 dtf/max=0.025263 dtf/min=0.021605 dtf/std=0.000825 dto/mean=0.694567 dto/max=0.701266 dto/min=0.687203 dto/std=0.004004 dtb/mean=0.020828 dtb/max=0.021858 dtb/min=0.020438 dtb/std=0.000346
[2025-12-31 12:16:41,771715][I][examples/vit:445:train_fn] iter=100 loss=7.000244 dt=0.760887 dtd=0.001625 dtf=0.026943 dto=0.711486 dtb=0.020833 loss/mean=6.960460 loss/max=7.000244 loss/min=6.919434 loss/std=0.022609 dt/mean=0.746579 dt/max=0.762490 dt/min=0.731703 dt/std=0.012070 dtd/mean=0.002105 dtd/max=0.003019 dtd/min=0.001625 dtd/std=0.000459 dtf/mean=0.022804 dtf/max=0.026943 dtf/min=0.021464 dtf/std=0.001315 dto/mean=0.700953 dto/max=0.715302 dto/min=0.686720 dto/std=0.011352 dtb/mean=0.020716 dtb/max=0.021132 dtb/min=0.020289 dtb/std=0.000214
/lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/src/ezpz/history.py:2223: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
Consider using tensor.detach() first. (Triggered internally at /lus/tegu/projects/datasets/software/wheelforge/repositories/pytorch_2p8_rel_07_18_2025/pytorch/torch/csrc/autograd/generated/python_variable_methods.cpp:835.)
x = torch.Tensor(x).numpy(force=True)
[2025-12-31 12:16:42,102625][I][ezpz/history:2385:finalize] Saving plots to /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/mplot (matplotlib) and /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot (tplot)
train_dt train_dt/min
┌─────────────────────────────────┐ ┌────────────────────────────────┐
0.792┤ ▗▌ │0.7469┤-- - -- │
0.777┤ ▐▌ ▗ │0.7304┤--------------------------------│
│ ▐▌▗▌ ▗█ ▗ ▗▌ ▟ │0.7138┤-- - - - --- ---------- │
0.762┤ ▐▌▐▌ █▌▌ █ ▐▌ ▟ █ ▖ ▞│0.6973┤- │
0.747┤▖▖ █▌▐▙▌▐█▌▌▖▗▌ █▜ ▗▐▌▌█▗█▐▌▗▟▟▌│ └┬───────┬───────┬──────┬───────┬┘
0.732┤█▌▟▗▜▌▌▛█▟▛▌█▚▐▌▗▜▝▙█▞▜▌█▐█▌▌▐▜▛▌│ 1.0 23.5 46.0 68.5 91.0
│██▘▀ ▝▘▘▝ ▌▘█ ▀█▞ █▌ ▙▘▘▝▌▙▀▐▘ │train_dt/min iter
0.717┤█▝ ▝ ▀▌ ▀▌ ▝ ▝ │ train_dt/std
0.702┤▜ │ ┌────────────────────────────────┐
└┬───────┬───────┬───────┬───────┬┘0.0225┤* * * *** ******** │
1.0 23.5 46.0 68.5 91.0 0.0188┤** ***** **** *************** │
train_dt iter 0.0113┤***** ** ** * * ******** **** *│
train_dt/mean 0.0075┤***** **** *** ******** ***│
┌────────────────────────────────┐0.0000┤** * * * │
0.7657┤ · · │ └┬───────┬───────┬──────┬───────┬┘
0.7588┤ ·· ·· │ 1.0 23.5 46.0 68.5 91.0
│ ··· · ·· · · · · │train_dt/std iter
0.7519┤ ······· ·· ··· · ········· │ train_dt/max
0.7450┤ ········· ···· ········· ···· ·│ ┌─────────────────────────────────┐
│····· ···· ··· ········ ······│0.793┤ + ++ + ++ ++ │
0.7381┤···· ·· ·· ···· ·· │0.783┤++ + +++ +++ ++++ ++++ +++++ │
0.7312┤··· │0.763┤+++++++++ + ++ + ++ ++++++ ++++ +│
│·· │0.752┤+++ + ++++ +++ ++++++ ++ +++│
0.7243┤ · │0.732┤ ++ │
└┬───────┬───────┬──────┬───────┬┘ └┬───────┬───────┬───────┬───────┬┘
1.0 23.5 46.0 68.5 91.0 1.0 23.5 46.0 68.5 91.0
train_dt/mean iter train_dt/max iter
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dt.txt
┌─────────────────────────────────────────────────────────────────────────┐
0.793┤ ++ train_dt/max + │
│ -- train_dt/min ++ + + + │
│ ·· train_dt/mean +++ + + + ++ ++ + ++ + + │
│ ▞▞ train_dt +++++ ++ ++ ++ + +++ ++++++ │
0.777┤ + ++ +█++ + ++++▖ +++ ++ + + ++ + +++ ++++++ + │
│ ++ ++ +█++++ +++▐▌ ++ + + ++ ++ +▖ + +++ + ++++ ++ │
│ +++ ++ +█+++▟ ++ ▐▌+ + + ++ +++ ▐▌ + +++ + ▖+++++++ │
│++++ ++ +█+++█ +▗ ▐▌+ + + ++ +++ ▐▌ + +++ +▐▌+++ + │
0.761┤+++ ++ + +▗▜ ++█ +█·▐▌+ + + ▟ ++ +++ ▐▌ + ++▖ +▐▌ ++ +++ ▗│
│+++ ++·· +▐▐ ++█ +█·▐▐+++ + ·█ ++ + ++▌▌ + +▐▌ +▐▌ + + + +▌│
│+++ ++·· ▟▐▐ +▐▐ + ▗▌█·▐▐+++ +· · █+ + + ++▌▚ + ·▐▌+ ▐▌ ▖ ··+ ++▌│
│+++··+·· █▐▐ ·▐▐ ▗+++▐▌█·▞▐ +++ +▖ ·· █+ + +·++▌▐+▗··▐▌+·▞▌·▐▌ ····+++▐ │
0.745┤++· ·+·· █▐▐··▐▐+█· ▐▌█ ▌▐ ·· ▐▌ · ▌▌▞▌+ · ▖▐·▐+█ ·▐▌+▖▌▐·▌▐· ·▟+▗▌▐·│
│▌+▗ ·· ·+▌█▐ ▞▐·█·▗·▌█▐ ▌▐··▖·+▐▌ ▌▌▌▌ +·▐▌▐ ▐▐▐ ·▌▌▐▌▌▐▐··▌ ▗▌█·▐▌▌ │
│▌+█ ·▟ ··▌█▐ ▗▘▐·▛▖█ ▌█▐ ▌▝▖▞▚··▞▌ ▌▙▘▐▟·▟▐▌▐ ▐▞▐ ▌▌▐▐▌▐▐ ▌ ▐▝█ ▐▐▌ │
│▌·█ ▖▛▖ ▐-█▐ ▐-▐▗▘▌▛▄▌█▐ ▌ ▙▘▝▖ ▌▚ ▗▘█-▐▛▄▜▐▌▞ ▝▌▐ ▌▌▐▝▌ █ ▌ ▐--▌▐▐▌ │
│▌·█▐▚▌▚▟▌ ▝▝▖▌ ▐▞ █ █ ▌▌-█ -▝▖▌▐ ▌ ▝- ▘▜▐▞▌▌-- -▌ ▌▌▌- █ -▌ ▐--▌▐-▘-│
0.729┤▌▐▐▐▝▌--▘ -▝▘ ▘ ▜ █-▝▌-█ ▝ ▐▗ ▐-- -- ▐▌▌▌-- -▌▞-▜ - -█--▌▗▐ -▙▘ │
│▚▐▐▌ - -- ▝ - -█ ▐█ ▌-- -- ▐▌▙▘-- -█--- ---▝--▌▌▀ ▝ │
│▐▐▝▌ - - ▜ ▝▛▄▘-- - ▐▌█ - -█--- -----▜ │
│▐▞- -█ - ▘█ ▝ - - - │
0.713┤▐▌- ▝ ▜ │
│▐▌ │
│▐▌ │
│-▘ │
0.697┤ - │
└┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬┘
1.0 23.5 46.0 68.5 91.0
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dt_summary.txt
train_dt/mean hist train_dt/max hist
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
22.0┤ ████ │23.0┤ ███ │
18.3┤ ████ │19.2┤ ███ │
│ ████████ │ │ ███ │
14.7┤ ██████████████ │15.3┤ ███ │
11.0┤ ██████████████ │11.5┤ ███ ████ ████ │
│ █████████████████ │ │ ███ ████ ████ │
7.3┤ █████████████████ │ 7.7┤ ████████████████████████ │
3.7┤ ████████████████████████ │ 3.8┤ ███████████████████████████████│
│ ███████████████████████████│ │ ███████████████████████████████│
0.0┤███ ███████████████████████████│ 0.0┤██████████████████████████████████│
└┬───────┬────────┬───────┬───────┬┘ └┬───────┬────────┬───────┬───────┬┘
0.722 0.734 0.745 0.756 0.768 0.730 0.746 0.763 0.779 0.795
train_dt/min hist train_dt/std hist
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
29.0┤ ███ │26.0┤ ████ │
│ ███ │ │ ████ │
24.2┤ ███████ │21.7┤ ████ │
19.3┤ ███████ │17.3┤ ████ │
│ ███████ │ │ ████ │
14.5┤ ███████ │13.0┤ ████ ███████│
│ ████ ███████ │ │ ████ ███████│
9.7┤ ██████████████ │ 8.7┤ ████ ████ ████ ██████████│
4.8┤ █████████████████████ │ 4.3┤ ███████████████████████████████│
│ ████████████████████████│ │██████████████████████████████████│
0.0┤███ ████████████████████████│ 0.0┤██████████████████████████████████│
└┬───────┬────────┬───────┬───────┬┘ └┬───────┬────────┬───────┬────────┘
0.695 0.709 0.722 0.736 0.749 -0.0010 0.0051 0.0113 0.0174
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dt_hist.txt
train_dtb train_dtb/min
┌──────────────────────────────┐ ┌──────────────────────────────┐
0.021110┤ ▌ ▖ │0.020440┤ -- - -------│
0.021031┤ ▌ ▗▐▌ ▗ │0.020336┤ - ------ ------------------│
│ ▌ █▐▙▌ ▛▖ │0.020233┤----------------- -- -- │
0.020952┤ ▖ ▌ ▌▖ ▗ ███▌ ▐▌▌ │0.020129┤ - -- - - - │
0.020873┤ ▗▚▗ ▐▌▖ ▙▌ █▙▖▞▜▟▐█▜▙▙▜▌▌▟ ▖│ └┬──────┬───────┬──────┬──────┬┘
0.020793┤▗ ▐▐█▗▌█▙▚▘▌▖█▌█▌▝█▝▜▐▝▜▐ ███▀│ 1.0 23.5 46.0 68.5 91.0
│▐▟▞▐▐█ ▜▛ █▙▀▌█ ▜ ▐▝ ▝ █▌▀ │train_dtb/min iter
0.020714┤▀▀▘ ▘ ▌ ██ ▌▝ ▝ ▐ ▝▘ │ train_dtb/std
0.020635┤ ▌ ▝▜ ▝ │ ┌──────────────────────────────┐
└┬──────┬───────┬──────┬──────┬┘0.000479┤ * * │
1.0 23.5 46.0 68.5 91.0 0.000431┤*** ** *** * * * │
train_dtb iter 0.000335┤*** * *** ******* * * ****│
train_dtb/mean 0.000288┤******************************│
┌──────────────────────────────┐0.000192┤ * ** * ** **│
0.020960┤ · │ └┬──────┬───────┬──────┬──────┬┘
0.020900┤ · │ 1.0 23.5 46.0 68.5 91.0
│ ··· · │train_dtb/std iter
0.020841┤ ····· · ··│ train_dtb/max
0.020781┤ · · · ······· · · ····│ ┌───────────────────────────────┐
│··· · ····· ······· ······ ··│0.02236┤ + + │
0.020722┤··· ··········· ··· ····· ·│0.02214┤ + + +++ ++ + ++ │
0.020662┤ · ···· ····· · · · ··· │0.02170┤+++ + + ++ ++++ ++ ++ ++++│
│ ··· · · │0.02148┤++++++++++++++++++ ++++++++++++│
0.020603┤ · │0.02104┤++ ++ + + +++ + +++ + +++ +│
└┬──────┬───────┬──────┬──────┬┘ └┬───────┬──────┬───────┬──────┬┘
1.0 23.5 46.0 68.5 91.0 1.0 23.5 46.0 68.5 91.0
train_dtb/mean iter train_dtb/max iter
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dtb.txt
┌───────────────────────────────────────────────────────────────────────┐
0.02236┤ ++ train_dtb/max + │
│ -- train_dtb/min ++ │
│ ·· train_dtb/mean ++ + │
│ ▞▞ train_dtb ++ + ++ + │
0.02199┤ ++++ + ++ + ++ │
│ + + ++++++ ++ ++ ++ + │
│ + ++ + + ++++++ ++ ++ ++ +++ │
│ ++ ++ ++ + + ++++++ ++ ++ + ++ +++ │
0.02162┤ ++ ++ + ++ ++ + + ++ ++ ++ ++ +++ ++ │
│+++ ++ + + + ++ +++++ ++ ++ + ++ ++ +++ ++ │
│+++ ++ ++ ++ + + + + +++++ ++ ++ ++ ++ ++ +++ ++ │
│+++ ++ ++ +++ + + ++ + +++++++ ++ + +++ + +++ + ++ + ++ │
0.02124┤++++ + ++ + +++ + +++ + +++ + + ++ +++ + +++ + ++ ++ │
│++++ + +++++ + + + + + + + + + ++ ++ ++ + ++ │
│ + + ++ + + ▗ + ++ ▖ + ++ +│
│ █ ▖ ▐▌ ▗ + ▗▚▖ │
│ ▗ ▗ ▗▜ ▖ ▖ · ▐▚ ▟▐▐ ▛▖ ▖ ▞▟ ▐ ▖ │
0.02087┤ ▗▀▌▗▗▌ ▖▞▀▖▗▌ ▄▖▗▞▚█ ▖ ▐▐▞▌▞▀▌▗▗▀▝▀▄▌·▌▝▄█▐ █ ▚▞▌▗▀▚▌▝ ▝▄▌▐▌▗▗▚▄│
│·▞▄▚▄▞▘ ▚▘▘▐▞▟▝▘·▝▌▌▞·▝▘· ·█▝▌▞ ▘▙▘ ▝▌▀·· ▝▟ ···█·▝· ·▝▀··▘·· █▝▌▝▘▘ ·│
│▀·▝· · ····▘ ·· ▜· ·· ··▜·▝▌ ▝ · · · ▜ ·· · ▝ │
│ · │
0.02050┤ │
│ - - - ----- - -- -- --- - ---- │
│- - ---------- - ------ - ---- --- - -- - - ---- --- -- -- - -│
│ -- -- - - - --- - - ---- - -- - -- ---- - │
0.02013┤ -- - - - - - │
└┬─────────────────┬────────────────┬─────────────────┬────────────────┬┘
1.0 23.5 46.0 68.5 91.0
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dtb_summary.txt
train_dtb/mean hist train_dtb/max hist
┌──────────────────────────────────┐ ┌────────────────────────────────────┐
23.0┤ ███ │24┤ ███ │
19.2┤ ███ ████ │20┤ ███████ │
│ ███ ████ │ │ ███████ │
15.3┤ ██████████ │16┤███████████ │
11.5┤ ██████████████ │12┤███████████ │
│ ██████████████ │ │███████████ │
7.7┤ ██████████████████ │ 8┤██████████████████ │
3.8┤ ████████████████████████ │ 4┤██████████████████ ████ │
│ ████████████████████████ │ │████████████████████████████████ │
0.0┤██████████████████████████████████│ 0┤████████████████████████████████████│
└┬───────┬────────┬───────┬────────┘ └┬────────┬────────┬───────┬─────────┘
0.02059 0.02068 0.02078 0.02088 0.02098 0.02134 0.02170 0.02206
train_dtb/min hist train_dtb/std hist
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
21.0┤ ████ │21.0┤ ████ │
│ ████ │ │ ███████ │
17.5┤ ████ │17.5┤ ███████████ │
14.0┤ ████ ████ │14.0┤ ███████████ │
│ ████ ████ │ │ ███████████ │
10.5┤ ████ ████ │10.5┤ ███████████ │
│ ████████████████████ ████│ │█████████████████████ │
7.0┤ ████████████████████ ████│ 7.0┤█████████████████████ │
3.5┤███████████████████████████ ████│ 3.5┤████████████████████████ │
│██████████████████████████████████│ │████████████████████████ ███████│
0.0┤██████████████████████████████████│ 0.0┤██████████████████████████████████│
└┬───────┬────────┬───────┬────────┘ └┬───────┬────────┬───────┬────────┘
0.02011 0.02020 0.02028 0.02037 0.000179 0.000257 0.000335 0.000413
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dtb_hist.txt
train_dtd train_dtd/min
┌────────────────────────────────┐ ┌───────────────────────────────┐
0.0122┤ ▟ │0.00312┤- │
0.0104┤ █ │0.00259┤- │
│ █ │0.00206┤-- │
0.0086┤ █ │0.00153┤-------------------------------│
0.0069┤ █ │ └┬───────┬──────┬───────┬──────┬┘
0.0051┤ ▗ █ │ 1.0 23.5 46.0 68.5 91.0
│ █ █ ▗ ▐ ▗ │train_dtd/min iter
0.0034┤▖ █ █ ▖ █ ▐ █ │ train_dtd/std
0.0016┤▚▄▄▄▄█▄▄▄▄▄▛▄▄▄▄▙▄▄▟▚▄▄▄▛▄▟▄█▄▄▙│ ┌───────────────────────────────┐
└┬───────┬───────┬──────┬───────┬┘0.00207┤ * │
1.0 23.5 46.0 68.5 91.0 0.00174┤ * **** *** * * *** ***│
train_dtd iter 0.00110┤********** **** ***************│
train_dtd/mean 0.00077┤******** **** **** ********* **│
┌───────────────────────────────┐0.00013┤** * ** *** **** ** ***** │
0.00410┤·· │ └┬───────┬──────┬───────┬──────┬┘
0.00371┤·· │ 1.0 23.5 46.0 68.5 91.0
│·· · · │train_dtd/std iter
0.00332┤····· ··· ·· ·· · · · │ train_dtd/max
0.00293┤·········· ·· ··· ···· ···│ ┌────────────────────────────────┐
│········ · ··· ········ ·····│0.0122┤ + │
0.00254┤········ · ·· · ············ ··│0.0105┤ + │
0.00215┤······· ···· ·············· ·│0.0071┤ + │
│······· ··· ···· ········· ·│0.0054┤+++++++++++++++ ++++++++++++++++│
0.00176┤·· · ·· ··· ···· ·· ····· │0.0020┤++++++++ +++ +++++++++++++++ +│
└┬───────┬──────┬───────┬──────┬┘ └┬───────┬───────┬──────┬───────┬┘
1.0 23.5 46.0 68.5 91.0 1.0 23.5 46.0 68.5 91.0
train_dtd/mean iter train_dtd/max iter
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dtd.txt
┌────────────────────────────────────────────────────────────────────────┐
0.0122┤ ++ train_dtd/max ▗▌ │
│ -- train_dtd/min ▐▌ │
│ ·· train_dtd/mean ▐▌ │
│ ▞▞ train_dtd ▐▌ │
0.0104┤ ▐▌ │
│ ▐▌ │
│ ▐▌ │
│ ▐▌ │
0.0086┤ ▐▌ │
│ ▐▌ │
│ ▐▌ │
│ ▐▌ │
0.0068┤ ▐▚ │
│ ▐▐ │
│ ▐▐ │
│ ▐▐ │
│+ + + + + + +++ ▐▐ ++ ++ + +++ + ++ + + ++ + + │
0.0051┤+++ ++ ++ ▗▌+ +++ + ▐▐++++ + +++ + ++++ + ++++ + + ++++ │
│+++ +++ ++ ▐▌+ + ++ + ▐▐++ + + +++ ++++++ + + + + ▟++ + ++ │
│·+·+ ++ ++ ▐▌+ + + + ▐▐++ + +++ ++++++ + + +▗+ +█++ ▗+ ++ │
│·+·+ +++ · ▐▌+ + + + ▐▐++· + + +++ ++++++ + + +█+ ++█+++█ ++ │
0.0033┤▖+·+ ·++··+▞▌·++······+++▐▐+·· ·· +++ +·+···+++·+·· + +█+ ++█+++█·· · ·+│
│▌··+··++··+▌▌·+· · ·+++▐▐·· · ·++++ ··· ·+▗··· · +··▛▖ +▐▐+·▗▜· · ·· │
│▚··+···+·· ▌▌··· · ·+··▐▐·· ·+++· ··· ··█··· · · ·▌▌++▐▐··▐▐· · │
│▐·-· ·· ··▌▌· · · ·▐▐·· ···▗·· · ▐▐· · ·· ·▌▌··▐▐··▐▐· ▟·│
0.0015┤ ▀▀▀▀▀▀▀▀▀▀▘▝▀▀▀▀▀▀▀▀▄▀▀▀▀-▀▀▀▀▀▀▀▀▀▘▀▀▀▀▀▀▀▀-▀▚▞▀▄▀▀▀▀▘▝▀▀▀-▀▀▀▝▄▀▀▀▀▘▚│
└┬─────────────────┬─────────────────┬────────────────┬─────────────────┬┘
1.0 23.5 46.0 68.5 91.0
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dtd_summary.txt
train_dtd/mean hist train_dtd/max hist
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
27.0┤████ │31.0┤ ████ │
22.5┤████ │25.8┤████ ████ │
│████ │ │████ ████ │
18.0┤████ │20.7┤████ ████ │
13.5┤████ │15.5┤████ ███████ │
│███████ ████ │ │██████████████ │
9.0┤███████ ██████████████ │10.3┤██████████████ │
4.5┤████████████████████████ │ 5.2┤██████████████ │
│███████████████████████████ ████│ │██████████████ │
0.0┤██████████████████████████████████│ 0.0┤█████████████ ████│
└┬───────┬────────┬───────┬────────┘ └┬───────┬────────┬───────┬────────┘
0.00166 0.00230 0.00293 0.00357 0.0016 0.0043 0.0071 0.0098
train_dtd/min hist train_dtd/std hist
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
88.0┤████ │22.0┤████ ████ │
│████ │ │████ ████ │
73.3┤████ │18.3┤████ ████ │
58.7┤████ │14.7┤████ ████ │
│████ │ │████ ███████ │
44.0┤████ │11.0┤████ ███ ███████ │
│████ │ │██████████ ███████████ │
29.3┤████ │ 7.3┤██████████ ███████████ │
14.7┤████ │ 3.7┤████████████████████████ │
│████ │ │████████████████████████ │
0.0┤███████ ████│ 0.0┤████████████████████████ ████│
└┬───────┬────────┬───────┬────────┘ └┬───────┬────────┬───────┬────────┘
0.00146 0.00189 0.00233 0.00276 0.00004 0.00057 0.00110 0.00163
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dtd_hist.txt
train_dtf train_dtf/min
┌───────────────────────────────┐ ┌───────────────────────────────┐
0.02730┤ ▟ ▖ ▗│0.02219┤ -- - - - --- │
0.02635┤ █ ▗ ▌▗▌│0.02173┤ -----------------------------│
│ ▗█ ▐ ▌█▌│0.02128┤-------- ---- --- -------------│
0.02540┤ ▗██ ▗▌ ▗ ▐ ▗ ▖ ▖ ▌▛▌│0.02083┤- - - │
0.02445┤ ▐██ ▖▌▌▗▌ █ ▐ █▗▐▌▐▌▗ ▌▌▌│ └┬───────┬──────┬───────┬──────┬┘
0.02350┤ ▐██▟ ▌▌▌▐▌▗▌ █▄█▟▜█▐▐▟▌█ ▌▌▘│ 1.0 23.5 46.0 68.5 91.0
│▐▌▖▟███▗▌▌▚▟██▌▟▐▐█▛▐█▐▐ ▝▝▖▙▌ │train_dtf/min iter
0.02254┤▐▜▝▌▝▘▀▛▛▘ ▀ ▀▀▘▐▝▜▘▝▘▜▝ ▀▛▌ │ train_dtf/std
0.02159┤▌ ▝ │ ┌───────────────────────────────┐
└┬───────┬──────┬───────┬──────┬┘0.00132┤ ** * * * * **│
1.0 23.5 46.0 68.5 91.0 0.00113┤ *** **** ** ********** ***│
train_dtf iter 0.00075┤ ******************************│
train_dtf/mean 0.00056┤***** *** ****** *** * * * * │
┌───────────────────────────────┐0.00019┤* * │
0.02330┤ · │ └┬───────┬──────┬───────┬──────┬┘
0.02294┤·· · · ·· · · ·│ 1.0 23.5 46.0 68.5 91.0
│···· · ······ ····· ········│train_dtf/std iter
0.02258┤······························ │ train_dtf/max
0.02222┤····· · · ··· ·· ···· · │ ┌───────────────────────────────┐
│··· · │0.02730┤ + + +│
0.02186┤· │0.02635┤ ++ + + ++ + +++++│
0.02150┤· │0.02445┤ ++++ ++++++++++++++++++++++++ │
│· │0.02350┤++++ + + +++ + + + + + │
0.02114┤· │0.02159┤+ │
└┬───────┬──────┬───────┬──────┬┘ └┬───────┬──────┬───────┬──────┬┘
1.0 23.5 46.0 68.5 91.0 1.0 23.5 46.0 68.5 91.0
train_dtf/mean iter train_dtf/max iter
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dtf.txt
┌────────────────────────────────────────────────────────────────────────┐
0.0273┤ ++ train_dtf/max │
│ -- train_dtf/min ▗▌ ▗│
│ ·· train_dtf/mean ▐▌ +▌│
│ ▞▞ train_dtf ▐▌ +▌│
0.0262┤ █+ ▟ ▐▌ ▗▌+▌│
│ █+ █ ▐▌ ▐▌+▌│
│ ▗▌█+ █ ▐▌ ▐▚+▌│
│ ▗▐▌█+ █ + ▐▌ ▐▐▗▘│
0.0251┤ █▐▌█+ ▖ + + █ ++ + ▐▌ ▐▝▟ │
│ █▐▌▛▖+ ▗▀▝▖ ++ ▖+ +█ ▟ +▖+ +▖ ++ ▐▌ ▐+█ │
│ + +█▐▌▌▌+ ▐ ▌ + ▖ + +▐▌+ ++█ + █ ▐▌ + ▐▌ ++ ▐▌ ▐ █ │
│ ++++ ++█▐▌▌▌+ + ▐ ▌+ ▐▌++ + ++▐▌++ +▛▖+ █ +▖ ▐▌ + ▐▌+++ ▌▌ ▐ █ │
0.0241┤ ++ + ++█▐▚▌▌ ▖+ ▗▌+▐ ▌+ ▐▌++ ▗ +++▐▌++ +▌▌+ █+▐▝▖+▐▐ + ▞▌+▗▌++▌▌+▐ █ │
│ ++ + +▐▝▟▐▌▌▐▌++▐▌+▞ ▌+ ▐▌ + █++++▐▌ ▗▌▌▌▗+█+▐+▌+▐▐ ++▌▌+▐▌ ▌▌ ▐ ▝ │
│ ++ + +▐ █▐▌▌▐▌ +▐▌ ▌ ▌+ ▐▌ +█+ ++▞▚▞▄▐▌▌▌█▐+▌▐+▌ ▐ ▌▗▖▌▌+▌▌ ▌▌ ▐ │
│ +· + ++▐ █▐▌▌▐▌ ▐▚ ▌ ▌+ ▌▌ ▖█ ▌▐▌▐▐▌▌▌█▌ ▌▐+▌ ▞ ▌▌▝▘▚▞▘▐ ▌▌ ▐ │
│ ▞▜ + ▗█ █▐▌▌▐▚ ▐▐▗▘ ▌+ ▌▌▟▐▌▌▌ ▗█ ▐▌▝▟▚▌▙▘▘ ▌▐+▌ ▌ ▙▘ ▐ ▌▌▟▐ │
0.0230┤+▌▐ ▗▌ ▐█ █▐▌▌▐▐ ▟▐▐▐· ·▚▗▌▌▝▐▌▐▌▌ ▐▝ ▐▌ █▐▌█ ▌▐ ▌▟▌·█· · ▝▖·▌▙▜▐···│
│+▌▐·▞▌ ▌▝·█▝▌▚▘▐·▛▟▝▟····▀▝▌··▘·▘▌·▌· ▐▌·▜▐▌▜· ·▚▐·▝▐▌·▝· ·····▌▟▌▝▝▟· │
│+▌▐·▌▝▀▌ ·▝ ▐▐·▜·▝· ·· ·▝▀▘ ·▝▌ ▐▌ ···▀· ▐▌· ·· · ▝▐▌ █ │
│▐··▀ ·· ▀ - - ▘ ·▝▌ ▘ ▜ │
0.0219┤▞ - -- --- - -- - - - -- -- - -- -- │
│▌ ----- - - - - --- --- ---- --------- -- - ---- -- --- - ---- │
│·-- - - -- --- - - --- - - - - -- --- - -- - -- -- - -│
│· - - - │
0.0208┤- │
└┬─────────────────┬─────────────────┬────────────────┬─────────────────┬┘
1.0 23.5 46.0 68.5 91.0
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dtf_summary.txt
train_dtf/mean hist train_dtf/max hist
┌──────────────────────────────────┐ ┌────────────────────────────────────┐
37.0┤ ████ │24┤ ████████ │
30.8┤ ████ │20┤ ████████ │
│ ████ │ │ ███████████ │
24.7┤ ████ │16┤ ███████████ │
18.5┤ ███████ │12┤ ██████████████ │
│ ██████████ │ │ ██████████████ │
12.3┤ ██████████████ │ 8┤ ██████████████ │
6.2┤ ██████████████ │ 4┤ ██████████████████ │
│ ██████████████████ │ │ ██████████████████ ███████│
0.0┤███ ████████████████████████│ 0┤████ █████████████████████████████│
└┬───────┬────────┬───────┬────────┘ └┬────────┬────────┬───────┬────────┬┘
0.02104 0.02163 0.02222 0.02281 0.0213 0.0229 0.0244 0.0260 0.0276
train_dtf/min hist train_dtf/std hist
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
23.0┤ ████ │19.0┤ ████ │
│ ████ │ │ ███████ │
19.2┤ ████ │15.8┤ ███████████ │
15.3┤ ███████████ │12.7┤ ███████████ │
│ ██████████████ │ │ ███████████ │
11.5┤ ██████████████ │ 9.5┤ ██████████████ ███ │
│ ██████████████ │ │ ██████████████ ███ │
7.7┤ ██████████████ ████ │ 6.3┤ ███████████████████████████│
3.8┤ █████████████████████ │ 3.2┤ ███████████████████████████│
│ ███████████████████████████████│ │██████████████████████████████████│
0.0┤██████████████████████████████████│ 0.0┤██████████████████████████████████│
└┬───────┬────────┬───────┬────────┘ └┬───────┬────────┬───────┬────────┘
0.02077 0.02114 0.02151 0.02188 0.00014 0.00044 0.00075 0.00106
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dtf_hist.txt
train_dto train_dto/min
┌─────────────────────────────────┐ ┌────────────────────────────────┐
0.743┤ ▗▌ │0.6998┤-- - - -- │
0.729┤ ▐▌ ▟ ▖ │0.6840┤--------------------------------│
│ ▐▌▗▌ ▗█ ▗ ▐▌ ▟ │0.6683┤-- - - - --- --- ------ │
0.715┤ ▐▌▐▌ ██ █ ▐▌ ▟ █ ▖ ▞│0.6526┤- │
0.700┤▖▖ █▌▐▙▌▐█▌▌▖▗▌ █▜ ▗▐▌▌█▐█▐▌▗▟▗▌│ └┬───────┬───────┬──────┬───────┬┘
0.686┤█▙▛▄▜▌▌▛█▞▛▌█▚▐▌▗▜▝▙█▐▜▌█▐█▌▌▐█▛▌│ 1.0 23.5 46.0 68.5 91.0
│█▜▘▝ ▝▌▘▝ ▌▘█ ▀█▞ █▌ ▙▘▘▝▌▙▜▐ │train_dto/min iter
0.672┤█ ▝ ▀▌ ▀▌ ▝ ▝ │ train_dto/std
0.657┤▜ │ ┌────────────────────────────────┐
└┬───────┬───────┬───────┬───────┬┘0.0228┤* * ** ******** │
1.0 23.5 46.0 68.5 91.0 0.0192┤** ***** **** *************** │
train_dto iter 0.0119┤******** ** * * ******** **** *│
train_dto/mean 0.0082┤***** **** *** ******** ****│
┌────────────────────────────────┐0.0010┤*** * **** ** ****** * ** │
0.7200┤ · · │ └┬───────┬───────┬──────┬───────┬┘
0.7128┤ ·· ·· │ 1.0 23.5 46.0 68.5 91.0
│ ··· · ·· · · · · ·· │train_dto/std iter
0.7056┤ ······· ··· ···· · ········· │ train_dto/max
0.6985┤·········· ···· ········· ······│ ┌─────────────────────────────────┐
│····· ···· ··· ········ ······│0.747┤ + +++ + ++ ++ │
0.6913┤···· ·· ·· ····· ·· │0.737┤++ + +++ +++ ++++ ++++ +++ + │
0.6842┤·· │0.716┤+++++++++ ++++ + ++ ++++++ ++++ +│
│·· │0.706┤+++ + ++++ +++ ++++++ ++ +++│
0.6770┤ · │0.685┤ + │
└┬───────┬───────┬──────┬───────┬┘ └┬───────┬───────┬───────┬───────┬┘
1.0 23.5 46.0 68.5 91.0 1.0 23.5 46.0 68.5 91.0
train_dto/mean iter train_dto/max iter
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dto.txt
┌─────────────────────────────────────────────────────────────────────────┐
0.747┤ ++ train_dto/max + │
│ -- train_dto/min ++ + + │
│ ·· train_dto/mean +++ + + + ++ ++ ++ + + │
│ ▞▞ train_dto +++++ ++ ++ ++ + + + ++++++ │
0.731┤ + ++ +█++ + +++▗▌ +++ ++ + ++ + +++ ++++++ + │
│ ++ ++ +█++++ +++▐▌ ++++ + + ++ +▖ + +++ + ++++ ++ │
│ +++ ++ +█+++▟ ++ ▐▌+ + + + ++ +++ ▐▌ + +++ + ▖+++++++ │
│++++ ++ +█+++█ + ▐▌+ + + ++ +++ ▐▌ + +++ +▐▌+++ + │
0.716┤+++ ++ + + █+++█ +▟·▐▌+ + + ▗ ++ +++ ▐▌ + +++ +▐▌ ++ +++ +│
│+++ ++·· +▐▐ ++█ +█·▐▌+++ + ·█ ++ + ++▐▌ + +▗▌ +▐▌ + + + +▗│
│+++ ++·· +▐▐ +▐▐ + ▖█·▐▌+++ +· · █+ + + ++▌▚ +··▐▌+ ▐▌ ▖ · + +▌│
│+++··+·· ▟▐▐ ·▐▐ ++ ▐▌█·▌▌ +++ +· ·· █+ + +·++▌▐+· ·▐▌+·▞▌·▐▌ ····+ +▗▘│
0.700┤++· ·+·· █▐▐··▐▐+▟+++▐▌█ ▌▐ ·+ ▗▌ · ▛▖▗▌+ ···+▌▐+▟ ·▐▌+▖▌▚·▞▐· ·▟+++▐·│
│▌+▗ ·· ·+▛▟▐··▞▐+█· ·▞▙▜ ▌▐····+▐▌ ▌▌▌▌++·▗▌▐ ▐▗▜ ·▌▌▐▌▌▐ ▌▝▖ ▗▌█·▗▌▞ │
│▌+█ ·▟ ··▌█▐ ▗▘▐·█·▟ ▌█▐ ▌▐ ▗▌· ▐▌ ▌▙▘▐▗+·▐▌▐ ▐▞▐ ▌▌▐▚▌▐▐ ▌ ▐▚█·▐▌▌ │
│▌·█▗▌▛▖ ▐-█▐ ▐-▐▗▘▌▛▄▌█▐ ▌▐ ▌▐-·▌▌ ▗▘█ ▐▛▖▟▐▌▐ ▝▌▐ ▌▌▐▐▌▝▟ ▌ ▐▐▌▌▐▐▌ │
│▌·█▐▚▌▚▟▌ ▝▐ ▌ ▐▞ █ █▝▖▌-█ -▀▖▌▐ ▌ ▝- ▘▜▐▐▌▌ - -▌ ▌▌▐-▘ █ ▌ ▐-▘▌▐▝▌-│
0.684┤▌▗▜▞▝▌ -▘ ▝▖▌ ▘ ▜ █-▝▌-█ ▝▘▐▗ ▐-- -- ▐▌▌▌-- -▌▞-▙▘- -█ -▌ ▐ -▙▘ │
│▌▐▐▌ - -▝▌ ▜ -█ ▐█ ▞-- -- ▐▌▌▌-- -█--▝ ---▜--▌▞▟ ▝ │
│▐▐▝▌ -- - ▜ ▐▛▖▌-- - ▐▌█ - -█--- -----▜ │
│▐▐- ▘█ - ▘█ ▝ - - - │
0.668┤▐▌ ▝ ▜ │
│▐▌ │
│▐▌ │
│▝▌ │
0.653┤ - │
└┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬┘
1.0 23.5 46.0 68.5 91.0
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dto_summary.txt
train_dto/mean hist train_dto/max hist
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
20.0┤ ████ │25.0┤ ███ │
16.7┤ ████ ████ │20.8┤ ███ │
│ ████ ███████ │ │ ███ │
13.3┤ ██████████████ │16.7┤ ███ │
10.0┤ █████████████████ │12.5┤ ███ ████ │
│ █████████████████ │ │ ███ ████ ████ ████ │
6.7┤ █████████████████████ │ 8.3┤ ████████████████████████ │
3.3┤ █████████████████████ │ 4.2┤ ███████████████████████████│
│ ████████████████████████│ │ ███████████████████████████████│
0.0┤███ ███████████████████████████│ 0.0┤██████████████████████████████████│
└┬───────┬────────┬───────┬───────┬┘ └┬───────┬────────┬───────┬───────┬┘
0.675 0.687 0.698 0.710 0.722 0.682 0.699 0.716 0.733 0.750
train_dto/min hist train_dto/std hist
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
28.0┤ ███ │20.0┤ ████ │
│ ███ │ │ ████ │
23.3┤ ███ │16.7┤ ████ │
18.7┤ ███████ │13.3┤ ████ │
│ ███████ │ │ ████ ████ │
14.0┤ ████ ███████ │10.0┤███████ ████ ███████│
│ ██████████████████ │ │██████████ █████████████████│
9.3┤ ██████████████████ │ 6.7┤██████████ █████████████████│
4.7┤ ██████████████████ │ 3.3┤██████████████████████████████████│
│ ████████████████████████│ │██████████████████████████████████│
0.0┤███ ████████████████████████│ 0.0┤██████████████████████████████████│
└┬───────┬────────┬───────┬───────┬┘ └┬───────┬────────┬───────┬────────┘
0.650 0.663 0.676 0.689 0.702 -0.0000 0.0059 0.0119 0.0178
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_dto_hist.txt
train_loss train_loss/min
┌─────────────────────────────────┐ ┌─────────────────────────────────┐
7.115┤▙▌ │6.963┤ -- - --- │
7.078┤█▌ │6.920┤-- --------------------- ---- - -│
│█▌▙ ▖ │6.876┤ - -- -- ---- -------- │
7.042┤▐▌▛▄ ▐▚ ▗▛▌ ▗▗ ▗▟ │6.833┤ - - │
7.006┤▐▝▘▐▞▌ ▌▐▗█▌▌▗▄██▙ ▌▖█▌▌ ▟▗▖▗ ▗│ └┬───────┬───────┬───────┬───────┬┘
6.969┤▝ ▐▌▝▙▌▐▀▛▌▌▟█▌▜▝▖▙▌█▌▙▙▐▐▌▌▙▐▗▌│ 1.0 23.5 46.0 68.5 91.0
│ ▝▌ ▝ ▐ ▌▜██▌▝ ▙█▜▝▌▌█▐▌▘ ██▌ │train_loss/min iter
6.933┤ ▐ ▜▜ ▝█ ▘▝▞▘ ▜▝▌ │ train_loss/std
6.897┤ ▝ ▜ │ ┌────────────────────────────────┐
└┬───────┬───────┬───────┬───────┬┘0.0517┤* * │
1.0 23.5 46.0 68.5 91.0 0.0465┤**** * * ** │
train_loss iter 0.0362┤ ****************** * * ** * * │
train_loss/mean 0.0310┤ * **** * **************** │
┌─────────────────────────────────┐0.0207┤ * * * * *│
7.039┤··· │ └┬───────┬───────┬──────┬───────┬┘
7.023┤··· │ 1.0 23.5 46.0 68.5 91.0
│··· · │train_loss/std iter
7.007┤·· ··· · │ train_loss/max
6.991┤ ······· · │ ┌─────────────────────────────────┐
│ ········ ···· │7.148┤+ │
6.975┤ ·· ·········· · · · │7.124┤++++++ │
6.959┤ · ·· ··· ··········│7.074┤++ ++++++++++ + + + │
│ · ·· ···│7.050┤ + + ++++++++++++++++++++ ++ │
6.942┤ · │7.000┤ + +++ + ++++ +│
└┬───────┬───────┬───────┬───────┬┘ └┬───────┬───────┬───────┬───────┬┘
1.0 23.5 46.0 68.5 91.0 1.0 23.5 46.0 68.5 91.0
train_loss/mean iter train_loss/max iter
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_loss.txt
┌─────────────────────────────────────────────────────────────────────────┐
7.148┤ ++ train_loss/max │
│ -- train_loss/min │
│ ·· train_loss/mean │
│ ▞▞ train_loss │
7.096┤▌ █ + +++ │
│▌+█ + + ++ + │
│▚ █ ▖ ++ + + + + + ++ + │
│▐ █ ▐▚ ++ ++ ++ + + + + ++ +++ ++ ++ + │
7.043┤▐▗▜ ▐▐ + +▗ + + +++ + +++++ ++ + +++ + + ++ │
│▝▟▐·▌·▌▗ ▛▖ +▗▌▗▜ ++ + + ++++++ ++ + +▖++ ++++ + ++ │
│·█▐ ▌ ▝▜ ▐ ▚ ▐▌▐▐ + ▖▟ ▗++ ++ +▗+▐▚++ +++++++ + +++++ + │
│ █·▀▘ ·▐ ▞▄▟ ▐ ▐ ▗▌▐▌▐▐ ▐▌▛▖█▗▌ ▟+ █ ▐▝▖++ + ▖ + + + │
6.991┤ █ ▐·▌ ▐ ···▞·▐ ·▌▌▐▌▐▝▖· ▞▜▐▌▌█▐▐▌▗ █ ▖▗▘▌▐ ▌ + ▐▚ ▗▄ ▗ ▞│
│ ▜ ▌▌ ▚▟·▗▌·▐·▞ ▙▘▐▌·▌· ▌▐▐▚▌▝▐▞▚▀▖ █ ▐▌▐·▐▌·▌▗▌ ▖ ▖ ▗▐ ▚▌▐ █ ▗▘│
│ ▚▌ ▝▖▛▌ ▐▐ ▝·▐▌·▌ ▖▌▐▞▐▌ ▐▌··▚·█·▌▚▞ ▐▌·▌▐▌▐▌▐▌·▛▟ ·▘▐▗▌ █▗▚▞ │
│ -▐▌ ▜ -█ ▐▌ ▚▀▌▌▐▌▐▌ ▝▌ ▐ █·▌▐▌ ▐▌ ▌▌▐▞▌▐▚▐ ▜···▝▟▝▟▝▟···│
│ ---- ▘- --█ -- ▝▌ ▚▌▐▌▝▌ ▐▐ ▌▌▝▌ ▘ ▙▘▝▌▌▐▐▌ █ ▜ █· │
6.938┤ -- -- -- -- ---█--- - - ▐▌▐▌ -- ▝▌ ▙▘ -▜- ▌▐ ▘ - █ █ │
│--- --- -------█ -- -- ▝▌-▘ - -- -█ - - - ▚▘ -- ▜ ▝ -│
│ -- ----- ---▝ - - --- -- - --█--- -- -- --- --- -- - │
│ - - --- -- -- - -- ▜ - -- - -- -- - -- -- - - │
6.885┤ -- -- -- -- -- -- --- -- │
│ -- -- -- - - -- - - │
│ -- - - -- │
│ -- - │
6.833┤ - │
└┬─────────────────┬─────────────────┬─────────────────┬─────────────────┬┘
1.0 23.5 46.0 68.5 91.0
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_loss_summary.txt
train_loss/mean hist train_loss/max hist
┌──────────────────────────────────┐ ┌────────────────────────────────────┐
22.0┤ ███████ │24┤ ███ │
18.3┤ ███████ │20┤ ███ │
│ ███████ │ │ ███ │
14.7┤ ███████ │16┤ ███████████ │
11.0┤ ██████████████ │12┤ ██████████████ │
│ ██████████████ │ │ ██████████████ │
7.3┤ █████████████████████ │ 8┤██████████████████ │
3.7┤ █████████████████████ ████│ 4┤██████████████████████ ████ │
│ ████████████████████████ ████│ │████████████████████████████████ │
0.0┤██████████████████████████████████│ 0┤████████████████████████████████████│
└┬───────┬────────┬───────┬───────┬┘ └┬────────┬────────┬───────┬────────┬┘
6.938 6.965 6.991 7.017 7.044 6.994 7.034 7.074 7.115 7.155
train_loss/min hist train_loss/std hist
┌────────────────────────────────────┐ ┌──────────────────────────────────┐
24┤ ████ │20.0┤ ████ │
│ ████ │ │ ███ ████ │
20┤ ████████ │16.7┤ ███ ████ │
16┤ ████████ │13.3┤ ███ ████ │
│ ███████████ │ │ ██████████ ████ │
12┤ ███████████████ │10.0┤ █████████████████████ │
│ ███████████████ │ │ █████████████████████ │
8┤ ██████████████████ │ 6.7┤████████████████████████ │
4┤ █████████████████████████ │ 3.3┤███████████████████████████ │
│████ █████████████████████████████│ │███████████████████████████ ████│
0┤████████████████████████████████████│ 0.0┤███████████████████████████ ████│
└┬────────┬────────┬───────┬────────┬┘ └┬───────┬────────┬───────┬────────┘
6.827 6.863 6.898 6.934 6.969 0.0193 0.0277 0.0362 0.0446
text saved in /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/plots/tplot/train_loss_hist.txt
[2025-12-31 12:16:47,448186][W][ezpz/history:2320:save_dataset] Unable to save dataset to W&B, skipping!
[2025-12-31 12:16:47,449805][I][utils/__init__:651:dataset_to_h5pyfile] Saving dataset to: /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/train_dataset.h5
[2025-12-31 12:16:47,467124][I][ezpz/history:2433:finalize] Saving history report to /lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/outputs/2025-12-31-121308/2025-12-31-121642/report.md
[2025-12-31 12:16:47,473135][I][examples/vit:463:train_fn] dataset=<xarray.Dataset> Size: 26kB
Dimensions: (draw: 91)
Coordinates:
* draw (draw) int64 728B 0 1 2 3 4 5 6 7 ... 84 85 86 87 88 89 90
Data variables: (12/35)
train_iter (draw) int64 728B 10 11 12 13 14 15 ... 95 96 97 98 99 100
train_loss (draw) float32 364B 7.112 7.033 6.982 ... 6.968 6.978 7.0
train_dt (draw) float64 728B 0.7444 0.7021 0.7178 ... 0.7456 0.7609
train_dtd (draw) float64 728B 0.003141 0.001719 ... 0.002238 0.001625
train_dtf (draw) float64 728B 0.02159 0.02241 ... 0.02394 0.02694
train_dto (draw) float64 728B 0.699 0.6572 0.672 ... 0.6986 0.7115
... ...
train_dto_min (draw) float64 728B 0.6972 0.6526 0.67 ... 0.6872 0.6867
train_dto_std (draw) float64 728B 0.0009612 0.02281 ... 0.004004 0.01135
train_dtb_mean (draw) float64 728B 0.02078 0.02071 ... 0.02083 0.02072
train_dtb_max (draw) float64 728B 0.02156 0.02113 ... 0.02186 0.02113
train_dtb_min (draw) float64 728B 0.02027 0.02019 ... 0.02044 0.02029
train_dtb_std (draw) float64 728B 0.0003202 0.0002283 ... 0.0002136
[2025-12-31 12:16:47,618825][I][examples/vit:544:<module>] Took 218.91 seconds
wandb:
wandb: 🚀 View run snowy-hill-239 at:
wandb: Find logs at: ../../../../../../lus/tegu/projects/datascience/foremans/projects/saforem2/ezpz/wandb/run-20251231_121309-g19jy6bl/logs
[2025-12-31 12:16:49,364101][I][ezpz/launch:447:launch] ----[🍋 ezpz.launch][stop][2025-12-31-121649]----
[2025-12-31 12:16:49,364806][I][ezpz/launch:448:launch] Execution finished with 0.
[2025-12-31 12:16:49,365202][I][ezpz/launch:449:launch] Executing finished in 227.18 seconds.
[2025-12-31 12:16:49,365551][I][ezpz/launch:450:launch] Took 227.18 seconds to run. Exiting.