2023-12-07 00:28:01.875 INFO: Process group initialized: True 2023-12-07 00:28:01.877 INFO: Processes: 80 2023-12-07 00:28:01.877 INFO: MACE version: 0.3.0 2023-12-07 00:28:01.877 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=32, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-07 00:28:01.878 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-07 00:28:01.878 INFO: Using statistics json file 2023-12-07 00:28:01.878 INFO: Using atomic numbers from statistics file 2023-12-07 00:28:01.878 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-07 00:28:01.878 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-07 00:28:01.879 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-07 00:28:34.538 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-07 00:28:34.541 INFO: Average number of neighbors: 61.964672446250916 2023-12-07 00:28:34.541 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-07 00:28:34.541 INFO: Building model 2023-12-07 00:28:34.542 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-07 00:28:39.317 WARNING: Cannot find checkpoint with tag '04-128-L2_run-1' in 'checkpoints' 2023-12-07 00:28:39.327 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-07 00:28:39.334 INFO: Number of parameters: 5725072 2023-12-07 00:28:39.334 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-07 00:28:39.334 INFO: Using Weights and Biases for logging 2023-12-07 00:28:59.853 INFO: Using gradient clipping with tolerance=100.000 2023-12-07 00:28:59.853 INFO: Started training 2023-12-07 00:29:08.004 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.004 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.004 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.004 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.005 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 00:29:08.006 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:20:37.598 INFO: Process group initialized: True 2023-12-07 20:20:37.600 INFO: Processes: 80 2023-12-07 20:20:37.600 INFO: MACE version: 0.3.0 2023-12-07 20:20:37.600 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=32, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-07 20:20:37.600 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-07 20:20:37.601 INFO: Using statistics json file 2023-12-07 20:20:37.601 INFO: Using atomic numbers from statistics file 2023-12-07 20:20:37.601 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-07 20:20:37.601 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-07 20:20:37.602 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-07 20:21:18.046 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-07 20:21:18.049 INFO: Average number of neighbors: 61.964672446250916 2023-12-07 20:21:18.049 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-07 20:21:18.049 INFO: Building model 2023-12-07 20:21:18.051 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-07 20:21:23.614 WARNING: Cannot find checkpoint with tag '04-128-L2_run-1' in 'checkpoints' 2023-12-07 20:21:23.619 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-07 20:21:23.626 INFO: Number of parameters: 5725072 2023-12-07 20:21:23.626 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-07 20:21:23.626 INFO: Using Weights and Biases for logging 2023-12-07 20:21:43.302 INFO: Using gradient clipping with tolerance=100.000 2023-12-07 20:21:43.303 INFO: Started training 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.735 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.737 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-07 20:21:53.738 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:54:30.846 INFO: Process group initialized: True 2023-12-11 01:54:30.848 INFO: Processes: 80 2023-12-11 01:54:30.848 INFO: MACE version: 0.3.0 2023-12-11 01:54:30.848 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=10, valid_batch_size=32, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-11 01:54:30.848 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-11 01:54:30.848 INFO: Using statistics json file 2023-12-11 01:54:30.848 INFO: Using atomic numbers from statistics file 2023-12-11 01:54:30.849 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-11 01:54:30.849 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-11 01:54:30.849 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-11 01:55:02.465 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-11 01:55:02.468 INFO: Average number of neighbors: 61.964672446250916 2023-12-11 01:55:02.468 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-11 01:55:02.468 INFO: Building model 2023-12-11 01:55:02.469 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-11 01:55:06.925 WARNING: Cannot find checkpoint with tag '04-128-L2_run-1' in 'checkpoints' 2023-12-11 01:55:06.929 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-11 01:55:06.936 INFO: Number of parameters: 5725072 2023-12-11 01:55:06.936 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-11 01:55:06.936 INFO: Using Weights and Biases for logging 2023-12-11 01:55:23.649 INFO: Using gradient clipping with tolerance=100.000 2023-12-11 01:55:23.650 INFO: Started training 2023-12-11 01:55:44.350 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.350 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.350 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.350 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.353 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.353 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.353 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.353 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.353 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.353 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.355 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.353 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.353 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 01:55:44.354 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:05:16.530 INFO: Process group initialized: True 2023-12-11 02:05:16.532 INFO: Processes: 80 2023-12-11 02:05:16.532 INFO: MACE version: 0.3.0 2023-12-11 02:05:16.532 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=10, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-11 02:05:16.533 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-11 02:05:16.533 INFO: Using statistics json file 2023-12-11 02:05:16.533 INFO: Using atomic numbers from statistics file 2023-12-11 02:05:16.533 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-11 02:05:16.533 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-11 02:05:16.534 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-11 02:05:48.156 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-11 02:05:48.159 INFO: Average number of neighbors: 61.964672446250916 2023-12-11 02:05:48.159 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-11 02:05:48.159 INFO: Building model 2023-12-11 02:05:48.161 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-11 02:05:52.069 WARNING: Cannot find checkpoint with tag '04-128-L2_run-1' in 'checkpoints' 2023-12-11 02:05:52.071 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-11 02:05:52.077 INFO: Number of parameters: 5725072 2023-12-11 02:05:52.078 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-11 02:05:52.078 INFO: Using Weights and Biases for logging 2023-12-11 02:06:06.487 INFO: Using gradient clipping with tolerance=100.000 2023-12-11 02:06:06.487 INFO: Started training 2023-12-11 02:06:26.538 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.538 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.538 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.538 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.539 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.539 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.541 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.539 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.541 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.543 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.539 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.543 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.543 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.542 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.543 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:06:26.543 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:15:34.062 INFO: Process group initialized: True 2023-12-11 02:15:34.064 INFO: Processes: 80 2023-12-11 02:15:34.064 INFO: MACE version: 0.3.0 2023-12-11 02:15:34.064 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=10, valid_batch_size=10, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-11 02:15:34.065 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-11 02:15:34.065 INFO: Using statistics json file 2023-12-11 02:15:34.065 INFO: Using atomic numbers from statistics file 2023-12-11 02:15:34.065 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-11 02:15:34.065 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-11 02:15:34.066 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-11 02:16:04.873 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-11 02:16:04.875 INFO: Average number of neighbors: 61.964672446250916 2023-12-11 02:16:04.875 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-11 02:16:04.876 INFO: Building model 2023-12-11 02:16:04.877 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-11 02:16:09.354 WARNING: Cannot find checkpoint with tag '04-128-L2_run-1' in 'checkpoints' 2023-12-11 02:16:09.358 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-11 02:16:09.365 INFO: Number of parameters: 5725072 2023-12-11 02:16:09.365 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-11 02:16:09.365 INFO: Using Weights and Biases for logging 2023-12-11 02:16:25.300 INFO: Using gradient clipping with tolerance=100.000 2023-12-11 02:16:25.300 INFO: Started training 2023-12-11 02:16:45.451 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.451 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.451 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.451 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.453 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.454 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 02:16:45.455 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:19:19.122 INFO: Process group initialized: True 2023-12-11 10:19:19.123 INFO: Processes: 80 2023-12-11 10:19:19.124 INFO: MACE version: 0.3.0 2023-12-11 10:19:19.124 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=8, valid_batch_size=8, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-11 10:19:19.124 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-11 10:19:19.125 INFO: Using statistics json file 2023-12-11 10:19:19.125 INFO: Using atomic numbers from statistics file 2023-12-11 10:19:19.126 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-11 10:19:19.126 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-11 10:19:19.126 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-11 10:19:51.417 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-11 10:19:51.420 INFO: Average number of neighbors: 61.964672446250916 2023-12-11 10:19:51.420 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-11 10:19:51.421 INFO: Building model 2023-12-11 10:19:51.422 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-11 10:19:55.945 WARNING: Cannot find checkpoint with tag '04-128-L2_run-1' in 'checkpoints' 2023-12-11 10:19:55.947 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-11 10:19:55.954 INFO: Number of parameters: 5725072 2023-12-11 10:19:55.954 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-11 10:19:55.954 INFO: Using Weights and Biases for logging 2023-12-11 10:20:08.206 INFO: Using gradient clipping with tolerance=100.000 2023-12-11 10:20:08.206 INFO: Started training 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.815 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.816 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 10:20:27.818 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-11 11:36:09.221 INFO: Epoch 0: loss=1.3756e-02, MAE_E_per_atom=145.6508 meV, MAE_F=110.4323 meV / A, MAE_stress_per_atom=0.3273 meV / A^3 2023-12-11 12:52:28.136 INFO: Epoch 1: loss=1.1419e-02, MAE_E_per_atom=90.7337 meV, MAE_F=96.6863 meV / A, MAE_stress_per_atom=0.2889 meV / A^3 2023-12-11 14:08:16.793 INFO: Epoch 2: loss=1.0566e-02, MAE_E_per_atom=71.8710 meV, MAE_F=89.4192 meV / A, MAE_stress_per_atom=0.2574 meV / A^3 2023-12-11 15:24:28.836 INFO: Epoch 3: loss=9.6194e-03, MAE_E_per_atom=61.5787 meV, MAE_F=84.4657 meV / A, MAE_stress_per_atom=0.1953 meV / A^3 2023-12-11 16:41:12.034 INFO: Epoch 4: loss=8.7870e-03, MAE_E_per_atom=53.5175 meV, MAE_F=78.0509 meV / A, MAE_stress_per_atom=0.1639 meV / A^3 2023-12-11 17:56:59.396 INFO: Epoch 5: loss=8.3183e-03, MAE_E_per_atom=49.0774 meV, MAE_F=74.1386 meV / A, MAE_stress_per_atom=0.1666 meV / A^3 2023-12-11 19:12:50.988 INFO: Epoch 6: loss=7.9122e-03, MAE_E_per_atom=45.5705 meV, MAE_F=71.1240 meV / A, MAE_stress_per_atom=0.1388 meV / A^3 2023-12-11 20:29:04.912 INFO: Epoch 7: loss=7.4001e-03, MAE_E_per_atom=42.3989 meV, MAE_F=67.1669 meV / A, MAE_stress_per_atom=0.1328 meV / A^3 2023-12-11 21:46:03.315 INFO: Epoch 8: loss=7.1911e-03, MAE_E_per_atom=39.9449 meV, MAE_F=64.2585 meV / A, MAE_stress_per_atom=0.1291 meV / A^3 2023-12-11 23:02:33.753 INFO: Epoch 9: loss=7.0474e-03, MAE_E_per_atom=38.9909 meV, MAE_F=62.7240 meV / A, MAE_stress_per_atom=0.1165 meV / A^3 2023-12-12 00:19:28.470 INFO: Epoch 10: loss=7.2313e-03, MAE_E_per_atom=39.9044 meV, MAE_F=65.9097 meV / A, MAE_stress_per_atom=0.1666 meV / A^3 2023-12-12 01:36:18.066 INFO: Epoch 11: loss=6.9742e-03, MAE_E_per_atom=36.9225 meV, MAE_F=62.4240 meV / A, MAE_stress_per_atom=0.1293 meV / A^3 2023-12-12 02:52:58.540 INFO: Epoch 12: loss=7.1774e-03, MAE_E_per_atom=38.0465 meV, MAE_F=63.7833 meV / A, MAE_stress_per_atom=0.1607 meV / A^3 2023-12-12 04:10:11.611 INFO: Epoch 13: loss=6.7827e-03, MAE_E_per_atom=35.3466 meV, MAE_F=60.4803 meV / A, MAE_stress_per_atom=0.1273 meV / A^3 2023-12-12 05:27:31.816 INFO: Epoch 14: loss=6.6998e-03, MAE_E_per_atom=34.0383 meV, MAE_F=59.3242 meV / A, MAE_stress_per_atom=0.1346 meV / A^3 2023-12-12 06:45:18.445 INFO: Epoch 15: loss=6.6691e-03, MAE_E_per_atom=33.1291 meV, MAE_F=58.4530 meV / A, MAE_stress_per_atom=0.1344 meV / A^3 2023-12-12 08:02:23.326 INFO: Epoch 16: loss=6.6172e-03, MAE_E_per_atom=32.6460 meV, MAE_F=57.4790 meV / A, MAE_stress_per_atom=0.1350 meV / A^3 2023-12-12 09:19:26.753 INFO: Epoch 17: loss=6.4730e-03, MAE_E_per_atom=31.8744 meV, MAE_F=57.7743 meV / A, MAE_stress_per_atom=0.1249 meV / A^3 2023-12-12 10:37:18.279 INFO: Epoch 18: loss=6.3338e-03, MAE_E_per_atom=31.1102 meV, MAE_F=56.3578 meV / A, MAE_stress_per_atom=0.1310 meV / A^3 2023-12-12 11:54:43.081 INFO: Epoch 19: loss=6.4129e-03, MAE_E_per_atom=31.0728 meV, MAE_F=56.2917 meV / A, MAE_stress_per_atom=0.1353 meV / A^3 2023-12-16 00:00:29.769 INFO: Process group initialized: True 2023-12-16 00:00:29.771 INFO: Processes: 80 2023-12-16 00:00:29.771 INFO: MACE version: 0.3.0 2023-12-16 00:00:29.771 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=8, valid_batch_size=8, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-16 00:00:29.771 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-16 00:00:29.772 INFO: Using statistics json file 2023-12-16 00:00:29.772 INFO: Using atomic numbers from statistics file 2023-12-16 00:00:29.772 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-16 00:00:29.772 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-16 00:00:29.773 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-16 00:01:03.221 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-16 00:01:03.223 INFO: Average number of neighbors: 61.964672446250916 2023-12-16 00:01:03.223 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-16 00:01:03.223 INFO: Building model 2023-12-16 00:01:03.225 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-16 00:01:07.933 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-16 00:01:07.934 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-19.pt 2023-12-16 00:01:08.204 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-16 00:01:08.210 INFO: Number of parameters: 5725072 2023-12-16 00:01:08.210 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-16 00:01:08.210 INFO: Using Weights and Biases for logging 2023-12-16 00:01:22.250 INFO: Using gradient clipping with tolerance=100.000 2023-12-16 00:01:22.250 INFO: Started training 2023-12-16 00:01:29.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 00:01:29.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 01:17:04.133 INFO: Epoch 19: loss=6.4981e-03, MAE_E_per_atom=30.4521 meV, MAE_F=56.8582 meV / A, MAE_stress_per_atom=0.1375 meV / A^3 2023-12-16 11:03:59.276 INFO: Process group initialized: True 2023-12-16 11:03:59.278 INFO: Processes: 80 2023-12-16 11:03:59.278 INFO: MACE version: 0.3.0 2023-12-16 11:03:59.278 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=8, valid_batch_size=8, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-16 11:03:59.278 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-16 11:03:59.279 INFO: Using statistics json file 2023-12-16 11:03:59.279 INFO: Using atomic numbers from statistics file 2023-12-16 11:03:59.279 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-16 11:03:59.279 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-16 11:03:59.280 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-16 11:04:31.862 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-16 11:04:31.864 INFO: Average number of neighbors: 61.964672446250916 2023-12-16 11:04:31.864 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-16 11:04:31.864 INFO: Building model 2023-12-16 11:04:31.866 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-16 11:04:36.555 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-16 11:04:36.556 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-19.pt 2023-12-16 11:04:36.817 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-16 11:04:36.823 INFO: Number of parameters: 5725072 2023-12-16 11:04:36.824 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-16 11:04:36.824 INFO: Using Weights and Biases for logging 2023-12-16 11:05:12.862 INFO: Using gradient clipping with tolerance=100.000 2023-12-16 11:05:12.863 INFO: Started training 2023-12-16 11:05:20.477 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.477 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.477 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.477 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.478 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.478 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.480 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.480 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.480 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.480 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.479 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.480 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.480 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:05:20.481 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:10:50.432 INFO: Process group initialized: True 2023-12-16 11:10:50.434 INFO: Processes: 80 2023-12-16 11:10:50.434 INFO: MACE version: 0.3.0 2023-12-16 11:10:50.435 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=8, valid_batch_size=8, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-16 11:10:50.435 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-16 11:10:50.435 INFO: Using statistics json file 2023-12-16 11:10:50.435 INFO: Using atomic numbers from statistics file 2023-12-16 11:10:50.435 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-16 11:10:50.436 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-16 11:10:50.436 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-16 11:11:21.939 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-16 11:11:21.941 INFO: Average number of neighbors: 61.964672446250916 2023-12-16 11:11:21.941 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-16 11:11:21.941 INFO: Building model 2023-12-16 11:11:21.943 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-16 11:11:26.589 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-16 11:11:26.590 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-19.pt 2023-12-16 11:11:26.857 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-16 11:11:26.863 INFO: Number of parameters: 5725072 2023-12-16 11:11:26.863 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-16 11:11:26.863 INFO: Using Weights and Biases for logging 2023-12-16 11:11:40.939 INFO: Using gradient clipping with tolerance=100.000 2023-12-16 11:11:40.939 INFO: Started training 2023-12-16 11:11:47.811 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.811 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.811 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.811 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.811 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.812 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.811 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.812 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.811 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.812 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.811 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.813 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.812 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.819 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 11:11:47.820 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 12:27:15.521 INFO: Epoch 19: loss=6.4620e-03, MAE_E_per_atom=29.4196 meV, MAE_F=56.0879 meV / A, MAE_stress_per_atom=0.1371 meV / A^3 2023-12-16 14:10:34.979 INFO: Process group initialized: True 2023-12-16 14:10:34.982 INFO: Processes: 120 2023-12-16 14:10:34.982 INFO: MACE version: 0.3.0 2023-12-16 14:10:34.982 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=5, valid_batch_size=5, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-16 14:10:34.982 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-16 14:10:34.982 INFO: Using statistics json file 2023-12-16 14:10:34.982 INFO: Using atomic numbers from statistics file 2023-12-16 14:10:34.982 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-16 14:10:34.983 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-16 14:10:34.983 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-16 14:11:08.310 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-16 14:11:08.313 INFO: Average number of neighbors: 61.964672446250916 2023-12-16 14:11:08.313 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-16 14:11:08.313 INFO: Building model 2023-12-16 14:11:08.314 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-16 14:11:13.051 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-16 14:11:13.052 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-19.pt 2023-12-16 14:11:13.314 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-16 14:11:13.321 INFO: Number of parameters: 5725072 2023-12-16 14:11:13.321 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.005 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-16 14:11:13.321 INFO: Using Weights and Biases for logging 2023-12-16 14:11:25.530 INFO: Using gradient clipping with tolerance=100.000 2023-12-16 14:11:25.530 INFO: Started training 2023-12-16 14:11:32.302 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.302 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.302 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.302 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.302 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.302 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.302 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.302 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.305 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 14:11:32.306 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-16 15:00:31.281 INFO: Epoch 19: loss=5.9170e-03, MAE_E_per_atom=35.3538 meV, MAE_F=52.1014 meV / A, MAE_stress_per_atom=0.1373 meV / A^3 2023-12-16 15:48:59.125 INFO: Epoch 20: loss=5.8602e-03, MAE_E_per_atom=33.9086 meV, MAE_F=51.8807 meV / A, MAE_stress_per_atom=0.1370 meV / A^3 2023-12-16 16:37:55.088 INFO: Epoch 21: loss=5.7387e-03, MAE_E_per_atom=33.9504 meV, MAE_F=51.2229 meV / A, MAE_stress_per_atom=0.1340 meV / A^3 2023-12-16 17:27:17.352 INFO: Epoch 22: loss=5.7838e-03, MAE_E_per_atom=33.0600 meV, MAE_F=52.1368 meV / A, MAE_stress_per_atom=0.1337 meV / A^3 2023-12-16 18:16:40.065 INFO: Epoch 23: loss=5.9000e-03, MAE_E_per_atom=33.0253 meV, MAE_F=51.9886 meV / A, MAE_stress_per_atom=0.1456 meV / A^3 2023-12-16 19:05:38.002 INFO: Epoch 24: loss=5.9388e-03, MAE_E_per_atom=34.1335 meV, MAE_F=53.1867 meV / A, MAE_stress_per_atom=0.1366 meV / A^3 2023-12-16 19:54:40.044 INFO: Epoch 25: loss=6.1181e-03, MAE_E_per_atom=34.8045 meV, MAE_F=55.1629 meV / A, MAE_stress_per_atom=0.1398 meV / A^3 2023-12-16 20:44:21.058 INFO: Epoch 26: loss=5.7510e-03, MAE_E_per_atom=31.3285 meV, MAE_F=50.8171 meV / A, MAE_stress_per_atom=0.1407 meV / A^3 2023-12-16 21:33:38.064 INFO: Epoch 27: loss=5.6152e-03, MAE_E_per_atom=30.4272 meV, MAE_F=50.4372 meV / A, MAE_stress_per_atom=0.1380 meV / A^3 2023-12-16 22:23:03.108 INFO: Epoch 28: loss=5.5144e-03, MAE_E_per_atom=30.0585 meV, MAE_F=49.7740 meV / A, MAE_stress_per_atom=0.1374 meV / A^3 2023-12-16 23:12:43.791 INFO: Epoch 29: loss=5.6003e-03, MAE_E_per_atom=29.3114 meV, MAE_F=50.1433 meV / A, MAE_stress_per_atom=0.1331 meV / A^3 2023-12-17 00:02:37.052 INFO: Epoch 30: loss=5.5268e-03, MAE_E_per_atom=29.6318 meV, MAE_F=49.5836 meV / A, MAE_stress_per_atom=0.1303 meV / A^3 2023-12-17 00:52:17.873 INFO: Epoch 31: loss=5.4327e-03, MAE_E_per_atom=29.5743 meV, MAE_F=48.9663 meV / A, MAE_stress_per_atom=0.1324 meV / A^3 2023-12-17 01:41:48.461 INFO: Epoch 32: loss=5.4362e-03, MAE_E_per_atom=28.5837 meV, MAE_F=49.2340 meV / A, MAE_stress_per_atom=0.1322 meV / A^3 2023-12-17 02:31:57.789 INFO: Epoch 33: loss=5.3778e-03, MAE_E_per_atom=28.3397 meV, MAE_F=48.8699 meV / A, MAE_stress_per_atom=0.1334 meV / A^3 2023-12-17 03:21:49.543 INFO: Epoch 34: loss=5.3589e-03, MAE_E_per_atom=28.0965 meV, MAE_F=48.5608 meV / A, MAE_stress_per_atom=0.1344 meV / A^3 2023-12-17 04:11:54.311 INFO: Epoch 35: loss=5.2980e-03, MAE_E_per_atom=28.0659 meV, MAE_F=48.1756 meV / A, MAE_stress_per_atom=0.1280 meV / A^3 2023-12-17 05:01:51.289 INFO: Epoch 36: loss=5.4585e-03, MAE_E_per_atom=27.9147 meV, MAE_F=49.0780 meV / A, MAE_stress_per_atom=0.1363 meV / A^3 2023-12-17 05:52:02.381 INFO: Epoch 37: loss=5.3918e-03, MAE_E_per_atom=27.7858 meV, MAE_F=48.6060 meV / A, MAE_stress_per_atom=0.1299 meV / A^3 2023-12-17 06:42:02.964 INFO: Epoch 38: loss=5.2139e-03, MAE_E_per_atom=27.4375 meV, MAE_F=47.9888 meV / A, MAE_stress_per_atom=0.1253 meV / A^3 2023-12-17 07:32:42.624 INFO: Epoch 39: loss=5.2484e-03, MAE_E_per_atom=27.1803 meV, MAE_F=47.5623 meV / A, MAE_stress_per_atom=0.1283 meV / A^3 2023-12-17 08:23:35.662 INFO: Epoch 40: loss=5.3645e-03, MAE_E_per_atom=27.6956 meV, MAE_F=47.9319 meV / A, MAE_stress_per_atom=0.1351 meV / A^3 2023-12-17 09:13:58.909 INFO: Epoch 41: loss=5.3947e-03, MAE_E_per_atom=28.2244 meV, MAE_F=47.8837 meV / A, MAE_stress_per_atom=0.1289 meV / A^3 2023-12-17 10:05:00.052 INFO: Epoch 42: loss=5.1575e-03, MAE_E_per_atom=27.2498 meV, MAE_F=46.7787 meV / A, MAE_stress_per_atom=0.1308 meV / A^3 2023-12-17 10:55:42.399 INFO: Epoch 43: loss=5.0785e-03, MAE_E_per_atom=26.7668 meV, MAE_F=46.7187 meV / A, MAE_stress_per_atom=0.1307 meV / A^3 2023-12-17 11:46:24.810 INFO: Epoch 44: loss=5.3324e-03, MAE_E_per_atom=27.8181 meV, MAE_F=48.0397 meV / A, MAE_stress_per_atom=0.1338 meV / A^3 2023-12-17 12:37:08.677 INFO: Epoch 45: loss=5.2258e-03, MAE_E_per_atom=27.5819 meV, MAE_F=47.4771 meV / A, MAE_stress_per_atom=0.1235 meV / A^3 2023-12-17 13:28:22.571 INFO: Epoch 46: loss=6.7323e-03, MAE_E_per_atom=41.6359 meV, MAE_F=57.2895 meV / A, MAE_stress_per_atom=0.1381 meV / A^3 2023-12-17 14:19:23.058 INFO: Epoch 47: loss=5.2437e-03, MAE_E_per_atom=28.0197 meV, MAE_F=46.9941 meV / A, MAE_stress_per_atom=0.1239 meV / A^3 2023-12-17 15:10:37.741 INFO: Epoch 48: loss=5.3288e-03, MAE_E_per_atom=26.6245 meV, MAE_F=47.1015 meV / A, MAE_stress_per_atom=0.1310 meV / A^3 2023-12-17 16:02:09.388 INFO: Epoch 49: loss=5.1615e-03, MAE_E_per_atom=25.8724 meV, MAE_F=46.7588 meV / A, MAE_stress_per_atom=0.1262 meV / A^3 2023-12-17 16:53:52.762 INFO: Epoch 50: loss=5.1438e-03, MAE_E_per_atom=25.4286 meV, MAE_F=46.2308 meV / A, MAE_stress_per_atom=0.1298 meV / A^3 2023-12-17 17:46:02.744 INFO: Epoch 51: loss=5.1015e-03, MAE_E_per_atom=25.5938 meV, MAE_F=45.9716 meV / A, MAE_stress_per_atom=0.1241 meV / A^3 2023-12-17 18:37:59.946 INFO: Epoch 52: loss=5.0755e-03, MAE_E_per_atom=24.8875 meV, MAE_F=45.5054 meV / A, MAE_stress_per_atom=0.1304 meV / A^3 2023-12-17 19:30:10.490 INFO: Epoch 53: loss=4.9733e-03, MAE_E_per_atom=25.2217 meV, MAE_F=45.2865 meV / A, MAE_stress_per_atom=0.1247 meV / A^3 2023-12-17 20:22:30.286 INFO: Epoch 54: loss=4.9005e-03, MAE_E_per_atom=25.1175 meV, MAE_F=45.0213 meV / A, MAE_stress_per_atom=0.1286 meV / A^3 2023-12-17 21:14:44.003 INFO: Epoch 55: loss=5.0536e-03, MAE_E_per_atom=24.3411 meV, MAE_F=46.0948 meV / A, MAE_stress_per_atom=0.1269 meV / A^3 2023-12-17 22:07:04.025 INFO: Epoch 56: loss=4.8464e-03, MAE_E_per_atom=24.4122 meV, MAE_F=45.2751 meV / A, MAE_stress_per_atom=0.1222 meV / A^3 2023-12-22 22:23:40.087 INFO: Process group initialized: True 2023-12-22 22:23:40.090 INFO: Processes: 120 2023-12-22 22:23:40.090 INFO: MACE version: 0.3.0 2023-12-22 22:23:40.090 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-22 22:23:40.091 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-22 22:23:40.091 INFO: Using statistics json file 2023-12-22 22:23:40.091 INFO: Using atomic numbers from statistics file 2023-12-22 22:23:40.092 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-22 22:23:40.092 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-22 22:23:40.092 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-22 22:24:16.771 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-22 22:24:16.774 INFO: Average number of neighbors: 61.964672446250916 2023-12-22 22:24:16.774 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-22 22:24:16.774 INFO: Building model 2023-12-22 22:24:16.775 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-22 22:24:21.644 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-22 22:24:21.646 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-56.pt 2023-12-22 22:24:21.934 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-22 22:24:21.941 INFO: Number of parameters: 5725072 2023-12-22 22:24:21.941 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-22 22:24:21.941 INFO: Using Weights and Biases for logging 2023-12-22 22:24:43.310 INFO: Using gradient clipping with tolerance=100.000 2023-12-22 22:24:43.310 INFO: Started training 2023-12-22 22:24:51.993 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.993 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.993 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.993 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.996 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.997 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.998 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:24:51.999 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-22 22:48:19.403 INFO: Epoch 56: loss=4.8523e-03, MAE_E_per_atom=24.0743 meV, MAE_F=44.3763 meV / A, MAE_stress_per_atom=0.1243 meV / A^3 2023-12-22 23:01:28.966 INFO: Epoch 57: loss=4.8077e-03, MAE_E_per_atom=23.9407 meV, MAE_F=43.8058 meV / A, MAE_stress_per_atom=0.1283 meV / A^3 2023-12-22 23:14:12.193 INFO: Epoch 58: loss=4.7890e-03, MAE_E_per_atom=24.1124 meV, MAE_F=43.5796 meV / A, MAE_stress_per_atom=0.1291 meV / A^3 2023-12-22 23:26:50.723 INFO: Epoch 59: loss=4.8921e-03, MAE_E_per_atom=23.5603 meV, MAE_F=44.4242 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2023-12-22 23:39:33.258 INFO: Epoch 60: loss=4.8608e-03, MAE_E_per_atom=23.8065 meV, MAE_F=44.3536 meV / A, MAE_stress_per_atom=0.1314 meV / A^3 2023-12-22 23:52:17.185 INFO: Epoch 61: loss=4.8170e-03, MAE_E_per_atom=23.7824 meV, MAE_F=43.7658 meV / A, MAE_stress_per_atom=0.1331 meV / A^3 2023-12-23 00:04:58.621 INFO: Epoch 62: loss=4.7765e-03, MAE_E_per_atom=23.3216 meV, MAE_F=43.5908 meV / A, MAE_stress_per_atom=0.1287 meV / A^3 2023-12-23 00:17:39.125 INFO: Epoch 63: loss=4.7956e-03, MAE_E_per_atom=23.5173 meV, MAE_F=43.6626 meV / A, MAE_stress_per_atom=0.1284 meV / A^3 2023-12-24 10:59:28.449 INFO: Process group initialized: True 2023-12-24 10:59:28.451 INFO: Processes: 120 2023-12-24 10:59:28.451 INFO: MACE version: 0.3.0 2023-12-24 10:59:28.451 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-24 10:59:28.451 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-24 10:59:28.452 INFO: Using statistics json file 2023-12-24 10:59:28.453 INFO: Using atomic numbers from statistics file 2023-12-24 10:59:28.453 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-24 10:59:28.453 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-24 10:59:28.453 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-24 11:00:00.751 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-24 11:00:00.753 INFO: Average number of neighbors: 61.964672446250916 2023-12-24 11:00:00.753 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-24 11:00:00.753 INFO: Building model 2023-12-24 11:00:00.754 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-24 11:00:05.396 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-24 11:00:05.397 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-63.pt 2023-12-24 11:00:05.661 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-24 11:00:05.667 INFO: Number of parameters: 5725072 2023-12-24 11:00:05.667 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-24 11:00:05.667 INFO: Using Weights and Biases for logging 2023-12-24 11:00:18.917 INFO: Using gradient clipping with tolerance=100.000 2023-12-24 11:00:18.917 INFO: Started training 2023-12-24 11:00:27.075 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.075 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.075 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.075 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.079 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.079 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.079 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.079 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.082 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:00:27.083 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-24 11:22:58.280 INFO: Epoch 63: loss=4.8201e-03, MAE_E_per_atom=23.5275 meV, MAE_F=43.7423 meV / A, MAE_stress_per_atom=0.1289 meV / A^3 2023-12-24 11:36:23.056 INFO: Epoch 64: loss=4.7345e-03, MAE_E_per_atom=23.2347 meV, MAE_F=43.4619 meV / A, MAE_stress_per_atom=0.1283 meV / A^3 2023-12-24 11:49:07.136 INFO: Epoch 65: loss=4.7228e-03, MAE_E_per_atom=23.2040 meV, MAE_F=43.1610 meV / A, MAE_stress_per_atom=0.1290 meV / A^3 2023-12-24 12:01:48.791 INFO: Epoch 66: loss=4.7169e-03, MAE_E_per_atom=23.1760 meV, MAE_F=42.9919 meV / A, MAE_stress_per_atom=0.1271 meV / A^3 2023-12-24 12:14:28.727 INFO: Epoch 67: loss=4.7093e-03, MAE_E_per_atom=23.0925 meV, MAE_F=42.9817 meV / A, MAE_stress_per_atom=0.1285 meV / A^3 2023-12-24 12:27:11.728 INFO: Epoch 68: loss=4.7212e-03, MAE_E_per_atom=23.0214 meV, MAE_F=43.1179 meV / A, MAE_stress_per_atom=0.1271 meV / A^3 2023-12-24 12:39:52.063 INFO: Epoch 69: loss=4.7131e-03, MAE_E_per_atom=22.9911 meV, MAE_F=43.1127 meV / A, MAE_stress_per_atom=0.1290 meV / A^3 2023-12-24 12:52:35.684 INFO: Epoch 70: loss=4.6844e-03, MAE_E_per_atom=23.0984 meV, MAE_F=42.8812 meV / A, MAE_stress_per_atom=0.1290 meV / A^3 2023-12-25 15:21:34.796 INFO: Process group initialized: True 2023-12-25 15:21:34.798 INFO: Processes: 120 2023-12-25 15:21:34.798 INFO: MACE version: 0.3.0 2023-12-25 15:21:34.799 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-25 15:21:34.799 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-25 15:21:34.800 INFO: Using statistics json file 2023-12-25 15:21:34.800 INFO: Using atomic numbers from statistics file 2023-12-25 15:21:34.800 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-25 15:21:34.800 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-25 15:21:34.801 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-25 15:22:10.456 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-25 15:22:10.458 INFO: Average number of neighbors: 61.964672446250916 2023-12-25 15:22:10.458 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-25 15:22:10.459 INFO: Building model 2023-12-25 15:22:10.460 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-25 15:22:15.275 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-25 15:22:15.277 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-70.pt 2023-12-25 15:22:15.551 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-25 15:22:15.557 INFO: Number of parameters: 5725072 2023-12-25 15:22:15.557 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-25 15:22:15.557 INFO: Using Weights and Biases for logging 2023-12-25 15:22:29.096 INFO: Using gradient clipping with tolerance=100.000 2023-12-25 15:22:29.096 INFO: Started training 2023-12-25 15:22:37.918 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.918 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.918 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.918 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.918 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.918 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.918 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.918 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.923 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.924 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:22:37.925 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-25 15:45:12.327 INFO: Epoch 70: loss=4.6714e-03, MAE_E_per_atom=23.1183 meV, MAE_F=42.8242 meV / A, MAE_stress_per_atom=0.1291 meV / A^3 2023-12-25 15:58:29.002 INFO: Epoch 71: loss=4.6598e-03, MAE_E_per_atom=23.0390 meV, MAE_F=42.7183 meV / A, MAE_stress_per_atom=0.1276 meV / A^3 2023-12-25 16:11:22.931 INFO: Epoch 72: loss=4.6828e-03, MAE_E_per_atom=23.0003 meV, MAE_F=42.9575 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2023-12-25 16:24:11.499 INFO: Epoch 73: loss=4.6842e-03, MAE_E_per_atom=22.8021 meV, MAE_F=42.7446 meV / A, MAE_stress_per_atom=0.1282 meV / A^3 2023-12-25 16:36:53.306 INFO: Epoch 74: loss=4.6386e-03, MAE_E_per_atom=22.9380 meV, MAE_F=42.4319 meV / A, MAE_stress_per_atom=0.1271 meV / A^3 2023-12-25 16:49:34.406 INFO: Epoch 75: loss=4.6130e-03, MAE_E_per_atom=22.7307 meV, MAE_F=42.3924 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2023-12-25 17:02:18.860 INFO: Epoch 76: loss=4.6429e-03, MAE_E_per_atom=23.0472 meV, MAE_F=42.5632 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2023-12-25 17:15:02.343 INFO: Epoch 77: loss=4.6361e-03, MAE_E_per_atom=22.8733 meV, MAE_F=42.6534 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2023-12-26 09:16:11.149 INFO: Process group initialized: True 2023-12-26 09:16:11.150 INFO: Processes: 1 2023-12-26 09:16:11.150 INFO: MACE version: 0.3.0 2023-12-26 09:16:11.150 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=False, wandb_project='', wandb_entity='', wandb_name='', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-26 09:16:11.151 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-26 09:16:11.152 INFO: Using statistics json file 2023-12-26 09:16:11.152 INFO: Using atomic numbers from statistics file 2023-12-26 09:16:11.152 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-26 09:16:11.152 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-26 09:16:11.153 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-26 09:16:41.850 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-26 09:16:41.853 INFO: Average number of neighbors: 61.964672446250916 2023-12-26 09:16:41.853 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-26 09:16:41.853 INFO: Building model 2023-12-26 09:16:41.854 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-26 09:16:45.781 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-26 09:16:45.782 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-77.pt 2023-12-26 09:16:46.036 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-26 09:16:46.041 INFO: Number of parameters: 5725072 2023-12-26 09:16:46.041 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-26 09:16:46.210 INFO: Computing metrics for training, validation, and test sets 2023-12-26 09:16:46.212 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-77.pt 2023-12-26 09:16:46.293 INFO: Loaded model from epoch 77 2023-12-26 09:16:46.294 INFO: Saving model to checkpoints/04-128-L2_run-1.model 2023-12-26 09:16:46.528 INFO: Evaluating train ... 2023-12-26 10:56:32.062 INFO: Process group initialized: True 2023-12-26 10:56:32.065 INFO: Processes: 120 2023-12-26 10:56:32.065 INFO: MACE version: 0.3.0 2023-12-26 10:56:32.065 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-26 10:56:32.065 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-26 10:56:32.066 INFO: Using statistics json file 2023-12-26 10:56:32.066 INFO: Using atomic numbers from statistics file 2023-12-26 10:56:32.066 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-26 10:56:32.066 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-26 10:56:32.067 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-26 10:57:04.964 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-26 10:57:04.967 INFO: Average number of neighbors: 61.964672446250916 2023-12-26 10:57:04.967 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-26 10:57:04.967 INFO: Building model 2023-12-26 10:57:04.969 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-26 10:57:09.709 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-26 10:57:09.711 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-77.pt 2023-12-26 10:57:09.991 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-26 10:57:09.997 INFO: Number of parameters: 5725072 2023-12-26 10:57:09.997 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-26 10:57:09.997 INFO: Using Weights and Biases for logging 2023-12-26 10:57:22.506 INFO: Using gradient clipping with tolerance=100.000 2023-12-26 10:57:22.507 INFO: Started training 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.724 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.725 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 10:57:30.726 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 11:20:40.676 INFO: Epoch 77: loss=4.6580e-03, MAE_E_per_atom=23.1133 meV, MAE_F=42.8592 meV / A, MAE_stress_per_atom=0.1288 meV / A^3 2023-12-26 11:33:51.598 INFO: Epoch 78: loss=4.6381e-03, MAE_E_per_atom=23.1353 meV, MAE_F=42.4877 meV / A, MAE_stress_per_atom=0.1259 meV / A^3 2023-12-26 11:46:35.179 INFO: Epoch 79: loss=4.6577e-03, MAE_E_per_atom=23.1233 meV, MAE_F=42.5512 meV / A, MAE_stress_per_atom=0.1289 meV / A^3 2023-12-26 11:59:23.456 INFO: Epoch 80: loss=4.5732e-03, MAE_E_per_atom=22.8887 meV, MAE_F=42.1375 meV / A, MAE_stress_per_atom=0.1270 meV / A^3 2023-12-26 12:12:06.825 INFO: Epoch 81: loss=4.6215e-03, MAE_E_per_atom=22.9679 meV, MAE_F=42.4798 meV / A, MAE_stress_per_atom=0.1271 meV / A^3 2023-12-26 12:24:51.351 INFO: Epoch 82: loss=4.6316e-03, MAE_E_per_atom=22.8871 meV, MAE_F=42.4670 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2023-12-26 12:37:26.831 INFO: Epoch 83: loss=4.6036e-03, MAE_E_per_atom=22.8138 meV, MAE_F=42.3416 meV / A, MAE_stress_per_atom=0.1270 meV / A^3 2023-12-26 12:50:08.495 INFO: Epoch 84: loss=4.6105e-03, MAE_E_per_atom=22.8955 meV, MAE_F=42.3072 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2023-12-26 23:15:08.677 INFO: Process group initialized: True 2023-12-26 23:15:08.680 INFO: Processes: 120 2023-12-26 23:15:08.680 INFO: MACE version: 0.3.0 2023-12-26 23:15:08.680 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-26 23:15:08.680 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-26 23:15:08.681 INFO: Using statistics json file 2023-12-26 23:15:08.681 INFO: Using atomic numbers from statistics file 2023-12-26 23:15:08.681 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-26 23:15:08.681 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-26 23:15:08.682 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-26 23:15:41.256 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-26 23:15:41.259 INFO: Average number of neighbors: 61.964672446250916 2023-12-26 23:15:41.259 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-26 23:15:41.259 INFO: Building model 2023-12-26 23:15:41.260 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-26 23:15:45.989 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-26 23:15:45.991 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-84.pt 2023-12-26 23:15:46.255 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-26 23:15:46.262 INFO: Number of parameters: 5725072 2023-12-26 23:15:46.262 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.004 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-26 23:15:46.262 INFO: Using Weights and Biases for logging 2023-12-26 23:15:58.578 INFO: Using gradient clipping with tolerance=100.000 2023-12-26 23:15:58.579 INFO: Started training 2023-12-26 23:16:06.870 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.870 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.870 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.870 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.872 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.872 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.872 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.873 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.872 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.873 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.872 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.873 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.875 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:16:06.876 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-26 23:38:52.994 INFO: Epoch 84: loss=4.6466e-03, MAE_E_per_atom=22.9602 meV, MAE_F=42.5453 meV / A, MAE_stress_per_atom=0.1276 meV / A^3 2023-12-26 23:52:20.501 INFO: Epoch 85: loss=4.6175e-03, MAE_E_per_atom=22.8635 meV, MAE_F=42.2849 meV / A, MAE_stress_per_atom=0.1271 meV / A^3 2023-12-27 00:05:05.547 INFO: Epoch 86: loss=4.6369e-03, MAE_E_per_atom=22.4999 meV, MAE_F=42.3401 meV / A, MAE_stress_per_atom=0.1294 meV / A^3 2023-12-27 00:17:54.030 INFO: Epoch 87: loss=4.5507e-03, MAE_E_per_atom=22.5242 meV, MAE_F=41.9748 meV / A, MAE_stress_per_atom=0.1271 meV / A^3 2023-12-27 00:30:32.675 INFO: Epoch 88: loss=4.5669e-03, MAE_E_per_atom=22.4718 meV, MAE_F=42.2104 meV / A, MAE_stress_per_atom=0.1265 meV / A^3 2023-12-27 00:43:16.455 INFO: Epoch 89: loss=4.5513e-03, MAE_E_per_atom=22.5063 meV, MAE_F=42.0624 meV / A, MAE_stress_per_atom=0.1274 meV / A^3 2023-12-27 00:55:56.556 INFO: Epoch 90: loss=4.5846e-03, MAE_E_per_atom=22.6860 meV, MAE_F=42.0623 meV / A, MAE_stress_per_atom=0.1277 meV / A^3 2023-12-27 01:08:39.132 INFO: Epoch 91: loss=4.5520e-03, MAE_E_per_atom=22.6590 meV, MAE_F=41.7997 meV / A, MAE_stress_per_atom=0.1275 meV / A^3 2023-12-27 02:01:23.886 INFO: Process group initialized: True 2023-12-27 02:01:23.888 INFO: Processes: 120 2023-12-27 02:01:23.888 INFO: MACE version: 0.3.0 2023-12-27 02:01:23.889 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-27 02:01:23.889 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-27 02:01:23.889 INFO: Using statistics json file 2023-12-27 02:01:23.889 INFO: Using atomic numbers from statistics file 2023-12-27 02:01:23.889 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-27 02:01:23.889 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-27 02:01:23.890 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-27 02:01:54.693 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-27 02:01:54.696 INFO: Average number of neighbors: 61.964672446250916 2023-12-27 02:01:54.696 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-27 02:01:54.696 INFO: Building model 2023-12-27 02:01:54.698 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-27 02:01:59.068 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-27 02:01:59.070 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-91.pt 2023-12-27 02:01:59.330 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-27 02:01:59.336 INFO: Number of parameters: 5725072 2023-12-27 02:01:59.336 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-27 02:01:59.336 INFO: Using Weights and Biases for logging 2023-12-27 02:02:13.833 INFO: Using gradient clipping with tolerance=100.000 2023-12-27 02:02:13.833 INFO: Started training 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.771 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.771 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.771 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.769 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.771 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.770 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:02:21.771 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-27 02:25:04.832 INFO: Epoch 91: loss=4.5711e-03, MAE_E_per_atom=22.3716 meV, MAE_F=41.9662 meV / A, MAE_stress_per_atom=0.1260 meV / A^3 2023-12-27 02:38:34.680 INFO: Epoch 92: loss=4.5433e-03, MAE_E_per_atom=22.4484 meV, MAE_F=41.7816 meV / A, MAE_stress_per_atom=0.1292 meV / A^3 2023-12-27 02:51:14.059 INFO: Epoch 93: loss=4.5439e-03, MAE_E_per_atom=22.4020 meV, MAE_F=41.8205 meV / A, MAE_stress_per_atom=0.1265 meV / A^3 2023-12-27 03:03:58.458 INFO: Epoch 94: loss=4.5635e-03, MAE_E_per_atom=22.4489 meV, MAE_F=41.8721 meV / A, MAE_stress_per_atom=0.1277 meV / A^3 2023-12-27 03:16:39.150 INFO: Epoch 95: loss=4.5681e-03, MAE_E_per_atom=22.4368 meV, MAE_F=41.6650 meV / A, MAE_stress_per_atom=0.1293 meV / A^3 2023-12-27 03:29:17.728 INFO: Epoch 96: loss=4.5148e-03, MAE_E_per_atom=22.5692 meV, MAE_F=41.4532 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2023-12-27 03:42:02.948 INFO: Epoch 97: loss=4.5474e-03, MAE_E_per_atom=22.4840 meV, MAE_F=41.6572 meV / A, MAE_stress_per_atom=0.1293 meV / A^3 2023-12-27 03:54:44.235 INFO: Epoch 98: loss=4.5442e-03, MAE_E_per_atom=22.2663 meV, MAE_F=41.5326 meV / A, MAE_stress_per_atom=0.1292 meV / A^3 2023-12-28 06:25:32.255 INFO: Process group initialized: True 2023-12-28 06:25:32.258 INFO: Processes: 120 2023-12-28 06:25:32.258 INFO: MACE version: 0.3.0 2023-12-28 06:25:32.259 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-28 06:25:32.259 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-28 06:25:32.259 INFO: Using statistics json file 2023-12-28 06:25:32.259 INFO: Using atomic numbers from statistics file 2023-12-28 06:25:32.259 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-28 06:25:32.259 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-28 06:25:32.260 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-28 06:26:07.062 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-28 06:26:07.064 INFO: Average number of neighbors: 61.964672446250916 2023-12-28 06:26:07.064 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-28 06:26:07.064 INFO: Building model 2023-12-28 06:26:07.066 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-28 06:26:11.777 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-28 06:26:11.780 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-98.pt 2023-12-28 06:26:12.051 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-28 06:26:12.059 INFO: Number of parameters: 5725072 2023-12-28 06:26:12.059 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0032 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-28 06:26:12.060 INFO: Using Weights and Biases for logging 2023-12-28 06:26:25.654 INFO: Using gradient clipping with tolerance=100.000 2023-12-28 06:26:25.654 INFO: Started training 2023-12-28 06:26:33.541 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.541 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.541 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.541 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.546 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.547 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:26:33.548 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 06:49:36.144 INFO: Epoch 98: loss=4.5500e-03, MAE_E_per_atom=22.2929 meV, MAE_F=41.7091 meV / A, MAE_stress_per_atom=0.1298 meV / A^3 2023-12-28 07:02:45.135 INFO: Epoch 99: loss=4.5472e-03, MAE_E_per_atom=22.4445 meV, MAE_F=41.5481 meV / A, MAE_stress_per_atom=0.1284 meV / A^3 2023-12-28 07:15:35.037 INFO: Epoch 100: loss=4.5300e-03, MAE_E_per_atom=22.3411 meV, MAE_F=41.5720 meV / A, MAE_stress_per_atom=0.1295 meV / A^3 2023-12-28 07:28:15.240 INFO: Epoch 101: loss=4.5286e-03, MAE_E_per_atom=22.3204 meV, MAE_F=41.7102 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2023-12-28 07:40:54.936 INFO: Epoch 102: loss=4.5414e-03, MAE_E_per_atom=22.5669 meV, MAE_F=41.6963 meV / A, MAE_stress_per_atom=0.1263 meV / A^3 2023-12-28 07:53:38.299 INFO: Epoch 103: loss=4.5057e-03, MAE_E_per_atom=22.1831 meV, MAE_F=41.4752 meV / A, MAE_stress_per_atom=0.1276 meV / A^3 2023-12-28 08:06:24.236 INFO: Epoch 104: loss=4.5071e-03, MAE_E_per_atom=22.2563 meV, MAE_F=41.4439 meV / A, MAE_stress_per_atom=0.1281 meV / A^3 2023-12-28 08:19:03.693 INFO: Epoch 105: loss=4.4769e-03, MAE_E_per_atom=22.3013 meV, MAE_F=41.2685 meV / A, MAE_stress_per_atom=0.1285 meV / A^3 2023-12-28 12:52:45.699 INFO: Process group initialized: True 2023-12-28 12:52:45.701 INFO: Processes: 120 2023-12-28 12:52:45.701 INFO: MACE version: 0.3.0 2023-12-28 12:52:45.701 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-28 12:52:45.702 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-28 12:52:45.702 INFO: Using statistics json file 2023-12-28 12:52:45.702 INFO: Using atomic numbers from statistics file 2023-12-28 12:52:45.702 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-28 12:52:45.702 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-28 12:52:45.703 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-28 12:53:18.316 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-28 12:53:18.317 INFO: Average number of neighbors: 61.964672446250916 2023-12-28 12:53:18.318 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-28 12:53:18.318 INFO: Building model 2023-12-28 12:53:18.319 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-28 12:53:22.943 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-28 12:53:22.946 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-105.pt 2023-12-28 12:53:23.214 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-28 12:53:23.221 INFO: Number of parameters: 5725072 2023-12-28 12:53:23.221 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-28 12:53:23.221 INFO: Using Weights and Biases for logging 2023-12-28 12:53:38.454 INFO: Using gradient clipping with tolerance=100.000 2023-12-28 12:53:38.455 INFO: Started training 2023-12-28 12:53:46.961 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.961 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.961 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.961 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.962 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.963 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.963 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.962 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.963 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.963 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.962 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.963 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.962 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.963 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.963 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.964 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 12:53:46.965 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 13:16:49.345 INFO: Epoch 105: loss=4.4828e-03, MAE_E_per_atom=22.4860 meV, MAE_F=41.3364 meV / A, MAE_stress_per_atom=0.1299 meV / A^3 2023-12-28 13:30:02.650 INFO: Epoch 106: loss=4.5188e-03, MAE_E_per_atom=22.2441 meV, MAE_F=41.3287 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2023-12-28 13:42:55.649 INFO: Epoch 107: loss=4.4739e-03, MAE_E_per_atom=22.3598 meV, MAE_F=41.0709 meV / A, MAE_stress_per_atom=0.1282 meV / A^3 2023-12-28 13:55:37.536 INFO: Epoch 108: loss=4.5051e-03, MAE_E_per_atom=22.2788 meV, MAE_F=41.3070 meV / A, MAE_stress_per_atom=0.1264 meV / A^3 2023-12-28 14:08:20.046 INFO: Epoch 109: loss=4.5144e-03, MAE_E_per_atom=22.0693 meV, MAE_F=41.2900 meV / A, MAE_stress_per_atom=0.1289 meV / A^3 2023-12-28 14:21:03.064 INFO: Epoch 110: loss=4.4770e-03, MAE_E_per_atom=21.9786 meV, MAE_F=41.0889 meV / A, MAE_stress_per_atom=0.1277 meV / A^3 2023-12-28 14:33:43.190 INFO: Epoch 111: loss=4.5056e-03, MAE_E_per_atom=22.0391 meV, MAE_F=41.2352 meV / A, MAE_stress_per_atom=0.1295 meV / A^3 2023-12-28 14:46:26.495 INFO: Epoch 112: loss=4.4724e-03, MAE_E_per_atom=22.1658 meV, MAE_F=41.0638 meV / A, MAE_stress_per_atom=0.1272 meV / A^3 2023-12-28 18:00:04.660 INFO: Process group initialized: True 2023-12-28 18:00:04.662 INFO: Processes: 120 2023-12-28 18:00:04.662 INFO: MACE version: 0.3.0 2023-12-28 18:00:04.663 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-28 18:00:04.663 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-28 18:00:04.663 INFO: Using statistics json file 2023-12-28 18:00:04.663 INFO: Using atomic numbers from statistics file 2023-12-28 18:00:04.663 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-28 18:00:04.663 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-28 18:00:04.664 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-28 18:00:35.811 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-28 18:00:35.814 INFO: Average number of neighbors: 61.964672446250916 2023-12-28 18:00:35.814 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-28 18:00:35.814 INFO: Building model 2023-12-28 18:00:35.815 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-28 18:00:40.233 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-28 18:00:40.236 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-112.pt 2023-12-28 18:00:40.507 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-28 18:00:40.513 INFO: Number of parameters: 5725072 2023-12-28 18:00:40.514 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-28 18:00:40.514 INFO: Using Weights and Biases for logging 2023-12-28 18:00:55.420 INFO: Using gradient clipping with tolerance=100.000 2023-12-28 18:00:55.421 INFO: Started training 2023-12-28 18:01:03.549 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.549 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.549 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.549 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.553 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.555 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:01:03.556 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-28 18:24:05.998 INFO: Epoch 112: loss=4.4587e-03, MAE_E_per_atom=22.1189 meV, MAE_F=41.1929 meV / A, MAE_stress_per_atom=0.1256 meV / A^3 2023-12-28 18:37:29.954 INFO: Epoch 113: loss=4.4523e-03, MAE_E_per_atom=21.9660 meV, MAE_F=41.2330 meV / A, MAE_stress_per_atom=0.1280 meV / A^3 2023-12-28 18:50:14.726 INFO: Epoch 114: loss=4.4498e-03, MAE_E_per_atom=21.9376 meV, MAE_F=41.0059 meV / A, MAE_stress_per_atom=0.1276 meV / A^3 2023-12-28 19:03:00.756 INFO: Epoch 115: loss=4.4642e-03, MAE_E_per_atom=21.7800 meV, MAE_F=41.0251 meV / A, MAE_stress_per_atom=0.1293 meV / A^3 2023-12-28 19:15:46.621 INFO: Epoch 116: loss=4.4663e-03, MAE_E_per_atom=21.9834 meV, MAE_F=40.9975 meV / A, MAE_stress_per_atom=0.1284 meV / A^3 2023-12-28 19:28:30.244 INFO: Epoch 117: loss=4.4366e-03, MAE_E_per_atom=21.8409 meV, MAE_F=40.8114 meV / A, MAE_stress_per_atom=0.1290 meV / A^3 2023-12-28 19:41:11.172 INFO: Epoch 118: loss=4.4366e-03, MAE_E_per_atom=21.7229 meV, MAE_F=40.6612 meV / A, MAE_stress_per_atom=0.1294 meV / A^3 2023-12-28 19:53:55.961 INFO: Epoch 119: loss=4.4349e-03, MAE_E_per_atom=22.0540 meV, MAE_F=41.0081 meV / A, MAE_stress_per_atom=0.1280 meV / A^3 2023-12-29 02:56:18.036 INFO: Process group initialized: True 2023-12-29 02:56:18.039 INFO: Processes: 120 2023-12-29 02:56:18.039 INFO: MACE version: 0.3.0 2023-12-29 02:56:18.039 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-29 02:56:18.039 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-29 02:56:18.040 INFO: Using statistics json file 2023-12-29 02:56:18.040 INFO: Using atomic numbers from statistics file 2023-12-29 02:56:18.040 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-29 02:56:18.040 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-29 02:56:18.041 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-29 02:56:49.249 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-29 02:56:49.251 INFO: Average number of neighbors: 61.964672446250916 2023-12-29 02:56:49.251 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-29 02:56:49.251 INFO: Building model 2023-12-29 02:56:49.253 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-29 02:56:53.788 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-29 02:56:53.790 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-119.pt 2023-12-29 02:56:54.059 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-29 02:56:54.066 INFO: Number of parameters: 5725072 2023-12-29 02:56:54.066 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-29 02:56:54.066 INFO: Using Weights and Biases for logging 2023-12-29 02:57:15.118 INFO: Using gradient clipping with tolerance=100.000 2023-12-29 02:57:15.118 INFO: Started training 2023-12-29 02:57:23.183 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.183 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.183 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.183 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.185 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.186 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 02:57:23.187 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 03:20:27.161 INFO: Epoch 119: loss=4.4512e-03, MAE_E_per_atom=22.0972 meV, MAE_F=41.3074 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2023-12-29 03:33:42.142 INFO: Epoch 120: loss=4.4490e-03, MAE_E_per_atom=21.8147 meV, MAE_F=41.1683 meV / A, MAE_stress_per_atom=0.1271 meV / A^3 2023-12-29 03:46:33.625 INFO: Epoch 121: loss=4.4322e-03, MAE_E_per_atom=21.8152 meV, MAE_F=40.8060 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2023-12-29 03:59:18.953 INFO: Epoch 122: loss=4.4480e-03, MAE_E_per_atom=21.9033 meV, MAE_F=40.9227 meV / A, MAE_stress_per_atom=0.1283 meV / A^3 2023-12-29 04:12:05.044 INFO: Epoch 123: loss=4.4278e-03, MAE_E_per_atom=21.8330 meV, MAE_F=40.6735 meV / A, MAE_stress_per_atom=0.1290 meV / A^3 2023-12-29 04:24:52.225 INFO: Epoch 124: loss=4.4275e-03, MAE_E_per_atom=22.0194 meV, MAE_F=40.8447 meV / A, MAE_stress_per_atom=0.1275 meV / A^3 2023-12-29 04:37:32.306 INFO: Epoch 125: loss=4.4359e-03, MAE_E_per_atom=21.8150 meV, MAE_F=40.8074 meV / A, MAE_stress_per_atom=0.1296 meV / A^3 2023-12-29 04:50:14.367 INFO: Epoch 126: loss=4.4462e-03, MAE_E_per_atom=21.9246 meV, MAE_F=40.8175 meV / A, MAE_stress_per_atom=0.1251 meV / A^3 2023-12-29 06:06:26.253 INFO: Process group initialized: True 2023-12-29 06:06:26.256 INFO: Processes: 120 2023-12-29 06:06:26.256 INFO: MACE version: 0.3.0 2023-12-29 06:06:26.256 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-29 06:06:26.256 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-29 06:06:26.256 INFO: Using statistics json file 2023-12-29 06:06:26.256 INFO: Using atomic numbers from statistics file 2023-12-29 06:06:26.256 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-29 06:06:26.257 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-29 06:06:26.257 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-29 06:06:59.813 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-29 06:06:59.815 INFO: Average number of neighbors: 61.964672446250916 2023-12-29 06:06:59.815 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-29 06:06:59.815 INFO: Building model 2023-12-29 06:06:59.817 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-29 06:07:04.547 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-29 06:07:04.550 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-126.pt 2023-12-29 06:07:04.818 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-29 06:07:04.825 INFO: Number of parameters: 5725072 2023-12-29 06:07:04.825 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-29 06:07:04.825 INFO: Using Weights and Biases for logging 2023-12-29 06:07:18.836 INFO: Using gradient clipping with tolerance=100.000 2023-12-29 06:07:18.836 INFO: Started training 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.504 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.505 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:07:27.506 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 06:29:58.323 INFO: Epoch 126: loss=4.4505e-03, MAE_E_per_atom=21.9259 meV, MAE_F=41.0704 meV / A, MAE_stress_per_atom=0.1234 meV / A^3 2023-12-29 06:43:22.065 INFO: Epoch 127: loss=4.4382e-03, MAE_E_per_atom=21.8953 meV, MAE_F=40.6127 meV / A, MAE_stress_per_atom=0.1302 meV / A^3 2023-12-29 06:56:06.377 INFO: Epoch 128: loss=4.4368e-03, MAE_E_per_atom=21.8390 meV, MAE_F=40.6097 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2023-12-29 07:08:52.008 INFO: Epoch 129: loss=4.4065e-03, MAE_E_per_atom=21.8022 meV, MAE_F=40.8261 meV / A, MAE_stress_per_atom=0.1287 meV / A^3 2023-12-29 07:21:34.809 INFO: Epoch 130: loss=4.4349e-03, MAE_E_per_atom=21.7340 meV, MAE_F=40.6597 meV / A, MAE_stress_per_atom=0.1264 meV / A^3 2023-12-29 07:34:19.675 INFO: Epoch 131: loss=4.4164e-03, MAE_E_per_atom=21.6467 meV, MAE_F=40.7758 meV / A, MAE_stress_per_atom=0.1280 meV / A^3 2023-12-29 07:47:01.578 INFO: Epoch 132: loss=4.4235e-03, MAE_E_per_atom=21.7737 meV, MAE_F=40.6142 meV / A, MAE_stress_per_atom=0.1290 meV / A^3 2023-12-29 07:59:44.766 INFO: Epoch 133: loss=4.4011e-03, MAE_E_per_atom=21.9393 meV, MAE_F=40.4133 meV / A, MAE_stress_per_atom=0.1292 meV / A^3 2023-12-29 11:23:37.606 INFO: Process group initialized: True 2023-12-29 11:23:37.608 INFO: Processes: 120 2023-12-29 11:23:37.608 INFO: MACE version: 0.3.0 2023-12-29 11:23:37.608 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-29 11:23:37.608 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-29 11:23:37.609 INFO: Using statistics json file 2023-12-29 11:23:37.609 INFO: Using atomic numbers from statistics file 2023-12-29 11:23:37.609 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-29 11:23:37.609 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-29 11:23:37.610 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-29 11:24:13.822 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-29 11:24:13.825 INFO: Average number of neighbors: 61.964672446250916 2023-12-29 11:24:13.825 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-29 11:24:13.825 INFO: Building model 2023-12-29 11:24:13.826 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-29 11:24:18.694 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-29 11:24:18.697 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-133.pt 2023-12-29 11:24:19.013 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-29 11:24:19.020 INFO: Number of parameters: 5725072 2023-12-29 11:24:19.020 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.00256 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-29 11:24:19.020 INFO: Using Weights and Biases for logging 2023-12-29 11:24:32.073 INFO: Using gradient clipping with tolerance=100.000 2023-12-29 11:24:32.073 INFO: Started training 2023-12-29 11:24:40.662 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.662 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.662 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.662 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.665 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.666 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.667 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.668 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:24:40.669 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-29 11:47:11.571 INFO: Epoch 133: loss=4.4364e-03, MAE_E_per_atom=22.0924 meV, MAE_F=40.6711 meV / A, MAE_stress_per_atom=0.1302 meV / A^3 2023-12-29 12:00:22.494 INFO: Epoch 134: loss=4.4203e-03, MAE_E_per_atom=21.8599 meV, MAE_F=40.5822 meV / A, MAE_stress_per_atom=0.1281 meV / A^3 2023-12-29 12:13:11.104 INFO: Epoch 135: loss=4.4276e-03, MAE_E_per_atom=21.6984 meV, MAE_F=40.6215 meV / A, MAE_stress_per_atom=0.1282 meV / A^3 2023-12-29 12:25:54.876 INFO: Epoch 136: loss=4.4082e-03, MAE_E_per_atom=21.7185 meV, MAE_F=40.5382 meV / A, MAE_stress_per_atom=0.1274 meV / A^3 2023-12-29 12:38:38.487 INFO: Epoch 137: loss=4.4078e-03, MAE_E_per_atom=21.5108 meV, MAE_F=40.5573 meV / A, MAE_stress_per_atom=0.1268 meV / A^3 2023-12-29 12:51:18.187 INFO: Epoch 138: loss=4.3981e-03, MAE_E_per_atom=21.4447 meV, MAE_F=40.4495 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2023-12-29 13:03:59.359 INFO: Epoch 139: loss=4.4160e-03, MAE_E_per_atom=21.7056 meV, MAE_F=40.5449 meV / A, MAE_stress_per_atom=0.1282 meV / A^3 2023-12-29 13:16:36.589 INFO: Epoch 140: loss=4.3485e-03, MAE_E_per_atom=21.5653 meV, MAE_F=40.4452 meV / A, MAE_stress_per_atom=0.1281 meV / A^3 2023-12-30 00:24:08.828 INFO: Process group initialized: True 2023-12-30 00:24:08.830 INFO: Processes: 120 2023-12-30 00:24:08.831 INFO: MACE version: 0.3.0 2023-12-30 00:24:08.831 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-30 00:24:08.831 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-30 00:24:08.832 INFO: Using statistics json file 2023-12-30 00:24:08.832 INFO: Using atomic numbers from statistics file 2023-12-30 00:24:08.832 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-30 00:24:08.832 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-30 00:24:08.833 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-30 00:24:44.272 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-30 00:24:44.275 INFO: Average number of neighbors: 61.964672446250916 2023-12-30 00:24:44.275 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-30 00:24:44.275 INFO: Building model 2023-12-30 00:24:44.276 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-30 00:24:49.095 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-30 00:24:49.098 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-140.pt 2023-12-30 00:24:49.377 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-30 00:24:49.384 INFO: Number of parameters: 5725072 2023-12-30 00:24:49.384 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-30 00:24:49.384 INFO: Using Weights and Biases for logging 2023-12-30 00:25:03.864 INFO: Using gradient clipping with tolerance=100.000 2023-12-30 00:25:03.864 INFO: Started training 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.201 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.202 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.203 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:25:12.204 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 00:47:39.569 INFO: Epoch 140: loss=4.3788e-03, MAE_E_per_atom=21.7055 meV, MAE_F=40.8423 meV / A, MAE_stress_per_atom=0.1295 meV / A^3 2023-12-30 01:00:55.089 INFO: Epoch 141: loss=4.3739e-03, MAE_E_per_atom=21.3528 meV, MAE_F=40.3666 meV / A, MAE_stress_per_atom=0.1294 meV / A^3 2023-12-30 01:13:46.796 INFO: Epoch 142: loss=4.3530e-03, MAE_E_per_atom=21.6408 meV, MAE_F=40.1551 meV / A, MAE_stress_per_atom=0.1298 meV / A^3 2023-12-30 01:26:28.667 INFO: Epoch 143: loss=4.3650e-03, MAE_E_per_atom=21.6786 meV, MAE_F=40.1316 meV / A, MAE_stress_per_atom=0.1284 meV / A^3 2023-12-30 01:39:14.444 INFO: Epoch 144: loss=4.4047e-03, MAE_E_per_atom=21.6211 meV, MAE_F=40.5102 meV / A, MAE_stress_per_atom=0.1299 meV / A^3 2023-12-30 01:51:52.484 INFO: Epoch 145: loss=4.3586e-03, MAE_E_per_atom=21.5802 meV, MAE_F=40.2380 meV / A, MAE_stress_per_atom=0.1270 meV / A^3 2023-12-30 02:04:31.692 INFO: Epoch 146: loss=4.4028e-03, MAE_E_per_atom=21.5145 meV, MAE_F=40.2603 meV / A, MAE_stress_per_atom=0.1297 meV / A^3 2023-12-30 02:17:10.592 INFO: Epoch 147: loss=4.3854e-03, MAE_E_per_atom=21.4506 meV, MAE_F=40.2442 meV / A, MAE_stress_per_atom=0.1272 meV / A^3 2023-12-30 09:46:15.918 INFO: Process group initialized: True 2023-12-30 09:46:15.921 INFO: Processes: 120 2023-12-30 09:46:15.921 INFO: MACE version: 0.3.0 2023-12-30 09:46:15.921 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-30 09:46:15.921 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-30 09:46:15.921 INFO: Using statistics json file 2023-12-30 09:46:15.921 INFO: Using atomic numbers from statistics file 2023-12-30 09:46:15.922 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-30 09:46:15.922 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-30 09:46:15.922 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-30 09:46:49.020 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-30 09:46:49.023 INFO: Average number of neighbors: 61.964672446250916 2023-12-30 09:46:49.023 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-30 09:46:49.023 INFO: Building model 2023-12-30 09:46:49.024 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-30 09:46:53.652 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-30 09:46:53.655 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-147.pt 2023-12-30 09:46:53.952 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-30 09:46:53.959 INFO: Number of parameters: 5725072 2023-12-30 09:46:53.959 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0020480000000000003 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-30 09:46:53.959 INFO: Using Weights and Biases for logging 2023-12-30 09:47:08.729 INFO: Using gradient clipping with tolerance=100.000 2023-12-30 09:47:08.729 INFO: Started training 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.099 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.100 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 09:47:17.101 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-30 10:10:07.224 INFO: Epoch 147: loss=4.3926e-03, MAE_E_per_atom=21.5321 meV, MAE_F=40.3273 meV / A, MAE_stress_per_atom=0.1255 meV / A^3 2023-12-30 10:22:57.925 INFO: Epoch 148: loss=4.3761e-03, MAE_E_per_atom=21.4143 meV, MAE_F=40.1603 meV / A, MAE_stress_per_atom=0.1280 meV / A^3 2023-12-30 10:35:47.756 INFO: Epoch 149: loss=4.3657e-03, MAE_E_per_atom=21.2935 meV, MAE_F=40.3572 meV / A, MAE_stress_per_atom=0.1277 meV / A^3 2023-12-30 10:48:28.781 INFO: Epoch 150: loss=4.3596e-03, MAE_E_per_atom=21.4239 meV, MAE_F=40.0654 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2023-12-30 11:01:13.722 INFO: Epoch 151: loss=4.3451e-03, MAE_E_per_atom=21.4164 meV, MAE_F=40.0362 meV / A, MAE_stress_per_atom=0.1289 meV / A^3 2023-12-30 11:13:52.411 INFO: Epoch 152: loss=4.3305e-03, MAE_E_per_atom=21.4986 meV, MAE_F=39.9532 meV / A, MAE_stress_per_atom=0.1291 meV / A^3 2023-12-30 11:26:31.754 INFO: Epoch 153: loss=4.3597e-03, MAE_E_per_atom=21.5822 meV, MAE_F=40.0393 meV / A, MAE_stress_per_atom=0.1291 meV / A^3 2023-12-30 11:39:11.253 INFO: Epoch 154: loss=4.3625e-03, MAE_E_per_atom=21.4279 meV, MAE_F=40.1447 meV / A, MAE_stress_per_atom=0.1285 meV / A^3 2023-12-31 13:46:53.445 INFO: Process group initialized: True 2023-12-31 13:46:53.448 INFO: Processes: 120 2023-12-31 13:46:53.448 INFO: MACE version: 0.3.0 2023-12-31 13:46:53.448 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2023-12-31 13:46:53.448 INFO: CUDA version: 11.8, CUDA device: 0 2023-12-31 13:46:53.449 INFO: Using statistics json file 2023-12-31 13:46:53.449 INFO: Using atomic numbers from statistics file 2023-12-31 13:46:53.449 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2023-12-31 13:46:53.449 INFO: Atomic Energies not in training file, using command line argument E0s 2023-12-31 13:46:53.450 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2023-12-31 13:47:26.092 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2023-12-31 13:47:26.095 INFO: Average number of neighbors: 61.964672446250916 2023-12-31 13:47:26.095 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2023-12-31 13:47:26.095 INFO: Building model 2023-12-31 13:47:26.096 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2023-12-31 13:47:30.743 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2023-12-31 13:47:30.746 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-154.pt 2023-12-31 13:47:31.018 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2023-12-31 13:47:31.025 INFO: Number of parameters: 5725072 2023-12-31 13:47:31.025 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0016384000000000004 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0016384000000000004 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0016384000000000004 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0016384000000000004 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0016384000000000004 maximize: False name: readouts weight_decay: 0.0 ) 2023-12-31 13:47:31.025 INFO: Using Weights and Biases for logging 2023-12-31 13:47:45.269 INFO: Using gradient clipping with tolerance=100.000 2023-12-31 13:47:45.269 INFO: Started training 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.373 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.373 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.373 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.373 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.373 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.373 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.372 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.373 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.373 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.375 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.376 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 13:47:53.377 INFO: Reducer buckets have been rebuilt in this iteration. 2023-12-31 14:10:59.458 INFO: Epoch 154: loss=4.3638e-03, MAE_E_per_atom=21.4655 meV, MAE_F=39.9796 meV / A, MAE_stress_per_atom=0.1292 meV / A^3 2023-12-31 14:23:54.699 INFO: Epoch 155: loss=4.3762e-03, MAE_E_per_atom=21.0918 meV, MAE_F=39.9617 meV / A, MAE_stress_per_atom=0.1300 meV / A^3 2023-12-31 14:36:43.304 INFO: Epoch 156: loss=4.3428e-03, MAE_E_per_atom=21.2606 meV, MAE_F=40.0200 meV / A, MAE_stress_per_atom=0.1282 meV / A^3 2023-12-31 14:49:27.468 INFO: Epoch 157: loss=4.3456e-03, MAE_E_per_atom=21.0961 meV, MAE_F=39.9568 meV / A, MAE_stress_per_atom=0.1271 meV / A^3 2023-12-31 15:02:16.816 INFO: Epoch 158: loss=4.3552e-03, MAE_E_per_atom=21.1007 meV, MAE_F=39.9851 meV / A, MAE_stress_per_atom=0.1272 meV / A^3 2023-12-31 15:15:01.791 INFO: Epoch 159: loss=4.3194e-03, MAE_E_per_atom=21.1617 meV, MAE_F=39.8250 meV / A, MAE_stress_per_atom=0.1282 meV / A^3 2023-12-31 15:27:38.805 INFO: Epoch 160: loss=4.3093e-03, MAE_E_per_atom=21.3021 meV, MAE_F=39.7095 meV / A, MAE_stress_per_atom=0.1269 meV / A^3 2023-12-31 15:40:17.158 INFO: Epoch 161: loss=4.3426e-03, MAE_E_per_atom=21.2505 meV, MAE_F=39.8669 meV / A, MAE_stress_per_atom=0.1293 meV / A^3 2024-01-01 04:59:36.013 INFO: Process group initialized: True 2024-01-01 04:59:36.015 INFO: Processes: 120 2024-01-01 04:59:36.015 INFO: MACE version: 0.3.0 2024-01-01 04:59:36.015 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2024-01-01 04:59:36.016 INFO: CUDA version: 11.8, CUDA device: 0 2024-01-01 04:59:36.016 INFO: Using statistics json file 2024-01-01 04:59:36.016 INFO: Using atomic numbers from statistics file 2024-01-01 04:59:36.016 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2024-01-01 04:59:36.016 INFO: Atomic Energies not in training file, using command line argument E0s 2024-01-01 04:59:36.017 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2024-01-01 05:00:08.094 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2024-01-01 05:00:08.096 INFO: Average number of neighbors: 61.964672446250916 2024-01-01 05:00:08.096 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2024-01-01 05:00:08.096 INFO: Building model 2024-01-01 05:00:08.098 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2024-01-01 05:00:12.725 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2024-01-01 05:00:12.728 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-161.pt 2024-01-01 05:00:12.995 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2024-01-01 05:00:13.002 INFO: Number of parameters: 5725072 2024-01-01 05:00:13.002 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: readouts weight_decay: 0.0 ) 2024-01-01 05:00:13.002 INFO: Using Weights and Biases for logging 2024-01-01 05:00:29.196 INFO: Using gradient clipping with tolerance=100.000 2024-01-01 05:00:29.196 INFO: Started training 2024-01-01 05:00:37.535 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.535 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.535 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.535 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.540 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.540 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.540 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.540 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.542 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.543 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:00:37.544 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 05:24:07.546 INFO: Epoch 161: loss=4.3738e-03, MAE_E_per_atom=21.3603 meV, MAE_F=40.0938 meV / A, MAE_stress_per_atom=0.1294 meV / A^3 2024-01-01 05:37:13.556 INFO: Epoch 162: loss=4.2995e-03, MAE_E_per_atom=21.0992 meV, MAE_F=39.8633 meV / A, MAE_stress_per_atom=0.1283 meV / A^3 2024-01-01 05:50:01.095 INFO: Epoch 163: loss=4.3275e-03, MAE_E_per_atom=21.3120 meV, MAE_F=39.9257 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2024-01-01 06:02:45.294 INFO: Epoch 164: loss=4.3076e-03, MAE_E_per_atom=21.1436 meV, MAE_F=39.7815 meV / A, MAE_stress_per_atom=0.1281 meV / A^3 2024-01-01 06:15:31.111 INFO: Epoch 165: loss=4.3342e-03, MAE_E_per_atom=21.1831 meV, MAE_F=39.8543 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2024-01-01 06:28:13.348 INFO: Epoch 166: loss=4.3297e-03, MAE_E_per_atom=21.0989 meV, MAE_F=39.9077 meV / A, MAE_stress_per_atom=0.1275 meV / A^3 2024-01-01 06:40:55.229 INFO: Epoch 167: loss=4.3007e-03, MAE_E_per_atom=21.0377 meV, MAE_F=39.7723 meV / A, MAE_stress_per_atom=0.1280 meV / A^3 2024-01-01 06:53:38.363 INFO: Epoch 168: loss=4.3081e-03, MAE_E_per_atom=21.0724 meV, MAE_F=39.7409 meV / A, MAE_stress_per_atom=0.1291 meV / A^3 2024-01-01 07:37:34.385 INFO: Process group initialized: True 2024-01-01 07:37:34.387 INFO: Processes: 120 2024-01-01 07:37:34.387 INFO: MACE version: 0.3.0 2024-01-01 07:37:34.387 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2024-01-01 07:37:34.388 INFO: CUDA version: 11.8, CUDA device: 0 2024-01-01 07:37:34.388 INFO: Using statistics json file 2024-01-01 07:37:34.388 INFO: Using atomic numbers from statistics file 2024-01-01 07:37:34.388 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2024-01-01 07:37:34.388 INFO: Atomic Energies not in training file, using command line argument E0s 2024-01-01 07:37:34.389 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2024-01-01 07:38:07.492 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2024-01-01 07:38:07.494 INFO: Average number of neighbors: 61.964672446250916 2024-01-01 07:38:07.495 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2024-01-01 07:38:07.495 INFO: Building model 2024-01-01 07:38:07.496 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2024-01-01 07:38:12.238 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2024-01-01 07:38:12.241 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-168.pt 2024-01-01 07:38:12.516 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2024-01-01 07:38:12.522 INFO: Number of parameters: 5725072 2024-01-01 07:38:12.523 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0013107200000000005 maximize: False name: readouts weight_decay: 0.0 ) 2024-01-01 07:38:12.523 INFO: Using Weights and Biases for logging 2024-01-01 07:38:25.787 INFO: Using gradient clipping with tolerance=100.000 2024-01-01 07:38:25.787 INFO: Started training 2024-01-01 07:38:35.248 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.248 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.248 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.248 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.248 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.248 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.251 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 07:38:35.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-01 08:01:58.619 INFO: Epoch 168: loss=4.3081e-03, MAE_E_per_atom=21.1526 meV, MAE_F=39.8320 meV / A, MAE_stress_per_atom=0.1290 meV / A^3 2024-01-01 08:15:01.671 INFO: Epoch 169: loss=4.3032e-03, MAE_E_per_atom=21.2291 meV, MAE_F=39.6327 meV / A, MAE_stress_per_atom=0.1302 meV / A^3 2024-01-01 08:27:49.411 INFO: Epoch 170: loss=4.3131e-03, MAE_E_per_atom=21.2362 meV, MAE_F=39.7383 meV / A, MAE_stress_per_atom=0.1291 meV / A^3 2024-01-01 08:40:35.916 INFO: Epoch 171: loss=4.2960e-03, MAE_E_per_atom=21.1774 meV, MAE_F=39.6154 meV / A, MAE_stress_per_atom=0.1284 meV / A^3 2024-01-01 08:53:19.773 INFO: Epoch 172: loss=4.2938e-03, MAE_E_per_atom=21.0990 meV, MAE_F=39.6871 meV / A, MAE_stress_per_atom=0.1277 meV / A^3 2024-01-01 09:06:06.236 INFO: Epoch 173: loss=4.3019e-03, MAE_E_per_atom=21.0492 meV, MAE_F=39.7788 meV / A, MAE_stress_per_atom=0.1292 meV / A^3 2024-01-01 09:18:50.173 INFO: Epoch 174: loss=4.3028e-03, MAE_E_per_atom=21.1165 meV, MAE_F=39.8400 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2024-01-01 09:31:32.266 INFO: Epoch 175: loss=4.2999e-03, MAE_E_per_atom=21.1068 meV, MAE_F=39.6837 meV / A, MAE_stress_per_atom=0.1284 meV / A^3 2024-01-02 11:17:54.303 INFO: Process group initialized: True 2024-01-02 11:17:54.306 INFO: Processes: 120 2024-01-02 11:17:54.306 INFO: MACE version: 0.3.0 2024-01-02 11:17:54.306 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2024-01-02 11:17:54.307 INFO: CUDA version: 11.8, CUDA device: 0 2024-01-02 11:17:54.307 INFO: Using statistics json file 2024-01-02 11:17:54.307 INFO: Using atomic numbers from statistics file 2024-01-02 11:17:54.307 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2024-01-02 11:17:54.307 INFO: Atomic Energies not in training file, using command line argument E0s 2024-01-02 11:17:54.308 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2024-01-02 11:18:27.091 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2024-01-02 11:18:27.093 INFO: Average number of neighbors: 61.964672446250916 2024-01-02 11:18:27.093 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2024-01-02 11:18:27.093 INFO: Building model 2024-01-02 11:18:27.095 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2024-01-02 11:18:31.744 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2024-01-02 11:18:31.747 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-175.pt 2024-01-02 11:18:32.025 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2024-01-02 11:18:32.032 INFO: Number of parameters: 5725072 2024-01-02 11:18:32.032 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: readouts weight_decay: 0.0 ) 2024-01-02 11:18:32.032 INFO: Using Weights and Biases for logging 2024-01-02 11:18:45.133 INFO: Using gradient clipping with tolerance=100.000 2024-01-02 11:18:45.133 INFO: Started training 2024-01-02 11:18:53.247 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.247 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.247 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.247 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.248 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.249 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.250 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.252 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.253 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.254 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:18:53.255 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-02 11:41:50.841 INFO: Epoch 175: loss=4.2957e-03, MAE_E_per_atom=21.1026 meV, MAE_F=39.6312 meV / A, MAE_stress_per_atom=0.1277 meV / A^3 2024-01-02 11:55:06.859 INFO: Epoch 176: loss=4.2757e-03, MAE_E_per_atom=21.1312 meV, MAE_F=39.5812 meV / A, MAE_stress_per_atom=0.1292 meV / A^3 2024-01-02 12:07:51.528 INFO: Epoch 177: loss=4.2830e-03, MAE_E_per_atom=21.2162 meV, MAE_F=39.6899 meV / A, MAE_stress_per_atom=0.1272 meV / A^3 2024-01-02 12:20:35.061 INFO: Epoch 178: loss=4.2591e-03, MAE_E_per_atom=21.1147 meV, MAE_F=39.5522 meV / A, MAE_stress_per_atom=0.1266 meV / A^3 2024-01-02 12:33:15.802 INFO: Epoch 179: loss=4.2913e-03, MAE_E_per_atom=21.0240 meV, MAE_F=39.6361 meV / A, MAE_stress_per_atom=0.1280 meV / A^3 2024-01-02 12:45:58.633 INFO: Epoch 180: loss=4.2819e-03, MAE_E_per_atom=20.9826 meV, MAE_F=39.4529 meV / A, MAE_stress_per_atom=0.1274 meV / A^3 2024-01-02 12:58:40.031 INFO: Epoch 181: loss=4.2766e-03, MAE_E_per_atom=20.9282 meV, MAE_F=39.6364 meV / A, MAE_stress_per_atom=0.1267 meV / A^3 2024-01-02 13:11:18.832 INFO: Epoch 182: loss=4.2864e-03, MAE_E_per_atom=20.9097 meV, MAE_F=39.5347 meV / A, MAE_stress_per_atom=0.1275 meV / A^3 2024-01-03 16:18:22.597 INFO: Process group initialized: True 2024-01-03 16:18:22.600 INFO: Processes: 120 2024-01-03 16:18:22.600 INFO: MACE version: 0.3.0 2024-01-03 16:18:22.600 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2024-01-03 16:18:22.600 INFO: CUDA version: 11.8, CUDA device: 0 2024-01-03 16:18:22.601 INFO: Using statistics json file 2024-01-03 16:18:22.601 INFO: Using atomic numbers from statistics file 2024-01-03 16:18:22.602 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2024-01-03 16:18:22.602 INFO: Atomic Energies not in training file, using command line argument E0s 2024-01-03 16:18:22.602 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2024-01-03 16:18:55.378 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2024-01-03 16:18:55.380 INFO: Average number of neighbors: 61.964672446250916 2024-01-03 16:18:55.380 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2024-01-03 16:18:55.380 INFO: Building model 2024-01-03 16:18:55.382 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2024-01-03 16:19:00.362 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2024-01-03 16:19:00.365 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-182.pt 2024-01-03 16:19:00.558 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2024-01-03 16:19:00.564 INFO: Number of parameters: 5725072 2024-01-03 16:19:00.564 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0010485760000000005 maximize: False name: readouts weight_decay: 0.0 ) 2024-01-03 16:19:00.564 INFO: Using Weights and Biases for logging 2024-01-03 16:19:13.478 INFO: Using gradient clipping with tolerance=100.000 2024-01-03 16:19:13.478 INFO: Started training 2024-01-03 16:19:21.857 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.857 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.857 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.857 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.857 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.857 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.858 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.858 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.860 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.860 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.860 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.860 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.860 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.860 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.860 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.860 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.861 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:19:21.862 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-03 16:42:33.154 INFO: Epoch 182: loss=4.2812e-03, MAE_E_per_atom=21.0682 meV, MAE_F=39.5490 meV / A, MAE_stress_per_atom=0.1267 meV / A^3 2024-01-03 16:55:45.429 INFO: Epoch 183: loss=4.2732e-03, MAE_E_per_atom=21.0743 meV, MAE_F=39.5254 meV / A, MAE_stress_per_atom=0.1286 meV / A^3 2024-01-03 17:08:31.759 INFO: Epoch 184: loss=4.2727e-03, MAE_E_per_atom=21.0778 meV, MAE_F=39.4805 meV / A, MAE_stress_per_atom=0.1265 meV / A^3 2024-01-03 17:21:16.481 INFO: Epoch 185: loss=4.2782e-03, MAE_E_per_atom=21.0040 meV, MAE_F=39.6127 meV / A, MAE_stress_per_atom=0.1279 meV / A^3 2024-01-03 17:34:03.223 INFO: Epoch 186: loss=4.2618e-03, MAE_E_per_atom=20.9988 meV, MAE_F=39.4248 meV / A, MAE_stress_per_atom=0.1268 meV / A^3 2024-01-03 17:46:49.843 INFO: Epoch 187: loss=4.2699e-03, MAE_E_per_atom=20.9965 meV, MAE_F=39.5172 meV / A, MAE_stress_per_atom=0.1270 meV / A^3 2024-01-03 17:59:28.232 INFO: Epoch 188: loss=4.2492e-03, MAE_E_per_atom=20.8614 meV, MAE_F=39.4962 meV / A, MAE_stress_per_atom=0.1263 meV / A^3 2024-01-03 18:12:10.879 INFO: Epoch 189: loss=4.2547e-03, MAE_E_per_atom=21.1323 meV, MAE_F=39.5221 meV / A, MAE_stress_per_atom=0.1262 meV / A^3 2024-01-04 04:59:40.623 INFO: Process group initialized: True 2024-01-04 04:59:40.624 INFO: Processes: 1 2024-01-04 04:59:40.624 INFO: MACE version: 0.3.0 2024-01-04 04:59:40.624 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=False, wandb_project='', wandb_entity='', wandb_name='', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2024-01-04 04:59:40.624 INFO: CUDA version: 11.8, CUDA device: 0 2024-01-04 04:59:40.626 INFO: Using statistics json file 2024-01-04 04:59:40.626 INFO: Using atomic numbers from statistics file 2024-01-04 04:59:40.626 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2024-01-04 04:59:40.626 INFO: Atomic Energies not in training file, using command line argument E0s 2024-01-04 04:59:40.627 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2024-01-04 05:00:11.593 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2024-01-04 05:00:11.596 INFO: Average number of neighbors: 61.964672446250916 2024-01-04 05:00:11.596 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2024-01-04 05:00:11.596 INFO: Building model 2024-01-04 05:00:11.598 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2024-01-04 05:00:15.761 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2024-01-04 05:00:15.764 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-189.pt 2024-01-04 05:00:16.024 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2024-01-04 05:00:16.031 INFO: Number of parameters: 5725072 2024-01-04 05:00:16.031 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: readouts weight_decay: 0.0 ) 2024-01-04 05:00:16.149 INFO: Computing metrics for training, validation, and test sets 2024-01-04 05:00:16.152 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-189.pt 2024-01-04 05:00:16.248 INFO: Loaded model from epoch 189 2024-01-04 05:00:16.248 INFO: Saving model to checkpoints/04-128-L2_run-1.model 2024-01-04 05:00:16.489 INFO: Evaluating train ... 2024-01-04 11:16:53.836 INFO: Process group initialized: True 2024-01-04 11:16:53.839 INFO: Processes: 120 2024-01-04 11:16:53.839 INFO: MACE version: 0.3.0 2024-01-04 11:16:53.839 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2024-01-04 11:16:53.839 INFO: CUDA version: 11.8, CUDA device: 0 2024-01-04 11:16:53.839 INFO: Using statistics json file 2024-01-04 11:16:53.839 INFO: Using atomic numbers from statistics file 2024-01-04 11:16:53.839 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2024-01-04 11:16:53.839 INFO: Atomic Energies not in training file, using command line argument E0s 2024-01-04 11:16:53.840 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2024-01-04 11:17:26.396 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2024-01-04 11:17:26.399 INFO: Average number of neighbors: 61.964672446250916 2024-01-04 11:17:26.399 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2024-01-04 11:17:26.399 INFO: Building model 2024-01-04 11:17:26.400 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2024-01-04 11:17:31.068 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2024-01-04 11:17:31.071 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-189.pt 2024-01-04 11:17:31.336 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2024-01-04 11:17:31.342 INFO: Number of parameters: 5725072 2024-01-04 11:17:31.342 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: readouts weight_decay: 0.0 ) 2024-01-04 11:17:31.342 INFO: Using Weights and Biases for logging 2024-01-04 11:17:47.425 INFO: Using gradient clipping with tolerance=100.000 2024-01-04 11:17:47.426 INFO: Started training 2024-01-04 11:17:55.652 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.652 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.653 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.654 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:17:55.655 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-04 11:41:17.006 INFO: Epoch 189: loss=4.2668e-03, MAE_E_per_atom=21.2564 meV, MAE_F=39.5555 meV / A, MAE_stress_per_atom=0.1250 meV / A^3 2024-01-04 11:54:36.240 INFO: Epoch 190: loss=4.2532e-03, MAE_E_per_atom=20.9412 meV, MAE_F=39.5461 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2024-01-04 12:07:18.783 INFO: Epoch 191: loss=4.2641e-03, MAE_E_per_atom=20.9468 meV, MAE_F=39.4657 meV / A, MAE_stress_per_atom=0.1270 meV / A^3 2024-01-04 12:20:04.840 INFO: Epoch 192: loss=4.2460e-03, MAE_E_per_atom=20.7746 meV, MAE_F=39.5281 meV / A, MAE_stress_per_atom=0.1281 meV / A^3 2024-01-04 12:32:49.746 INFO: Epoch 193: loss=4.2778e-03, MAE_E_per_atom=20.9193 meV, MAE_F=39.4483 meV / A, MAE_stress_per_atom=0.1278 meV / A^3 2024-01-04 12:45:31.891 INFO: Epoch 194: loss=4.2328e-03, MAE_E_per_atom=20.9344 meV, MAE_F=39.2945 meV / A, MAE_stress_per_atom=0.1267 meV / A^3 2024-01-04 12:58:12.347 INFO: Epoch 195: loss=4.2331e-03, MAE_E_per_atom=20.9707 meV, MAE_F=39.3877 meV / A, MAE_stress_per_atom=0.1263 meV / A^3 2024-01-04 13:10:54.495 INFO: Epoch 196: loss=4.2702e-03, MAE_E_per_atom=20.9773 meV, MAE_F=39.4520 meV / A, MAE_stress_per_atom=0.1264 meV / A^3 2024-01-05 03:49:39.679 INFO: Process group initialized: True 2024-01-05 03:49:39.681 INFO: Processes: 120 2024-01-05 03:49:39.681 INFO: MACE version: 0.3.0 2024-01-05 03:49:39.681 INFO: Configuration: Namespace(name='04-128-L2', seed=1, log_dir='logs', model_dir='.', checkpoints_dir='checkpoints', results_dir='results', downloads_dir='downloads', device='cuda', default_dtype='float64', distributed=True, log_level='INFO', error_table='PerAtomMAE', model='ScaleShiftMACE', r_max=6.0, radial_type='bessel', num_radial_basis=10, num_cutoff_basis=5, interaction='RealAgnosticResidualInteractionBlock', interaction_first='RealAgnosticResidualInteractionBlock', max_ell=3, correlation=3, num_interactions=2, MLP_irreps='16x0e', radial_MLP='[64, 64, 64]', hidden_irreps='128x0e + 128x1o', num_channels=128, max_L=2, gate='silu', scaling='rms_forces_scaling', avg_num_neighbors=1, compute_avg_num_neighbors=True, compute_stress=True, compute_forces=True, train_file='../../dataset/mptrj-gga-ggapu-train', valid_file='../../dataset/mptrj-gga-ggapu-val', valid_fraction=0.1, test_file=None, test_dir=None, multi_processed_test=False, num_workers=16, pin_memory=True, atomic_numbers=None, mean=None, std=None, statistics_file='../../dataset/mptrj-gga-ggapu-statistics.json', E0s=None, energy_key='energy', forces_key='forces', virials_key='virials', stress_key='stress', dipole_key='dipole', charges_key='charges', loss='universal', forces_weight=10.0, swa_forces_weight=100.0, energy_weight=1.0, swa_energy_weight=1000.0, virials_weight=1.0, swa_virials_weight=10.0, stress_weight=100.0, swa_stress_weight=10.0, dipole_weight=1.0, swa_dipole_weight=1.0, config_type_weights='{"Default":1.0}', huber_delta=0.01, optimizer='adam', batch_size=16, valid_batch_size=16, lr=0.005, swa_lr=0.001, weight_decay=1e-08, amsgrad=True, scheduler='ReduceLROnPlateau', lr_factor=0.8, scheduler_patience=5, lr_scheduler_gamma=0.9993, swa=False, start_swa=None, ema=True, ema_decay=0.995, max_num_epochs=200, patience=50, eval_interval=1, keep_checkpoints=True, restart_latest=True, save_cpu=True, clip_grad=100.0, wandb=True, wandb_project='mace-universal', wandb_entity='astagroup', wandb_name='04-128-L2', wandb_log_hypers=['num_channels', 'max_L', 'correlation', 'lr', 'swa_lr', 'weight_decay', 'batch_size', 'max_num_epochs', 'start_swa', 'energy_weight', 'forces_weight']) 2024-01-05 03:49:39.682 INFO: CUDA version: 11.8, CUDA device: 0 2024-01-05 03:49:39.683 INFO: Using statistics json file 2024-01-05 03:49:39.683 INFO: Using atomic numbers from statistics file 2024-01-05 03:49:39.683 INFO: AtomicNumberTable: (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 89, 90, 91, 92, 93, 94) 2024-01-05 03:49:39.683 INFO: Atomic Energies not in training file, using command line argument E0s 2024-01-05 03:49:39.684 INFO: Atomic energies: [-3.667168021358939, -1.3320953124042916, -3.482100566595956, -4.736697230897597, -7.724935420523256, -8.405573550273285, -7.360100452662763, -7.28459863421322, -4.896490881731322, 1.3917755836700962e-12, -2.7593613569762425, -2.814047612069227, -4.846881245288104, -7.694793133351899, -6.9632957911820235, -4.672630400190884, -2.8116892814008096, -0.06259504416367478, -2.6176454856894793, -5.390461060484104, -7.8857952163517675, -10.268392986214433, -8.665147785496703, -9.233050763772013, -8.304951520770791, -7.0489865771593765, -5.577439766222147, -5.172747618813715, -3.2520726958619472, -1.2901611618726314, -3.527082192997912, -4.70845955030298, -3.9765109025623238, -3.886231055836541, -2.5184940099633986, 6.766947645687137, -2.5634958965928316, -4.938005211501922, -10.149818838085771, -11.846857579882572, -12.138896361658485, -8.791678800595722, -8.78694939675911, -7.78093221529871, -6.850021409115055, -4.891019073240479, -2.0634296773864045, -0.6395695518943755, -2.7887442084286693, -3.818604275441892, -3.587068329278862, -2.8804045971118897, -1.6355986842433357, 9.846723842807721, -2.765284507132287, -4.990956432167774, -8.933684809576345, -8.735591176647514, -8.018966025544966, -8.251491970213372, -7.591719594359237, -8.169659881166858, -13.592664636171698, -18.517523458456985, -7.647396572993602, -8.122981037851925, -7.607787319678067, -6.85029094445494, -7.8268821327130365, -3.584786591677161, -7.455406192077973, -12.796283502572146, -14.108127281277586, -9.354916969477486, -11.387537567890853, -9.621909492152557, -7.324393429417677, -5.3046964808341945, -2.380092582080244, 0.24948924158195362, -2.3239789120665026, -3.730042357127322, -3.438792347649683, -5.062878214511315, -11.02462566385297, -12.265613551943261, -13.855648206100362, -14.933092020258243, -15.282826131998245] 2024-01-05 03:50:11.896 INFO: UniversalLoss(energy_weight=1.000, forces_weight=10.000, stress_weight=100.000) 2024-01-05 03:50:11.898 INFO: Average number of neighbors: 61.964672446250916 2024-01-05 03:50:11.898 INFO: Selected the following outputs: {'energy': True, 'forces': True, 'virials': False, 'stress': True, 'dipoles': False} 2024-01-05 03:50:11.898 INFO: Building model 2024-01-05 03:50:11.899 INFO: Hidden irreps: 128x0e+128x1o+128x2e 2024-01-05 03:50:16.511 WARNING: No SWA checkpoint found, while SWA is enabled. Compare the swa_start parameter and the latest checkpoint. 2024-01-05 03:50:16.515 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-196.pt 2024-01-05 03:50:16.782 INFO: ScaleShiftMACE( (node_embedding): LinearNodeEmbeddingBlock( (linear): Linear(89x0e -> 128x0e | 11392 weights) ) (radial_embedding): RadialEmbeddingBlock( (bessel_fn): BesselBasis(r_max=6.0, num_basis=10, trainable=False) (cutoff_fn): PolynomialCutoff(p=5.0, r_max=6.0) ) (spherical_harmonics): SphericalHarmonics() (atomic_energies_fn): AtomicEnergiesBlock(energies=[-3.6672, -1.3321, -3.4821, -4.7367, -7.7249, -8.4056, -7.3601, -7.2846, -4.8965, 0.0000, -2.7594, -2.8140, -4.8469, -7.6948, -6.9633, -4.6726, -2.8117, -0.0626, -2.6176, -5.3905, -7.8858, -10.2684, -8.6651, -9.2331, -8.3050, -7.0490, -5.5774, -5.1727, -3.2521, -1.2902, -3.5271, -4.7085, -3.9765, -3.8862, -2.5185, 6.7669, -2.5635, -4.9380, -10.1498, -11.8469, -12.1389, -8.7917, -8.7869, -7.7809, -6.8500, -4.8910, -2.0634, -0.6396, -2.7887, -3.8186, -3.5871, -2.8804, -1.6356, 9.8467, -2.7653, -4.9910, -8.9337, -8.7356, -8.0190, -8.2515, -7.5917, -8.1697, -13.5927, -18.5175, -7.6474, -8.1230, -7.6078, -6.8503, -7.8269, -3.5848, -7.4554, -12.7963, -14.1081, -9.3549, -11.3875, -9.6219, -7.3244, -5.3047, -2.3801, 0.2495, -2.3240, -3.7300, -3.4388, -5.0629, -11.0246, -12.2656, -13.8556, -14.9331, -15.2828]) (interactions): ModuleList( (0): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e -> 128x0e | 16384 weights) (conv_tp): TensorProduct(128x0e x 1x0e+1x1o+1x2e+1x3o -> 128x0e+128x1o+128x2e+128x3o | 512 paths | 512 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 512] (linear): Linear(128x0e+128x1o+128x2e+128x3o -> 128x0e+128x1o+128x2e+128x3o | 65536 weights) (skip_tp): FullyConnectedTensorProduct(128x0e x 89x0e -> 128x0e+128x1o+128x2e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) (1): RealAgnosticResidualInteractionBlock( (linear_up): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) (conv_tp): TensorProduct(128x0e+128x1o+128x2e x 1x0e+1x1o+1x2e+1x3o -> 384x0e+640x1o+640x2e+512x3o | 2176 paths | 2176 weights) (conv_tp_weights): FullyConnectedNet[10, 64, 64, 64, 2176] (linear): Linear(384x0e+640x1o+640x2e+512x3o -> 128x0e+128x1o+128x2e+128x3o | 278528 weights) (skip_tp): FullyConnectedTensorProduct(128x0e+128x1o+128x2e x 89x0e -> 128x0e | 1458176 paths | 1458176 weights) (reshape): reshape_irreps() ) ) (products): ModuleList( (0): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (1): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x6x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) (2): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x7x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e+128x1o+128x2e -> 128x0e+128x1o+128x2e | 49152 weights) ) (1): EquivariantProductBasisBlock( (symmetric_contractions): SymmetricContraction( (contractions): ModuleList( (0): Contraction( (contractions_weighting): ModuleList( (0-1): 2 x GraphModule() ) (contractions_features): ModuleList( (0-1): 2 x GraphModule() ) (weights): ParameterList( (0): Parameter containing: [torch.float64 of size 89x4x128 (GPU 0)] (1): Parameter containing: [torch.float64 of size 89x1x128 (GPU 0)] ) (graph_opt_main): GraphModule() ) ) ) (linear): Linear(128x0e -> 128x0e | 16384 weights) ) ) (readouts): ModuleList( (0): LinearReadoutBlock( (linear): Linear(128x0e+128x1o+128x2e -> 1x0e | 128 weights) ) (1): NonLinearReadoutBlock( (linear_1): Linear(128x0e -> 16x0e | 2048 weights) (non_linearity): Activation [x] (16x0e -> 16x0e) (linear_2): Linear(16x0e -> 1x0e | 16 weights) ) ) (scale_shift): ScaleShiftBlock(scale=0.804154, shift=0.164097) ) 2024-01-05 03:50:16.788 INFO: Number of parameters: 5725072 2024-01-05 03:50:16.788 INFO: Optimizer: Adam ( Parameter Group 0 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: embedding weight_decay: 0.0 Parameter Group 1 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: interactions_decay weight_decay: 1e-08 Parameter Group 2 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: interactions_no_decay weight_decay: 0.0 Parameter Group 3 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: products weight_decay: 1e-08 Parameter Group 4 amsgrad: True betas: (0.9, 0.999) capturable: False differentiable: False eps: 1e-08 foreach: None fused: None lr: 0.0008388608000000005 maximize: False name: readouts weight_decay: 0.0 ) 2024-01-05 03:50:16.788 INFO: Using Weights and Biases for logging 2024-01-05 03:50:31.958 INFO: Using gradient clipping with tolerance=100.000 2024-01-05 03:50:31.958 INFO: Started training 2024-01-05 03:50:39.924 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.924 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.924 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.924 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.929 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.930 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.931 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 03:50:39.932 INFO: Reducer buckets have been rebuilt in this iteration. 2024-01-05 04:13:50.625 INFO: Epoch 196: loss=4.2845e-03, MAE_E_per_atom=20.9517 meV, MAE_F=39.5308 meV / A, MAE_stress_per_atom=0.1258 meV / A^3 2024-01-05 04:26:54.125 INFO: Epoch 197: loss=4.2164e-03, MAE_E_per_atom=20.8925 meV, MAE_F=39.4606 meV / A, MAE_stress_per_atom=0.1265 meV / A^3 2024-01-05 04:39:40.862 INFO: Epoch 198: loss=4.2461e-03, MAE_E_per_atom=20.8929 meV, MAE_F=39.4227 meV / A, MAE_stress_per_atom=0.1272 meV / A^3 2024-01-05 04:52:26.220 INFO: Epoch 199: loss=4.2518e-03, MAE_E_per_atom=20.9068 meV, MAE_F=39.4856 meV / A, MAE_stress_per_atom=0.1272 meV / A^3 2024-01-05 04:52:26.527 INFO: Training complete 2024-01-05 04:52:26.528 INFO: Computing metrics for training, validation, and test sets 2024-01-05 04:52:26.534 INFO: Loading checkpoint: checkpoints/04-128-L2_run-1_epoch-199.pt 2024-01-05 04:52:26.946 INFO: Loaded model from epoch 199 2024-01-05 04:52:26.947 INFO: Evaluating train ... 2024-01-05 04:56:35.605 INFO: Evaluating valid ... 2024-01-05 04:56:37.622 INFO: +-------------+--------------------+-----------------+------------------+ | config_type | MAE E / meV / atom | MAE F / meV / A | relative F MAE % | +-------------+--------------------+-----------------+------------------+ | train | 21.2 | 38.4 | 24.33 | | valid | 20.9 | 39.9 | 30.99 | +-------------+--------------------+-----------------+------------------+ 2024-01-05 04:56:37.622 INFO: Saving model to checkpoints/04-128-L2_run-1.model 2024-01-05 04:56:37.925 INFO: Done