Davide Caroselli winglian commited on
Commit
6a5a725
·
unverified ·
1 Parent(s): f5febc7

Fix: ensure correct handling of `val_set_size` as `float` or `int` (#1655)

Browse files

* Fix: ensure correct handling of val_set_size as float or int

* chore: lint

---------

Co-authored-by: Wing Lian <[email protected]>

Files changed (1) hide show
  1. src/axolotl/utils/data/sft.py +3 -1
src/axolotl/utils/data/sft.py CHANGED
@@ -494,7 +494,9 @@ def load_prepare_datasets(
494
  test_fingerprint = md5(to_hash_test)
495
 
496
  dataset = dataset.train_test_split(
497
- test_size=cfg.val_set_size,
 
 
498
  shuffle=False,
499
  seed=cfg.seed or 42,
500
  train_new_fingerprint=train_fingerprint,
 
494
  test_fingerprint = md5(to_hash_test)
495
 
496
  dataset = dataset.train_test_split(
497
+ test_size=int(cfg.val_set_size)
498
+ if cfg.val_set_size == int(cfg.val_set_size)
499
+ else cfg.val_set_size,
500
  shuffle=False,
501
  seed=cfg.seed or 42,
502
  train_new_fingerprint=train_fingerprint,