Hugging Face's logo

Spaces:
saeedabc
/
llm-text-tiling-demo
Running

App Files Community
saeedabc commited on
Commit
57944b5
·
1 Parent(s): 13ecc63

Add punkt_tab

Browse files
Files changed (1) hide show
  1. util.py +1 -0
util.py CHANGED
@@ -2,6 +2,7 @@
2
  import nltk
3
 
4
  nltk.download('punkt')
 
5
 
6
  def nltk_sent_tokenize(texts: list[str]):
7
  return (sent for text in texts for sent in nltk.sent_tokenize(text))
 
2
  import nltk
3
 
4
  nltk.download('punkt')
5
+ nltk.download('punkt_tab')
6
 
7
  def nltk_sent_tokenize(texts: list[str]):
8
  return (sent for text in texts for sent in nltk.sent_tokenize(text))