Hugging Face's logo

impresso-project
/
OCR-quality-assessment-unigram

Model card Files Community
Simon Clematide commited on
Commit
7ca08a0
·
1 Parent(s): 5f39ead

Add Bloom Filter files and logs for German and French datasets

Browse files
Files changed (4) hide show
  1. ocrqa-wp_v1.0.5-de.bloom +3 -0
  2. ocrqa-wp_v1.0.5-de.bloom.log +31 -0
  3. ocrqa-wp_v1.0.5-fr.bloom +3 -0
  4. ocrqa-wp_v1.0.5-fr.bloom.log +33 -0
ocrqa-wp_v1.0.5-de.bloom ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:645eeb974398aac1c5662789e0bfe21abc916f903777befaec9be6da075535c2
3
+ size 10059530
ocrqa-wp_v1.0.5-de.bloom.log ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2025-02-10 09:54:25,418 ocrqa_create_bloom_filter.py:425 INFO: Namespace(input_files=['lex/de/realwords.rw.txt.bz2', 'lex/de/dewiki.unigram.freq.tsv.bz2', 'lex/de/nonwords.nw.txt.bz2'], bloom_path='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-de.bloom', fp_probability=1e-05, log_level='INFO', log_file='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-de.bloom.log', config=None, min_frequency=2, single_char_min_frequency=20, diagnose_bloom=True)
2
+ 2025-02-10 09:54:25,418 ocrqa_create_bloom_filter.py:226 INFO: Starting Bloom Filter creation...
3
+ 2025-02-10 09:54:25,418 ocrqa_create_bloom_filter.py:178 INFO: Processing nonword file: lex/de/nonwords.nw.txt.bz2
4
+ 2025-02-10 09:54:25,418 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['negierung']
5
+ 2025-02-10 09:54:25,418 ocrqa_create_bloom_filter.py:190 INFO: Excluded 1 words that should never be added
6
+ 2025-02-10 09:54:25,418 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/de/realwords.rw.txt.bz2
7
+ 2025-02-10 09:54:25,418 ocrqa_create_bloom_filter.py:135 INFO: Processing frequency file: lex/de/dewiki.unigram.freq.tsv.bz2
8
+ 2025-02-10 09:54:37,205 ocrqa_create_bloom_filter.py:240 INFO: low_freq_excluded before removing parts from high-frequency words: 3780824
9
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:248 INFO: low_freq_excluded after removing parts from high-frequency words: 3288865
10
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:252 INFO: Lexical processing complete.
11
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - nonwords_read: 1
12
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - nonwords_count: 1
13
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - realwords_read: 8
14
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - realwords_accepted: 8
15
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - realwords_nonwords_filtered: 0
16
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - freq_words_read: 9199714
17
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - words_accepted: 4143119
18
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - subwords_accepted: 5029202
19
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - subwords_filtered: 1
20
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - low_freq_excluded: 3288865
21
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - single_char_words_filtered: 6719
22
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:256 INFO: - freq_words_filtered: 5049876
23
+ 2025-02-10 09:54:37,366 ocrqa_create_bloom_filter.py:259 INFO: Estimated word count: 3357976
24
+ 2025-02-10 09:54:38,245 ocrqa_create_bloom_filter.py:263 INFO: Bloom Filter created and saved to build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-de.bloom
25
+ 2025-02-10 09:54:38,860 ocrqa_create_bloom_filter.py:285 INFO: Diagnosis Results:
26
+ 2025-02-10 09:54:38,860 ocrqa_create_bloom_filter.py:286 INFO: - Excluded words in bloom filter: 0
27
+ 2025-02-10 09:54:38,860 ocrqa_create_bloom_filter.py:287 INFO: - Known words not in bloom filter: 0
28
+ 2025-02-10 09:54:39,386 ocrqa_create_bloom_filter.py:294 INFO: - Low-frequency words in bloom filter: 33
29
+ 2025-02-10 09:54:39,386 ocrqa_create_bloom_filter.py:300 INFO: - Proportion of excluded words in bloom filter: 0.00000000
30
+ 2025-02-10 09:54:39,386 ocrqa_create_bloom_filter.py:306 INFO: - Proportion of known words not in bloom filter: 0.00000000
31
+ 2025-02-10 09:54:39,386 ocrqa_create_bloom_filter.py:314 INFO: - Proportion of low-frequency words in bloom filter: 0.00001003
ocrqa-wp_v1.0.5-fr.bloom ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a92ff06d9bde64f36e106a4177737becfd4f39bbed693bddb38d74d481b55dd0
3
+ size 5810022
ocrqa-wp_v1.0.5-fr.bloom.log ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:425 INFO: Namespace(input_files=['lex/fr/realwords.rw.txt.bz2', 'lex/fr/morph.rw.txt.bz2', 'lex/fr/single_char.rw.txt.bz2', 'lex/fr/frwiki.unigram.freq.tsv.bz2', 'lex/fr/nonwords.nw.txt.bz2'], bloom_path='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-fr.bloom', fp_probability=1e-05, log_level='INFO', log_file='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-fr.bloom.log', config=None, min_frequency=2, single_char_min_frequency=20, diagnose_bloom=True)
2
+ 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:226 INFO: Starting Bloom Filter creation...
3
+ 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:178 INFO: Processing nonword file: lex/fr/nonwords.nw.txt.bz2
4
+ 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:187 INFO: Nonword tokens: ['gazelle']
5
+ 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:190 INFO: Excluded 1 words that should never be added
6
+ 2025-02-10 09:54:39,729 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/realwords.rw.txt.bz2
7
+ 2025-02-10 09:54:39,730 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/morph.rw.txt.bz2
8
+ 2025-02-10 09:54:40,372 ocrqa_create_bloom_filter.py:196 INFO: Processing real-word file: lex/fr/single_char.rw.txt.bz2
9
+ 2025-02-10 09:54:40,373 ocrqa_create_bloom_filter.py:135 INFO: Processing frequency file: lex/fr/frwiki.unigram.freq.tsv.bz2
10
+ 2025-02-10 09:54:45,072 ocrqa_create_bloom_filter.py:240 INFO: low_freq_excluded before removing parts from high-frequency words: 1552240
11
+ 2025-02-10 09:54:45,172 ocrqa_create_bloom_filter.py:248 INFO: low_freq_excluded after removing parts from high-frequency words: 1228188
12
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:252 INFO: Lexical processing complete.
13
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - nonwords_read: 1
14
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - nonwords_count: 1
15
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - realwords_read: 723250
16
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - realwords_accepted: 736917
17
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - realwords_nonwords_filtered: 1
18
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - freq_words_read: 4047539
19
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - words_accepted: 1992185
20
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - subwords_accepted: 2383447
21
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - subwords_filtered: 9
22
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - low_freq_excluded: 1228188
23
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - single_char_words_filtered: 6597
24
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:256 INFO: - freq_words_filtered: 2048757
25
+ 2025-02-10 09:54:45,173 ocrqa_create_bloom_filter.py:259 INFO: Estimated word count: 1939265
26
+ 2025-02-10 09:54:45,657 ocrqa_create_bloom_filter.py:263 INFO: Bloom Filter created and saved to build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-fr.bloom
27
+ 2025-02-10 09:54:45,996 ocrqa_create_bloom_filter.py:285 INFO: Diagnosis Results:
28
+ 2025-02-10 09:54:45,996 ocrqa_create_bloom_filter.py:286 INFO: - Excluded words in bloom filter: 0
29
+ 2025-02-10 09:54:45,996 ocrqa_create_bloom_filter.py:287 INFO: - Known words not in bloom filter: 0
30
+ 2025-02-10 09:54:46,146 ocrqa_create_bloom_filter.py:294 INFO: - Low-frequency words in bloom filter: 8
31
+ 2025-02-10 09:54:46,146 ocrqa_create_bloom_filter.py:300 INFO: - Proportion of excluded words in bloom filter: 0.00000000
32
+ 2025-02-10 09:54:46,146 ocrqa_create_bloom_filter.py:306 INFO: - Proportion of known words not in bloom filter: 0.00000000
33
+ 2025-02-10 09:54:46,146 ocrqa_create_bloom_filter.py:314 INFO: - Proportion of low-frequency words in bloom filter: 0.00000651