Simon Clematide
commited on
Commit
·
6a465e0
1
Parent(s):
d6e2ffe
Update Bloom Filter binary and log files for version 1.0.5 with new data and apostrophe handling of old spelling Luxembourgish
Browse files- ocrqa-wp_v1.0.5-lb.bloom +2 -2
- ocrqa-wp_v1.0.5-lb.bloom.log +45 -45
ocrqa-wp_v1.0.5-lb.bloom
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6475a1aca81db756cbbc0041bf083a184f6b982e97c7fbbd86db876c00e1ba1a
|
| 3 |
+
size 856612
|
ocrqa-wp_v1.0.5-lb.bloom.log
CHANGED
|
@@ -1,45 +1,45 @@
|
|
| 1 |
-
2025-02-
|
| 2 |
-
2025-02-
|
| 3 |
-
2025-02-
|
| 4 |
-
2025-02-
|
| 5 |
-
2025-02-
|
| 6 |
-
2025-02-
|
| 7 |
-
2025-02-
|
| 8 |
-
2025-02-
|
| 9 |
-
2025-02-
|
| 10 |
-
2025-02-
|
| 11 |
-
2025-02-
|
| 12 |
-
2025-02-
|
| 13 |
-
2025-02-
|
| 14 |
-
2025-02-
|
| 15 |
-
2025-02-
|
| 16 |
-
2025-02-
|
| 17 |
-
2025-02-
|
| 18 |
-
2025-02-
|
| 19 |
-
2025-02-
|
| 20 |
-
2025-02-
|
| 21 |
-
2025-02-
|
| 22 |
-
2025-02-
|
| 23 |
-
2025-02-
|
| 24 |
-
2025-02-
|
| 25 |
-
2025-02-
|
| 26 |
-
2025-02-
|
| 27 |
-
2025-02-
|
| 28 |
-
2025-02-
|
| 29 |
-
2025-02-
|
| 30 |
-
2025-02-
|
| 31 |
-
2025-02-
|
| 32 |
-
2025-02-
|
| 33 |
-
2025-02-
|
| 34 |
-
2025-02-
|
| 35 |
-
2025-02-
|
| 36 |
-
2025-02-
|
| 37 |
-
2025-02-
|
| 38 |
-
2025-02-
|
| 39 |
-
2025-02-
|
| 40 |
-
2025-02-
|
| 41 |
-
2025-02-
|
| 42 |
-
2025-02-
|
| 43 |
-
2025-02-
|
| 44 |
-
2025-02-
|
| 45 |
-
2025-02-
|
|
|
|
| 1 |
+
2025-02-28 23:52:03,275 ocrqa_create_bloom_filter.py:121 INFO: Git version: 50f305d
|
| 2 |
+
2025-02-28 23:52:03,275 ocrqa_create_bloom_filter.py:434 INFO: Namespace(input_files=['lex/lb/lbwiki.unigram.freq.tsv.bz2', 'lex/lb/wortschatz.freq.tsv.bz2', 'lex/lb/modern_spelling.rw.txt', 'lex/lb/old_spelling.rw.txt', 'lex/lb/ocr_errors.nw.txt', 'lex/lb/realword_ocr_errors.nw.txt'], bloom_path='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-lb.bloom', fp_probability=1e-05, log_level='INFO', log_file='build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-lb.bloom.log', config=None, min_frequency=2, single_char_min_frequency=20, diagnose_bloom=True, git_version='50f305d')
|
| 3 |
+
2025-02-28 23:52:03,275 ocrqa_create_bloom_filter.py:229 INFO: Starting Bloom Filter creation...
|
| 4 |
+
2025-02-28 23:52:03,275 ocrqa_create_bloom_filter.py:181 INFO: Processing nonword file: lex/lb/ocr_errors.nw.txt
|
| 5 |
+
2025-02-28 23:52:03,275 ocrqa_create_bloom_filter.py:190 INFO: Nonword tokens: ['ail']
|
| 6 |
+
2025-02-28 23:52:03,275 ocrqa_create_bloom_filter.py:190 INFO: Nonword tokens: ['dalt']
|
| 7 |
+
2025-02-28 23:52:03,275 ocrqa_create_bloom_filter.py:190 INFO: Nonword tokens: ['erl']
|
| 8 |
+
2025-02-28 23:52:03,275 ocrqa_create_bloom_filter.py:190 INFO: Nonword tokens: ['liât']
|
| 9 |
+
2025-02-28 23:52:03,276 ocrqa_create_bloom_filter.py:190 INFO: Nonword tokens: ['liuet']
|
| 10 |
+
2025-02-28 23:52:03,276 ocrqa_create_bloom_filter.py:190 INFO: Nonword tokens: ['secli']
|
| 11 |
+
2025-02-28 23:52:03,276 ocrqa_create_bloom_filter.py:190 INFO: Nonword tokens: ['zc']
|
| 12 |
+
2025-02-28 23:52:03,276 ocrqa_create_bloom_filter.py:193 INFO: Excluded 7 words that should never be added
|
| 13 |
+
2025-02-28 23:52:03,276 ocrqa_create_bloom_filter.py:181 INFO: Processing nonword file: lex/lb/realword_ocr_errors.nw.txt
|
| 14 |
+
2025-02-28 23:52:03,276 ocrqa_create_bloom_filter.py:193 INFO: Excluded 7 words that should never be added
|
| 15 |
+
2025-02-28 23:52:03,276 ocrqa_create_bloom_filter.py:138 INFO: Processing frequency file: lex/lb/lbwiki.unigram.freq.tsv.bz2
|
| 16 |
+
2025-02-28 23:52:03,289 ocrqa_create_bloom_filter.py:163 WARNING: Skipping invalid line in frequency file lex/lb/lbwiki.unigram.freq.tsv.bz2: 446
|
| 17 |
+
2025-02-28 23:52:03,591 ocrqa_create_bloom_filter.py:138 INFO: Processing frequency file: lex/lb/wortschatz.freq.tsv.bz2
|
| 18 |
+
2025-02-28 23:52:03,814 ocrqa_create_bloom_filter.py:163 WARNING: Skipping invalid line in frequency file lex/lb/wortschatz.freq.tsv.bz2:
|
| 19 |
+
2025-02-28 23:52:03,814 ocrqa_create_bloom_filter.py:199 INFO: Processing real-word file: lex/lb/modern_spelling.rw.txt
|
| 20 |
+
2025-02-28 23:52:03,925 ocrqa_create_bloom_filter.py:199 INFO: Processing real-word file: lex/lb/old_spelling.rw.txt
|
| 21 |
+
2025-02-28 23:52:03,925 ocrqa_create_bloom_filter.py:243 INFO: low_freq_excluded before removing parts from high-frequency words: 156199
|
| 22 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:251 INFO: low_freq_excluded after removing parts from high-frequency words: 113360
|
| 23 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:255 INFO: Lexical processing complete.
|
| 24 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - nonwords_read: 7
|
| 25 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - nonwords_count: 7
|
| 26 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - freq_words_read: 487375
|
| 27 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - words_accepted: 314693
|
| 28 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - subwords_accepted: 366393
|
| 29 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - subwords_filtered: 0
|
| 30 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - low_freq_excluded: 113360
|
| 31 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - invalid_lines_filtered: 2
|
| 32 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - single_char_words_filtered: 334
|
| 33 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - freq_words_filtered: 172346
|
| 34 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - realwords_read: 152110
|
| 35 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - realwords_accepted: 154531
|
| 36 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:259 INFO: - realwords_nonwords_filtered: 0
|
| 37 |
+
2025-02-28 23:52:03,942 ocrqa_create_bloom_filter.py:262 INFO: Estimated word count: 285555
|
| 38 |
+
2025-02-28 23:52:04,014 ocrqa_create_bloom_filter.py:266 INFO: Bloom Filter created and saved to build.d/fp_prob_0.00001/ocrqa-wp_v1.0.5-lb.bloom
|
| 39 |
+
2025-02-28 23:52:04,066 ocrqa_create_bloom_filter.py:288 INFO: Diagnosis Results:
|
| 40 |
+
2025-02-28 23:52:04,066 ocrqa_create_bloom_filter.py:289 INFO: - Excluded words in bloom filter: 0
|
| 41 |
+
2025-02-28 23:52:04,066 ocrqa_create_bloom_filter.py:290 INFO: - Known words not in bloom filter: 0
|
| 42 |
+
2025-02-28 23:52:04,079 ocrqa_create_bloom_filter.py:297 INFO: - Low-frequency words in bloom filter: 1
|
| 43 |
+
2025-02-28 23:52:04,079 ocrqa_create_bloom_filter.py:303 INFO: - Proportion of excluded words in bloom filter: 0.00000000
|
| 44 |
+
2025-02-28 23:52:04,079 ocrqa_create_bloom_filter.py:309 INFO: - Proportion of known words not in bloom filter: 0.00000000
|
| 45 |
+
2025-02-28 23:52:04,079 ocrqa_create_bloom_filter.py:317 INFO: - Proportion of low-frequency words in bloom filter: 0.00000882
|