repo.txt

Repository Documentation
This document provides a comprehensive overview of the repository's structure and contents.
The first section, titled 'Directory/File Tree', displays the repository's hierarchy in a tree format.
In this section, directories and files are listed using tree branches to indicate their structure and relationships.
Following the tree representation, the 'File Content' section details the contents of each file in the repository.
Each file's content is introduced with a '[File Begins]' marker followed by the file's relative path,
and the content is displayed verbatim. The end of each file's content is marked with a '[File Ends]' marker.
This format ensures a clear and orderly presentation of both the structure and the detailed contents of the repository.

Directory/File Tree Begins -->

/
├── README.md
├── app.py
├── app.py.bak
├── bible.py
├── database-structure.txt
├── gematria.py
├── hindu.py
├── populate_translations.py
├── quran.py
├── requirements-all.txt
├── requirements.txt
├── texts
│   ├── bible
│   ├── mahabharata
│   ├── quran
│   ├── rigveda
│   ├── torah
│   └── tripitaka
├── torah.py
├── translation_utils.py
├── tripitaka.py
├── util.py
└── utils.py

<-- Directory/File Tree Ends

File Content Begin -->
[File Begins] README.md
---
title: Book of Souls - Search your name+day (journal) oracle with ELS over Torah, Bible, Quran, Rigveda, Tripitaka
emoji: 📊
colorFrom: green
colorTo: pink
sdk: gradio
sdk_version: 4.39.0
app_file: app.py
pinned: false
---

This application searches for equidistant letter sequences (ELS) in the Torah, Bible, Quran, and Rigveda. It also integrates a network search functionality to find related phrases based on gematria.

**Inputs:**

* **Target Language for Translation:**  The language to translate the results into.
* **Date to investigate (optional):** A date to include in the gematria calculation.
* **Language of the person/topic (optional) (Date Word Language):** The language to use for converting the date to words.
* **Name and/or Topic (required):** The text to calculate the gematria for.
* **Jump Width (Steps) (optional) for ELS:** The step size for the ELS search.
* **Round (1) / Round (2) (optional):** The number of rounds for the ELS search (positive or negative).
* **Include Torah / Include Bible / Include Quran / Include Rigveda:** Checkboxes to select which texts to search.
* **Strip Spaces from Books / Strip Text in Braces from Books / Strip Diacritics from Books:** Options for text preprocessing.

**Outputs:**

* **ELS Results:** A dataframe containing the ELS search results.
* **Most Frequent Phrase in Network Search:** The most frequent phrase found in the network search.
* **JSON Output:**  A JSON representation of the search results.


**How to Use:**

1. Enter the name or topic you want to investigate.
2. Optionally, select a date and the language for its representation.
3. Set the jump width (steps) and rounds for the ELS search.
4. Choose which texts to include in the search.
5. Configure text preprocessing options as needed.
6. Click "Search with ELS".
7. The results will be displayed in the output sections.  You can copy the JSON output using the provided button.


**Network Search:**

The network search functionality uses the calculated gematria of the ELS results to search a database for phrases with the same gematria.  It displays the most frequent matching phrase.  If no exact match is found, it attempts to find the closest match based on similarity and word count difference.
[File Ends] README.md

[File Begins] app.py
#TODO: Quran results have numbers

import logging
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

import gradio as gr
import torah
import bible
import quran
import hindu
import tripitaka
from utils import number_to_ordinal_word, custom_normalize, date_to_words, translate_date_to_words
from gematria import calculate_gematria, strip_diacritics

import pandas as pd
from deep_translator import GoogleTranslator
from gradio_calendar import Calendar
from datetime import datetime, timedelta
import math
import json
import re
import sqlite3
from collections import defaultdict
from typing import List, Tuple
import rich
from fuzzywuzzy import fuzz
import calendar
import translation_utils
import hashlib

translation_utils.create_translation_table()

# Create a translator instance *once* globally
translator = GoogleTranslator(source='auto', target='auto')
LANGUAGES_SUPPORTED = translator.get_supported_languages(as_dict=True)  # Corrected dictionary name

LANGUAGE_CODE_MAP = LANGUAGES_SUPPORTED # Use deep_translator's mapping directly

# --- Constants ---
DATABASE_FILE = 'gematria.db'
MAX_PHRASE_LENGTH_LIMIT = 20

ELS_CACHE_DB = "els_cache.db"
DATABASE_TIMEOUT = 60

# --- Database Initialization ---
def initialize_database():
    global conn
    conn = sqlite3.connect(DATABASE_FILE)
    cursor = conn.cursor()
    cursor.execute('''
    CREATE TABLE IF NOT EXISTS results (
        gematria_sum INTEGER,
        words TEXT,
        translation TEXT,
        book TEXT,
        chapter INTEGER,
        verse INTEGER,
        phrase_length INTEGER,
        word_position TEXT, 
        PRIMARY KEY (gematria_sum, words, book, chapter, verse, word_position)
    )
    ''')
    cursor.execute('''
    CREATE INDEX IF NOT EXISTS idx_results_gematria
    ON results (gematria_sum)
    ''')
    cursor.execute('''
    CREATE TABLE IF NOT EXISTS processed_books (
        book TEXT PRIMARY KEY,
        max_phrase_length INTEGER
    )
    ''')
    conn.commit()

# --- Initialize Database ---
initialize_database()

# --- ELS Cache Functions ---
def create_els_cache_table():
    with sqlite3.connect(ELS_CACHE_DB) as conn:
        conn.execute('''
            CREATE TABLE IF NOT EXISTS els_cache (
                query_hash TEXT PRIMARY KEY,
                results TEXT
            )
        ''')

def get_query_hash(func, *args, **kwargs):
    key = (func.__name__, args, tuple(sorted(kwargs.items())))
    return hashlib.sha256(json.dumps(key).encode()).hexdigest()


def cached_process_json_files(func, *args, **kwargs):
    query_hash = get_query_hash(func, *args, **kwargs)

    try:
        with sqlite3.connect(ELS_CACHE_DB, timeout=DATABASE_TIMEOUT) as conn:
            cursor = conn.cursor()
            cursor.execute("SELECT results FROM els_cache WHERE query_hash = ?", (query_hash,))
            result = cursor.fetchone()
            if result:
                logger.info(f"Cache hit for query: {query_hash}")
                return json.loads(result[0])
    except sqlite3.Error as e:
        logger.error(f"Database error checking cache: {e}")

    logger.info(f"Cache miss for query: {query_hash}")
    results = func(*args, **kwargs)

    try:
        with sqlite3.connect(ELS_CACHE_DB, timeout=DATABASE_TIMEOUT) as conn:
            cursor = conn.cursor()
            cursor.execute("INSERT INTO els_cache (query_hash, results) VALUES (?, ?)", (query_hash, json.dumps(results)))
            conn.commit()
    except sqlite3.Error as e:
        logger.error(f"Database error caching results: {e}")

    return results

# --- Helper Functions (from Network app.py) ---
def flatten_text(text: List) -> str:
    if isinstance(text, list):
        return " ".join(flatten_text(item) if isinstance(item, list) else item for item in text)
    return text

def search_gematria_in_db(gematria_sum: int, max_words: int) -> List[Tuple[str, str, int, int, int, str]]:
    global conn
    with sqlite3.connect(DATABASE_FILE) as conn:
        cursor = conn.cursor()
        cursor.execute('''
        SELECT words, book, chapter, verse, phrase_length, word_position 
        FROM results 
        WHERE gematria_sum = ? AND phrase_length <= ?
        ''', (gematria_sum, max_words))
        results = cursor.fetchall()
    return results

def get_most_frequent_phrase(results):
    phrase_counts = defaultdict(int)
    for words, book, chapter, verse, phrase_length, word_position in results:
        phrase_counts[words] += 1
    most_frequent_phrase = max(phrase_counts, key=phrase_counts.get) if phrase_counts else None  # Handle empty results
    return most_frequent_phrase

# --- Functions from BOS app.py ---
def create_language_dropdown(label, default_value='English', show_label=True): # Default value must be in LANGUAGE_CODE_MAP
    return gr.Dropdown(
        choices=list(LANGUAGE_CODE_MAP.keys()),  # Correct choices
        label=label,
        value=default_value,
        show_label=show_label
    )

def calculate_gematria_sum(text, date_words):
    if text or date_words:
        combined_input = f"{text} {date_words}"
        logger.info(f"searching for input: {combined_input}")
        numbers = re.findall(r'\d+', combined_input)
        text_without_numbers = re.sub(r'\d+', '', combined_input)
        number_sum = sum(int(number) for number in numbers)
        text_gematria = calculate_gematria(strip_diacritics(text_without_numbers))
        total_sum = text_gematria + number_sum
        return total_sum
    else:
        return None

def perform_els_search(step, rounds_combination, tlang, strip_spaces, strip_in_braces, strip_diacritics_chk, include_torah, include_bible, include_quran, include_hindu, include_tripitaka):
    if step == 0 or rounds_combination == "0,0":
        return None

    results = {}
    length = 0

    selected_language_long = tlang  # From the Gradio dropdown (long form)
    tlang = LANGUAGES_SUPPORTED.get(selected_language_long) #Get the short code.
    if tlang is None: # Handle unsupported languages
        tlang = "en"
        logger.warning(f"Unsupported language selected: {selected_language_long}. Defaulting to English (en).")

    if include_torah:
        logger.debug(f"Arguments for Torah: {(1, 39, step, rounds_combination, length, tlang, strip_spaces, strip_in_braces, strip_diacritics_chk)}")
        results["Torah"] = cached_process_json_files(torah.process_json_files, 1, 39, step, rounds_combination, length, tlang, strip_spaces, strip_in_braces, strip_diacritics_chk)
    else:
        results["Torah"] = []

    if include_bible:
        results["Bible"] = cached_process_json_files(bible.process_json_files, 40, 66, step, rounds_combination, length, tlang, strip_spaces, strip_in_braces, strip_diacritics_chk)
    else:
        results["Bible"] = []

    if include_quran:
        results["Quran"] = cached_process_json_files(quran.process_json_files, 1, 114, step, rounds_combination, length, tlang, strip_spaces, strip_in_braces, strip_diacritics_chk)
    else:
        results["Quran"] = []

    if include_hindu:
        results["Rig Veda"] = cached_process_json_files(hindu.process_json_files, 1, 10, step, rounds_combination, length, tlang, False, strip_in_braces, strip_diacritics_chk)
    else:
        results["Rig Veda"] = []

    if include_tripitaka:
        results["Tripitaka"] = cached_process_json_files(tripitaka.process_json_files, 1, 52, step, rounds_combination, length, tlang, strip_spaces, strip_in_braces, strip_diacritics_chk)
    else:
        results["Tripitaka"] = []

    return results


def add_24h_projection(results_dict): #Now takes a dictionary of results
    for book_name, results in results_dict.items():  # Iterate per book
        num_results = len(results)
        if num_results > 0:
            time_interval = timedelta(minutes=24 * 60 / num_results)
            current_time = datetime.min.time()
            for i in range(num_results):
                next_time = (datetime.combine(datetime.min, current_time) + time_interval).time()
                time_range_str = f"{current_time.strftime('%H:%M')}-{next_time.strftime('%H:%M')}"
                results[i]['24h Projection'] = time_range_str
                current_time = next_time
    return results_dict


def add_monthly_projection(results_dict, selected_date):
    if selected_date is None:
        return results_dict  # Return if no date is selected

    for book_name, results in results_dict.items():  # Iterate per book
        num_results = len(results)
        if num_results > 0:
            days_in_month = calendar.monthrange(selected_date.year, selected_date.month)[1]
            total_seconds = (days_in_month - 1) * 24 * 3600
            seconds_interval = total_seconds / num_results
            start_datetime = datetime(selected_date.year, selected_date.month, 1)
            current_datetime = start_datetime


            for i in range(num_results):
                next_datetime = current_datetime + timedelta(seconds=seconds_interval)
                current_date = current_datetime.date() # Moved assignment inside loop
                next_date = next_datetime.date()
                date_range_str = f"{current_date.strftime('%h %d')} - {next_date.strftime('%h %d')}"
                results[i]['Monthly Projection'] = date_range_str
                current_datetime = next_datetime # Add this
                current_date = next_datetime.date() # Add this too
    return results_dict


def add_yearly_projection(results_dict, selected_date): #Correct name, handle dictionary input
    if selected_date is None:
        return results_dict  # Return if no date is selected

    for book_name, results in results_dict.items(): # Iterate per book
        num_results = len(results)
        if num_results > 0:
            days_in_year = 366 if calendar.isleap(selected_date.year) else 365
            total_seconds = (days_in_year - 1) * 24 * 3600
            seconds_interval = total_seconds / num_results
            start_datetime = datetime(selected_date.year, 1, 1)
            current_datetime = start_datetime


            for i in range(num_results):
                next_datetime = current_datetime + timedelta(seconds=seconds_interval)
                current_date = current_datetime.date() # Move assignment inside loop
                next_date = next_datetime.date()
                date_range_str = f"{current_date.strftime('%b %d')} - {next_date.strftime('%b %d')}"
                results[i]['Yearly Projection'] = date_range_str
                current_datetime = next_datetime # Update current datetime for next iteration

    return results_dict


def sort_results(results):
    def parse_time(time_str):
        try:
            hours, minutes = map(int, time_str.split(':'))
            return hours * 60 + minutes  # Convert to total minutes
        except ValueError:
            return 24 * 60  # Sort invalid times to the end

    return sorted(results, key=lambda x: (
        parse_time(x.get('24h Projection', '23:59').split('-')[0]),  # Sort by start time first
        parse_time(x.get('24h Projection', '23:59').split('-')[1])   # Then by end time
    ))

# --- Main Gradio App ---
with gr.Blocks() as app:
    with gr.Column():
        with gr.Row():
            tlang = create_language_dropdown("Target Language for Result Translation", default_value='english')
            selected_date = Calendar(type="datetime", label="Date to investigate (optional)", info="Pick a date from the calendar")
            use_day = gr.Checkbox(label="Use Day", info="Check to include day in search", value=True)
            use_month = gr.Checkbox(label="Use Month", info="Check to include month in search", value=True)
            use_year = gr.Checkbox(label="Use Year", info="Check to include year in search", value=True)
            date_language_input = create_language_dropdown("Language of the person/topic (optional) (Date Word Language)", default_value='english')
        with gr.Row():
            gematria_text = gr.Textbox(label="Name and/or Topic (required)", value="Hans Albert Einstein Mileva Marity-Einstein")
            date_words_output = gr.Textbox(label="Date in Words Translated (optional)")
            gematria_result = gr.Number(label="Journal Sum")
        #with gr.Row():


        with gr.Row():
            step = gr.Number(label="Jump Width (Steps) for ELS")
            float_step = gr.Number(visible=False, value=1) 
            half_step_btn = gr.Button("Steps / 2")
            double_step_btn = gr.Button("Steps * 2")
        
            with gr.Column():
                round_x = gr.Number(label="Round (1)", value=1)
                round_y = gr.Number(label="Round (2)", value=-1)
            
            rounds_combination = gr.Textbox(label="Combined Rounds", value="1,-1")

        with gr.Row():
            include_torah_chk = gr.Checkbox(label="Include Torah", value=True)
            include_bible_chk = gr.Checkbox(label="Include Bible", value=True)
            include_quran_chk = gr.Checkbox(label="Include Quran", value=True)
            include_hindu_chk = gr.Checkbox(label="Include Rigveda", value=True)
            include_tripitaka_chk = gr.Checkbox(label="Include Tripitaka", value=True)
            merge_results_chk = gr.Checkbox(label="Merge Results (Torah-Bible-Quran)", value=True)
        
            strip_spaces = gr.Checkbox(label="Strip Spaces from Books", value=True)
            strip_in_braces = gr.Checkbox(label="Strip Text in Braces from Books", value=True)
            strip_diacritics_chk = gr.Checkbox(label="Strip Diacritics from Books", value=True)

        translate_btn = gr.Button("Search with ELS")

        # --- Output Components ---
        markdown_output = gr.Dataframe(label="ELS Results")
        most_frequent_phrase_output = gr.Textbox(label="Most Frequent Phrase in Network Search")
        json_output = gr.Textbox(label="JSON Output")
        copy_json_button = gr.Button("Copy JSON to Clipboard")

        # --- Hidden HTML component to hold the JavaScript function ---
        html_js = gr.HTML(
            """
            <script>
            function copyJSON(json_string) {
                navigator.clipboard.writeText(json_string);
            }
            </script>
            """, visible=False  # Hide the HTML component
        )


    # --- Event Handlers ---

    def update_date_words(selected_date, date_language_input, use_day, use_month, use_year):
        if selected_date is None:
            return ""

        if not use_year and not use_month and not use_day:
            return translate_date_to_words(selected_date, date_language_input)

        year = selected_date.year if use_year else None
        month = selected_date.month if use_month else None
        day = selected_date.day if use_day else None

        if year is not None and month is not None and day is not None:
            date_obj = selected_date
        elif year is not None and month is not None:
            date_obj = str(f"{year}-{month}")
        elif year is not None:
            date_obj = str(f"{year}")
        else:  # Return empty string if no date components are selected
            return ""


        date_in_words = date_to_words(date_obj)


        translator = GoogleTranslator(source='auto', target=date_language_input)
        translated_date_words = translator.translate(date_in_words)
        return custom_normalize(translated_date_words)

    def update_journal_sum(gematria_text, date_words_output):
        sum_value = calculate_gematria_sum(gematria_text, date_words_output)
        return sum_value, sum_value, sum_value

    def update_rounds_combination(round_x, round_y):
        return f"{int(round_x)},{int(round_y)}"

    def update_step_half(float_step):
        new_step = math.ceil(float_step / 2)
        return new_step, float_step / 2

    def update_step_double(float_step):
        new_step = math.ceil(float_step * 2)
        return new_step, float_step * 2


    def find_closest_phrase(target_phrase, phrases):
        best_match = None
        best_score = 0

        logging.debug(f"Target phrase for similarity search: {target_phrase}")  # Log target phrase

        for phrase, _, _, _, _, _ in phrases:
            word_length_diff = abs(len(target_phrase.split()) - len(phrase.split()))
            similarity_score = fuzz.ratio(target_phrase, phrase)
            combined_score = similarity_score - word_length_diff

            logging.debug(f"Comparing with phrase: {phrase}")  # Log each phrase being compared
            logging.debug(
                f"Word Length Difference: {word_length_diff}, Similarity Score: {similarity_score}, Combined Score: {combined_score}")  # Log scores

            if combined_score > best_score:
                best_score = combined_score
                best_match = phrase

        logging.debug(f"Closest phrase found: {best_match} with score: {best_score}")  # Log the best match
        return best_match

    def perform_search(step, rounds_combination, tlang, strip_spaces, strip_in_braces, strip_diacritics_chk, include_torah, include_bible, include_quran, include_hindu, include_tripitaka, gematria_text, date_words_output, selected_date):
        # Inside perform_search
        els_results = perform_els_search(step, rounds_combination, tlang, strip_spaces, strip_in_braces,
                                         strip_diacritics_chk, include_torah, include_bible, include_quran,
                                         include_hindu,
                                         include_tripitaka)

        # --- Network Search Integration ---
        most_frequent_phrases = {}
        combined_and_sorted_results = []  # Combined list to hold all results

        for book_name, book_results in els_results.items():
            if book_results:  # Add this check to ensure book_results is not empty
                most_frequent_phrases[book_name] = ""  # Default value

                for result in book_results:
                    try:
                        gematria_sum = calculate_gematria(result['result_text'])  # Calculate gematria
                        max_words = len(result['result_text'].split())
                        matching_phrases = search_gematria_in_db(gematria_sum, max_words)
                        max_words_limit = 20
                        while not matching_phrases and max_words < max_words_limit:  # Increase max_words for more results
                            max_words += 1
                            matching_phrases = search_gematria_in_db(gematria_sum, max_words)

                        if matching_phrases:
                            most_frequent_phrase = get_most_frequent_phrase(matching_phrases)
                            most_frequent_phrases[book_name] = most_frequent_phrase
                        else:
                            closest_phrase = find_closest_phrase(result['result_text'],
                                                                 search_gematria_in_db(gematria_sum, max_words_limit))
                            most_frequent_phrases[
                                book_name] = closest_phrase or ""  # Update most frequent phrases even if no phrase found

                        result['Most Frequent Phrase'] = most_frequent_phrases[book_name]
                        if 'book' in result:
                            if isinstance(result['book'], int): # Torah, Bible, Quran case
                                result['book'] = f"{book_name} {result['book']}."
                        combined_and_sorted_results.append(result)

                    except KeyError as e:
                        print(f"DEBUG: KeyError - Key '{e.args[0]}' not found in result. Skipping this result.")
                        continue

        # --- Batch Translation ---
        selected_language_long = tlang  # From the Gradio dropdown (long form)
        tlang_short = LANGUAGES_SUPPORTED.get(selected_language_long) #Get the short code.
        if tlang_short is None: # Handle unsupported languages
            tlang_short = "en"
            logger.warning(f"Unsupported language selected: {selected_language_long}. Defaulting to English (en).")

        phrases_to_translate = [result.get('Most Frequent Phrase', '') for result in combined_and_sorted_results]
        translated_phrases = translation_utils.batch_translate(phrases_to_translate, tlang_short)  # Use short code here
        result_texts_to_translate = [result.get('result_text', '') for result in combined_and_sorted_results]
        translated_result_texts = translation_utils.batch_translate(result_texts_to_translate, tlang_short)  # And here

        for i, result in enumerate(combined_and_sorted_results):
            result['translated_text'] = translated_result_texts.get(result_texts_to_translate[i],
                                                                    None)  # Store translated_text
            result['Translated Most Frequent Phrase'] = translated_phrases.get(phrases_to_translate[i],
                                                                               None)  # Use get to handle missing keys

        # Time Projections (using els_results dictionary)
        updated_els_results = add_24h_projection(els_results) # Use original els_results dictionary
        updated_els_results = add_monthly_projection(updated_els_results, selected_date) #Call correct functions with correct params
        updated_els_results = add_yearly_projection(updated_els_results, selected_date)

        combined_and_sorted_results = []
        for book_results in updated_els_results.values(): #Combine results for dataframe and json
            combined_and_sorted_results.extend(book_results)
        combined_and_sorted_results = sort_results(combined_and_sorted_results) #sort combined results

        df = pd.DataFrame(combined_and_sorted_results)
        df.index = range(1, len(df) + 1)
        df.reset_index(inplace=True)
        df.rename(columns={'index': 'Result Number'}, inplace=True)

        for i, result in enumerate(combined_and_sorted_results): # Iterate through the combined list
            result['Result Number'] = i + 1

        search_config = {
            "step": step,
            "rounds_combination": rounds_combination,
            "target_language": tlang,
            "strip_spaces": strip_spaces,
            "strip_in_braces": strip_in_braces,
            "strip_diacritics": strip_diacritics_chk,
            "include_torah": include_torah,
            "include_bible": include_bible,
            "include_quran": include_quran,
            "include_hindu": include_hindu,
            "include_tripitaka": include_tripitaka,
            "gematria_text": gematria_text,
            "date_words": date_words_output
        }

        output_data = {
            "search_configuration": search_config,
            "results": combined_and_sorted_results # Use the combined list here
        }

        json_data = json.dumps(output_data, ensure_ascii=False, indent=4)

        # --- Return results ---
        combined_most_frequent = "\n".join(
            f"{book}: {phrase}" for book, phrase in most_frequent_phrases.items())  # Combine phrases
        return df, combined_most_frequent, json_data



    # --- Event Triggers ---
    round_x.change(update_rounds_combination, inputs=[round_x, round_y], outputs=rounds_combination)
    round_y.change(update_rounds_combination, inputs=[round_x, round_y], outputs=rounds_combination)

    selected_date.change(update_date_words, inputs=[selected_date, date_language_input, use_day, use_month, use_year], outputs=[date_words_output])
    date_language_input.change(update_date_words, inputs=[selected_date, date_language_input, use_day, use_month, use_year], outputs=[date_words_output])

    gematria_text.change(update_journal_sum, inputs=[gematria_text, date_words_output], outputs=[gematria_result, step, float_step])
    date_words_output.change(update_journal_sum, inputs=[gematria_text, date_words_output], outputs=[gematria_result, step, float_step])

    half_step_btn.click(update_step_half, inputs=[float_step], outputs=[step, float_step])
    double_step_btn.click(update_step_double, inputs=[float_step], outputs=[step, float_step])

    translate_btn.click(
        perform_search,
        inputs=[step, rounds_combination, tlang, strip_spaces, strip_in_braces, strip_diacritics_chk, include_torah_chk, include_bible_chk, include_quran_chk, include_hindu_chk, include_tripitaka_chk, gematria_text, date_words_output, selected_date],
        outputs=[markdown_output, most_frequent_phrase_output, json_output]
    )

    app.load(
        update_date_words,
        inputs=[selected_date, date_language_input, use_day, use_month, use_year],  # Include all 5 inputs
        outputs=[date_words_output]
    )

    copy_json_button.click(
        js="""
        (json_string) => {
            copyJSON(json_string);  // Call the JavaScript function defined in the HTML component
        }""",
        inputs=json_output, # Make sure json_output is an input
    )

    use_day.change(
        update_date_words,
        inputs=[selected_date, date_language_input, use_day, use_month, use_year],
        outputs=[date_words_output]
    )
    use_month.change(
        update_date_words,
        inputs=[selected_date, date_language_input, use_day, use_month, use_year],
        outputs=[date_words_output]
    )
    use_year.change(
        update_date_words,
        inputs=[selected_date, date_language_input, use_day, use_month, use_year],
        outputs=[date_words_output]
    )

    def checkbox_behavior(use_day_value, use_month_value):
        if use_day_value: # Tick month and year automatically when day is ticked.
            return True, True

        return use_month_value, True # return  month value unchanged and automatically tick year if month is checked

    use_day.change(checkbox_behavior, inputs=[use_day, use_month], outputs=[use_month, use_year])
    use_month.change(checkbox_behavior, inputs=[use_day, use_month], outputs=[use_month, use_year]) #No need for use_day here, day won't be changed by month


if __name__ == "__main__":
    app.launch(share=False)

[File Ends] app.py

[File Begins] app.py.bak
import gradio as gr
import json
import re
import sqlite3
import logging
from collections import defaultdict
from typing import Tuple, Dict, List

# Assuming you have these files in your project
from util import process_json_files
from gematria import calculate_gematria
from deep_translator import GoogleTranslator, exceptions
from urllib.parse import quote_plus
from tqdm import tqdm

# Constants
DATABASE_FILE = 'gematria.db'
MAX_PHRASE_LENGTH_LIMIT = 20
BATCH_SIZE = 10000

# Set up logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')

# Global variables
conn: sqlite3.Connection = None
translator: GoogleTranslator = None
book_names: Dict[int, str] = {}
gematria_cache: Dict[Tuple[int, int], List[Tuple[str, str, int, int, int, str]]] = {}
translation_cache: Dict[str, str] = {}
total_word_count: int = 0  # Global counter for word position


def initialize_database() -> None:
    """Initializes the SQLite database."""
    global conn
    conn = sqlite3.connect(DATABASE_FILE)
    cursor = conn.cursor()

    cursor.execute('''
    CREATE TABLE IF NOT EXISTS results (
        gematria_sum INTEGER,
        words TEXT,
        translation TEXT,
        book TEXT,
        chapter INTEGER,
        verse INTEGER,
        phrase_length INTEGER,
        word_position TEXT, 
        PRIMARY KEY (gematria_sum, words, book, chapter, verse, word_position)
    )
    ''')

    cursor.execute('''
    CREATE INDEX IF NOT EXISTS idx_results_gematria
    ON results (gematria_sum)
    ''')

    cursor.execute('''
    CREATE TABLE IF NOT EXISTS processed_books (
        book TEXT PRIMARY KEY,
        max_phrase_length INTEGER
    )
    ''')

    conn.commit()


def initialize_translator() -> None:
    """Initializes the Google Translator."""
    global translator
    translator = GoogleTranslator(source='iw', target='en')
    logging.info("Translator initialized.")


def process_book(book_id: int, max_phrase_length: int, cursor):
    """Processes a single book and returns phrases to insert."""
    global book_names, total_word_count
    book_data = process_json_files(book_id, book_id)
    phrases_to_insert = []

    if book_id in book_data:
        book_data = book_data[book_id]
        if 'title' not in book_data or not isinstance(book_data['title'], str):
            logging.warning(f"Skipping book {book_id} due to missing 'title' field.")
            return phrases_to_insert

        title = book_data['title']
        book_names[book_id] = title

        # Check if this book has already been processed for this phrase length
        cursor.execute('''SELECT max_phrase_length FROM processed_books WHERE book = ?''', (title,))
        result = cursor.fetchone()
        if result and result[0] >= max_phrase_length:
            logging.info(f"Skipping book {title}: Already processed with max_phrase_length {result[0]}")
            return phrases_to_insert

        if 'text' not in book_data or not isinstance(book_data['text'], list):
            logging.warning(f"Skipping book {book_id} due to missing 'text' field.")
            return phrases_to_insert

        chapters = book_data['text']
        for chapter_id, chapter in enumerate(chapters):
            for verse_id, verse in enumerate(chapter):
                verse_text = flatten_text(verse)
                verse_text = re.sub(r'\[.*?\]', '', verse_text)
                verse_text = re.sub(r"[^\u05D0-\u05EA ]+", "", verse_text)
                verse_text = re.sub(r" +", " ", verse_text)
                words = verse_text.split()

                for length in range(1, max_phrase_length + 1):
                    for start in range(len(words) - length + 1):
                        phrase_candidate = " ".join(words[start:start + length])
                        gematria_sum = calculate_gematria(phrase_candidate.replace(" ", ""))

                        word_position_range = f"{total_word_count + start + 1}-{total_word_count + start + length}"

                        phrases_to_insert.append(
                            (gematria_sum, phrase_candidate, None, title, chapter_id + 1, verse_id + 1, length,
                             word_position_range))

                total_word_count += len(words)

    return phrases_to_insert


def populate_database(start_book: int, end_book: int, max_phrase_length: int = 1) -> None:
    """Populates the database with phrases from the Tanach."""
    global conn, book_names, total_word_count
    logging.info(f"Populating database with books from {start_book} to {end_book}...")

    with sqlite3.connect(DATABASE_FILE) as conn:
        cursor = conn.cursor()

        for book_id in tqdm(range(start_book, end_book + 1), desc="Processing Books"):
            phrases_to_insert = process_book(book_id, max_phrase_length, cursor)

            if phrases_to_insert:
                cursor.executemany('''
                INSERT OR IGNORE INTO results (gematria_sum, words, translation, book, chapter, verse, phrase_length, word_position) 
                VALUES (?, ?, ?, ?, ?, ?, ?, ?)
                ''', phrases_to_insert)

                # Update processed_books after processing each book
                cursor.execute('''
                INSERT OR REPLACE INTO processed_books (book, max_phrase_length)
                VALUES (?, ?)
                ''', (book_names[book_id], max_phrase_length))

                conn.commit()

        total_word_count = 0  # Reset for the next set of phrase lengths


def get_translation(phrase: str) -> str:
    """Retrieves or generates the English translation of a Hebrew phrase
    and caches it in the database.
    """
    global conn, translator, translation_cache

    # Check if the translation exists in the database
    with sqlite3.connect(DATABASE_FILE) as conn:
        cursor = conn.cursor()
        cursor.execute("SELECT translation FROM results WHERE words = ? LIMIT 1", (phrase,))
        result = cursor.fetchone()
        if result and result[0]:  # If a translation exists, use it
            return result[0]

    # If no translation in the database, translate and store it
    translation = translate_and_store(phrase)
    translation_cache[phrase] = translation

    # Update the database with the new translation
    with sqlite3.connect(DATABASE_FILE) as conn:
        cursor = conn.cursor()
        cursor.execute("UPDATE results SET translation = ? WHERE words = ?", (translation, phrase))
        conn.commit()

    return translation


def translate_and_store(phrase: str) -> str:
    """Translates a Hebrew phrase to English using Google Translate."""
    global translator
    max_retries = 3
    retries = 0
    while retries < max_retries:
        try:
            translation = translator.translate(phrase)
            return translation
        except (exceptions.TranslationNotFound, exceptions.NotValidPayload,
                exceptions.ServerException, exceptions.RequestError) as e:
            retries += 1
            logging.warning(f"Error translating phrase '{phrase}': {e}. Retrying... ({retries}/{max_retries})")
    logging.error(f"Failed to translate phrase '{phrase}' after {max_retries} retries.")
    return "[Translation Error]"


def search_gematria_in_db(gematria_sum: int, max_words: int) -> List[Tuple[str, str, int, int, int, str]]:
    """Searches the database for phrases with a given Gematria value."""
    global conn
    with sqlite3.connect(DATABASE_FILE) as conn:
        cursor = conn.cursor()
        cursor.execute('''
        SELECT words, book, chapter, verse, phrase_length, word_position 
        FROM results 
        WHERE gematria_sum = ? AND phrase_length <= ?
        ''', (gematria_sum, max_words))
        results = cursor.fetchall()
    return results


def gematria_search_interface(phrases: str, max_words: int, show_translation: bool) -> str:
    """The main function for the Gradio interface, handling multiple phrases."""
    global conn, book_names, gematria_cache

    results = []
    all_results = []  # Store results for each phrase
    middle_words_results = []  # Store middle word results for all books
    all_names_average_position = 0  # Initialize variable for average position across all names and books
    total_name_count = 0  # Initialize counter for the total number of names processed

    phrases = phrases.strip().splitlines()
    if not phrases:
        return "Please enter at least one phrase."

    for phrase in phrases:
        if not phrase.strip():
            continue  # Skip empty lines

        numbers = re.findall(r'\d+', phrase)
        text_without_numbers = re.sub(r'\d+', '', phrase)
        phrase_gematria = calculate_gematria(text_without_numbers.replace(" ", ""))
        phrase_gematria += sum(int(number) for number in numbers)

        if (phrase_gematria, max_words) in gematria_cache:
            matching_phrases = gematria_cache[(phrase_gematria, max_words)]
        else:
            matching_phrases = search_gematria_in_db(phrase_gematria, max_words)
            gematria_cache[(phrase_gematria, max_words)] = matching_phrases

        if not matching_phrases:
            results.append(f"No matching phrases found for: {phrase}")
            continue

        sorted_phrases = sorted(matching_phrases,
                                key=lambda x: (int(list(book_names.keys())[list(book_names.values()).index(x[1])]), x[2],
                                               x[3]))
        results_by_book = defaultdict(list)
        for words, book, chapter, verse, phrase_length, word_position in sorted_phrases:
            results_by_book[book].append((words, chapter, verse, phrase_length, word_position))

        results.append(f"<h2>Results for: {phrase} (Gematria: {phrase_gematria})</h2>")
        results.append("<div class='results-container'>")
        for book, phrases in results_by_book.items():
            for words, chapter, verse, phrase_length, word_position in phrases:
                translation = get_translation(words) if show_translation else ""
                link = f"https://www.biblegateway.com/passage/?search={quote_plus(book)}+{chapter}%3A{verse}&version=CJB"
                results.append(f"""
                <div class='result-item'>
                  <p><b>Book:</b> {book}</p>
                  <p><b>Chapter:</b> {chapter}, <b>Verse:</b> {verse}</p>
                  <p class='hebrew-phrase'><b>Hebrew Phrase:</b> {words}</p>
                  <p><b>Translation:</b> {translation}</p>
                  <p><b>Phrase Length:</b> {phrase_length} words</p>
                  <p><b>Phrase Gematria:</b> {phrase_gematria}</p>
                  <p><b>Word Position in the Tanach:</b> {word_position}</p> 
                  <a href='{link}' target='_blank' class='bible-link'>[See on Bible Gateway]</a>
                </div>
                """)

        # Calculate average position for the current name across all books
        name_average_position = calculate_average_position_for_name(results_by_book)
        if name_average_position is not None:
            results.append(f"<p><b>Average Word Position for '{phrase}' across all books:</b> {name_average_position:.2f}</p>")
            all_names_average_position += name_average_position
            total_name_count += 1

        results.append("</div>")
        all_results.append(results_by_book)  # Store results by book without the phrase

    # Calculate the average word position across all names and all their books
    if total_name_count > 0:
        all_names_average_position /= total_name_count
        results.append(f"<h2>Average Word Position Across All Names and Books: {all_names_average_position:.2f}</h2>")

    # Calculate middle words for all input lines (common books)
    if len(all_results) >= 2:
        results.append("<h2>Middle Words (Common Books):</h2>")
        results.append("<div class='results-container'>")

        common_books = set.intersection(*[set(results.keys()) for results in all_results])
        logging.debug(f"Common books: {common_books}")

        for book in common_books:
            logging.debug(f"Processing book: {book}")

            # Find nearest positions for all phrases in the current book
            nearest_positions = find_nearest_positions([results[book] for results in all_results])
            logging.debug(f"Nearest positions in {book}: {nearest_positions}")

            if nearest_positions:
                middle_word_position = sum(nearest_positions) / len(nearest_positions)
                logging.debug(f"Calculated middle word position in {book}: {middle_word_position}")

                start_position = int(middle_word_position)
                end_position = start_position + 1 if middle_word_position % 1 != 0 else start_position
                logging.debug(f"Middle word position range in {book}: {start_position}-{end_position}")

                middle_words_data = get_words_from_db(book, start_position, end_position)
                logging.debug(f"Middle words data fetched from database: {middle_words_data}")

                if middle_words_data:
                    # Store middle word data along with book name for sorting
                    middle_words_results.extend([(book, data) for data in middle_words_data])
                else:
                    # Handle edge case: fetch words independently for start and end positions
                    logging.debug(f"No middle words found for range {start_position}-{end_position}. "
                                  f"Fetching words independently.")
                    middle_words_data_start = get_words_from_db(book, start_position, start_position)
                    middle_words_data_end = get_words_from_db(book, end_position, end_position)

                    if middle_words_data_start or middle_words_data_end:
                        middle_words_results.extend([(book, data) for data in middle_words_data_start + middle_words_data_end])

        # Sort middle words results by book order before displaying
        middle_words_results.sort(key=lambda x: int(list(book_names.keys())[list(book_names.values()).index(x[0])]))

        for book, (words, chapter, verse, phrase_length, word_position) in middle_words_results:
            translation = get_translation(words) if show_translation else ""
            link = f"https://www.biblegateway.com/passage/?search={quote_plus(book)}+{chapter}%3A{verse}&version=CJB"
            results.append(f"""
                        <div class='result-item'>  
                          <p><b>Book:</b> {book}</p>
                          <p><b>Chapter:</b> {chapter}, <b>Verse:</b> {verse}</p>
                          <p class='hebrew-phrase'><b>Hebrew Phrase:</b> {words}</p>
                          <p><b>Translation:</b> {translation}</p>
                          <p><b>Phrase Length:</b> {phrase_length} words</p>
                          <p><b>Word Position in the Tanach:</b> {word_position}</p>
                          <a href='{link}' target='_blank' class='bible-link'>[See on Bible Gateway]</a>
                        </div>
                        """)
        results.append("</div>")

    # Style modified to position search on top and results below
    style = """
        <style>
          .results-container {
            display: grid;
            grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
            gap: 20px;
            width: 100%;  /* Make results container take full width */
          }
          .result-item {
            border: 1px solid #ccc;
            padding: 15px;
            border-radius: 5px;
            box-shadow: 2px 2px 5px rgba(0, 0, 0, 0.1);
          }
          .hebrew-phrase {
            font-family: 'SBL Hebrew', 'Ezra SIL', serif;
            direction: rtl;
          }
          .bible-link {
            display: block;
            margin-top: 10px;
            color: #007bff;
            text-decoration: none;
          }
        </style>
        """
    return style + "\n".join(results)

def calculate_average_position_for_name(results_by_book: Dict[str, List[Tuple]]) -> float:
    """Calculates the average word position for a single name across all books."""
    positions = []
    for book, phrases in results_by_book.items():
        for _, _, _, _, word_position in phrases:
            start, end = map(int, word_position.split('-'))
            positions.append((start + end) / 2)
    return sum(positions) / len(positions) if positions else None

def find_nearest_positions(results_lists: List[List]) -> List[int]:
    """Finds the nearest word positions among multiple lists of results."""
    nearest_positions = []
    for i in range(len(results_lists)):
        positions_i = [(int(pos.split('-')[0]) + int(pos.split('-')[1])) / 2
                       for _, _, _, _, pos in results_lists[i]]  # Get average of start and end positions
        logging.debug(f"Positions for phrase {i+1}: {positions_i}")

        # Calculate the average position for the current phrase
        average_position = sum(positions_i) / len(positions_i) if positions_i else None
        logging.debug(f"Average position for phrase {i+1}: {average_position}")

        if average_position is not None:
            nearest_positions.append(average_position)

    return nearest_positions


def get_words_from_db(book: str, start_position: int, end_position: int) -> List[Tuple]:
    """Fetches words from the database based on the book and exact word position range."""
    global conn
    logging.debug(f"Fetching words from database for {book} at positions {start_position}-{end_position}")
    with sqlite3.connect(DATABASE_FILE) as conn:
        cursor = conn.cursor()
        cursor.execute("""
            SELECT words, chapter, verse, phrase_length, word_position
            FROM results
            WHERE book = ? AND word_position = ?
        """, (book, f"{start_position}-{end_position}"))  # Directly compare word_position
        results = cursor.fetchall()
        logging.debug(f"Words fetched from database: {results}")
    return results


def flatten_text(text: List) -> str:
    """Flattens nested lists into a single list."""
    if isinstance(text, list):
        return " ".join(flatten_text(item) if isinstance(item, list) else item for item in text)
    return text


def run_app() -> None:
    """Initializes and launches the Gradio app."""
    global conn
    initialize_database()
    initialize_translator()

    logging.info("Starting database population...")
    for max_phrase_length in range(1, MAX_PHRASE_LENGTH_LIMIT + 1):
        populate_database(1, 39, max_phrase_length=max_phrase_length)
    logging.info("Database population complete.")

    with gr.Blocks() as iface:  # Use gr.Blocks() for layout control
        with gr.Row():  # Place inputs in a row
            textbox = gr.Textbox(label="Enter word(s) or numbers (one phrase per line)", lines=5)
            slider = gr.Slider(label="Max Word Count in Result Phrases", minimum=1,
                              maximum=MAX_PHRASE_LENGTH_LIMIT, step=1,
                              value=1)
            checkbox = gr.Checkbox(label="Show Translation", value=True)
        with gr.Row():  # Place buttons in a row
            clear_button = gr.Button("Clear")
            submit_button = gr.Button("Submit", variant="primary")

        html_output = gr.HTML(label="Results")  # Output for the results

        submit_button.click(fn=gematria_search_interface,
                            inputs=[textbox, slider, checkbox],
                            outputs=html_output)
        clear_button.click(fn=lambda: "", inputs=None, outputs=html_output)  # Clear the output

    iface.launch()


if __name__ == "__main__":
    run_app()
[File Ends] app.py.bak

[File Begins] bible.py
import logging
logger = logging.getLogger(__name__)


import json
import os
import re
from deep_translator import GoogleTranslator
from gematria import calculate_gematria
import math


# Hebrew gematria values for relevant characters
gematria_values = {
    'א': 1, 'ב': 2, 'ג': 3, 'ד': 4, 'ה': 5, 'ו': 6, 'ז': 7, 'ח': 8, 'ט': 9,
    'י': 10, 'כ': 20, 'ך': 500, 'ל': 30, 'מ': 40, 'ם': 600, 'נ': 50, 'ן': 700,
    'ס': 60, 'ע': 70, 'פ': 80, 'ף': 800, 'צ': 90, 'ץ': 900, 'ק': 100,
    'ר': 200, 'ש': 300, 'ת': 400
}

# Reverse dictionary for converting gematria values back to Hebrew characters
reverse_gematria_values = {v: k for k, v in gematria_values.items()}

# Function to convert a Hebrew string to its gematria values
def string_to_gematria(s):
    return [gematria_values.get(char, 0) for char in s]  # Handle characters not in the dictionary

# Function to convert a single gematria value to Hebrew characters
def gematria_to_string(value):
    result = []
    for val in sorted(reverse_gematria_values.keys(), reverse=True):
        while value >= val:
            result.append(reverse_gematria_values[val])
            value -= val
    return ''.join(result)

# Function to calculate the average gematria values of corresponding characters and convert them to Hebrew characters
def average_gematria(str1, str2):
    # Convert strings to gematria values
    gematria1 = string_to_gematria(str1)
    gematria2 = string_to_gematria(str2)

    # Handle cases where strings have different lengths by padding with 0s
    max_len = max(len(gematria1), len(gematria2))
    gematria1.extend([0] * (max_len - len(gematria1)))
    gematria2.extend([0] * (max_len - len(gematria2)))

    # Calculate the average of corresponding gematria values and apply math.ceil
    average_gematria_values = [math.ceil((g1 + g2) / 2) for g1, g2 in zip(gematria1, gematria2)]

    # Convert the average gematria values back to Hebrew characters
    return ''.join(gematria_to_string(val) for val in average_gematria_values)


from deep_translator import GoogleTranslator
import os
import re
import csv


def process_json_files(start=1, end=66, step=1, rounds="1", length=0, tlang="en", strip_spaces=True,
                       strip_in_braces=True, strip_diacritics=True, average_compile=False):
    file_name = "texts/bible/OpenGNT_version3_3.csv"
    translator = GoogleTranslator(source='auto', target=tlang)
    results = []

    # Dictionary für die 27 Bücher des Neuen Testaments (Englische Namen)
    nt_books = {
        40: "Matthew",
        41: "Mark",
        42: "Luke",
        43: "John",
        44: "Acts",
        45: "Romans",
        46: "1. Corinthians",
        47: "2. Corinthians",
        48: "Galatians",
        49: "Ephesians",
        50: "Philippians",
        51: "Colossians",
        52: "1. Thessalonians",
        53: "2. Thessalonians",
        54: "1. Timothy",
        55: "2. Timothy",
        56: "Titus",
        57: "Philemon",
        58: "Hebrews",
        59: "James",
        60: "1. Peter",
        61: "2. Peter",
        62: "1. John",
        63: "2. John",
        64: "3. John",
        65: "Jude",
        66: "Revelation"
    }

    try:
        with open(file_name, 'r', encoding='utf-8') as file:
            reader = csv.DictReader(file, delimiter='\t')

            book_texts = {}
            current_book = None
            for row in reader:
                book = int(row['〔Book｜Chapter｜Verse〕'].split('｜')[0][1:])
                if book < start or book > end:
                    continue
                if current_book != book:
                    current_book = book
                    book_texts[book] = ""
                greek_text = row['〔OGNTk｜OGNTu｜OGNTa｜lexeme｜rmac｜sn〕']
                greek_text = greek_text.split('〔')[1]
                greek_text = greek_text.split('｜')[0]
                book_texts[book] += greek_text + " "

            for book, full_text in book_texts.items():
                logger.debug(f"Processing book {book}")
                clean_text = full_text
                if strip_in_braces:
                    clean_text = re.sub(r"\[.*?\]", "", clean_text, flags=re.DOTALL)
                if strip_diacritics:
                    clean_text = re.sub(r"[^\u0370-\u03FF\u1F00-\u1FFF ]+", "", clean_text)
                if strip_spaces:
                    clean_text = clean_text.replace(" ", "")
                else:
                    clean_text = clean_text.replace("  ", " ")
                    clean_text = clean_text.replace("  ", " ")
                    clean_text = clean_text.replace("  ", " ")

                text_length = len(clean_text)

                selected_characters_per_round = {}
                for round_num in map(int, rounds.split(',')):
                    if not (round_num == 1 and step > text_length) and not (round_num == -1 and step > text_length):
                        if round_num > 0:
                            current_position = step - 1
                        else:
                            current_position = text_length - 1 if step == 1 else text_length - step

                        completed_rounds = 0
                        selected_characters = ""

                        while completed_rounds < abs(round_num):
                            selected_characters += clean_text[current_position % text_length]
                            current_position += step if round_num > 0 else -step

                            if (round_num > 0 and current_position >= text_length * (completed_rounds + 1)) or \
                               (round_num < 0 and current_position < 0):
                                completed_rounds += 1

                        selected_characters_per_round[round_num] = selected_characters

                if average_compile and len(selected_characters_per_round) > 1:
                    result_text = ""
                    keys = sorted(selected_characters_per_round.keys())
                    for i in range(len(keys) - 1):
                        result_text = average_gematria(selected_characters_per_round[keys[i]], selected_characters_per_round[keys[i+1]])
                else:
                    result_text = ''.join(selected_characters_per_round.values())

                if length != 0:
                    result_text = result_text[:length]

                translated_text = translator.translate(result_text) if result_text else ""

                result_sum = calculate_gematria(result_text)

                if result_text:
                    logger.debug(f"Result for book {book}: {result_text}")
                    result = {
                        'book': f"Bible {book}.",  # Use the correct 'book' variable
                        'title': nt_books.get(book, "Unknown Book"),  # Get book name from dictionary
                        'result_text': result_text,
                        'result_sum': result_sum,  # Make sure result_sum is calculated correctly
                        'translated_text': translated_text
                    }
                    results.append(result)


    except FileNotFoundError:
        results.append({"error": f"File {file_name} not found."})

    return results
        # Tests
test_results = [
    #(process_json_files(1, 1, 21, rounds="3", length=0), ""),
    #(process_json_files(1, 1, 22, rounds="1", length=0), ""), 
    #(process_json_files(1, 1, 22, rounds="3", length=0), ""),
    #(process_json_files(1, 1, 23, rounds="3", length=0), ""),
    #(process_json_files(1, 1, 11, rounds="1", length=0), ""),
    #(process_json_files(1, 1, 2, rounds="1", length=0), ""),
    #(process_json_files(1, 1, 23, rounds="1", length=0), None),  # Expect None, when no results
    #(process_json_files(1, 1, 23, rounds="-1", length=0), None),  # Expect None, when no results
    #(process_json_files(1, 1, 22, rounds="-1", length=0), ""),
    #(process_json_files(1, 1, 22, rounds="-2", length=0), ""),
    #(process_json_files(1, 1, 1, rounds="-1", length=0), ""), # Reversed Hebrew alphabet
    #(process_json_files(1, 1, 1, rounds="1,-1", length=0), ""), # Combined rounds
    #(process_json_files(1, 1, 22, rounds="1,-1", length=0, average_compile=True), ""),  # average compile test (400+1) / 2 = math.ceil(200.5)=201=200+1="רא"
]

all_tests_passed = True
for result, expected in test_results:
    if expected is None:  # Check if no result is expected
        if not result:
            logger.info(f"Test passed: Expected no results, got no results.")
        else:
            logger.error(f"Test failed: Expected no results, but got: {result}")
            all_tests_passed = False
    else:
        # Check if result is not empty before accessing elements
        if result: 
            #result_text = result[0]['result_text']
            result_text = None
            if result_text == expected:
                logger.info(f"Test passed: Expected '{expected}', got '{result_text}'")
            else:
                logger.error(f"Test failed: Expected '{expected}', but got '{result_text}'")
                all_tests_passed = False
        else:
            logger.error(f"Test failed: Expected '{expected}', but got no results")
            all_tests_passed = False

if all_tests_passed:
    logger.info("All round tests passed.")

[File Ends] bible.py

[File Begins] database-structure.txt
Gematria Sum, Words, Translation, Book, Chapter, Verse, Phrase Length, Phrase Position
913	בראשית		Genesis	1	1	1	1-1
1116	בראשית ברא		Genesis	1	1	2	1-2
1762	בראשית ברא אלהים		Genesis	1	1	3	1-3
2163	בראשית ברא אלהים את		Genesis	1	1	4	1-4
3118	בראשית ברא אלהים את השמים		Genesis	1	1	5	1-5
3525	בראשית ברא אלהים את השמים ואת		Genesis	1	1	6	1-6

[File Ends] database-structure.txt

[File Begins] gematria.py
import unicodedata
import logging

logger = logging.getLogger(__name__)

def strip_diacritics(text):
    """
    Entfernt Diakritika von Unicode-Zeichen, um den Basisbuchstaben zu erhalten, und gibt Warnungen
    für tatsächlich unbekannte Zeichen aus.
    """
    stripped_text = ''
    for char in unicodedata.normalize('NFD', text):
        if unicodedata.category(char) not in ['Mn', 'Cf']:
            stripped_text += char
        else:
            logger.info(f"Info: Diakritisches Zeichen '{char}' wird ignoriert.")
    return stripped_text

def letter_to_value(letter):
    """
    Konvertiert einen einzelnen Buchstaben in seinen Gematria-Wert, ignoriert Leerzeichen
    und Nicht-Buchstaben-Zeichen.
    """
    # Dein vorhandenes Wörterbuch bleibt unverändert
    values = {
    # Lateinische Buchstaben
    'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 600,
    'k': 10, 'l': 20, 'm': 30, 'n': 40, 'o': 50, 'p': 60, 'q': 70, 'r': 80, 's': 90,
    't': 100, 'u': 200, 'v': 700, 'w': 900, 'x': 300, 'y': 400, 'z': 500,

    'A': 1, 'B': 2, 'C': 3, 'D': 4, 'E': 5, 'F': 6, 'G': 7, 'H': 8, 'I': 9, 'J': 600,
    'K': 10, 'L': 20, 'M': 30, 'N': 40, 'O': 50, 'P': 60, 'Q': 70, 'R': 80, 'S': 90,
    'T': 100, 'U': 200, 'V': 700, 'W': 900, 'X': 300, 'Y': 400, 'Z': 500,

    # Basisbuchstaben und einige bereits genannte Varianten
    'ا': 1, 'أ': 1, 'إ': 1, 'آ': 1, 'ب': 2, 'ج': 3, 'د': 4, 'ه': 5, 'و': 6, 'ز': 7, 'ح': 8, 'ط': 9,
    'ي': 10, 'ى': 10, 'ك': 20, 'ک': 20, 'ل': 30, 'م': 40, 'ن': 50, 'س': 60, 'ع': 70, 'ف': 80,
    'ص': 90, 'ق': 100, 'ر': 200, 'ش': 300, 'ت': 400, 'ث': 500, 'خ': 600, 'ذ': 700, 'ض': 800, 'ظ': 900, 'غ': 1000,
    'ٱ': 1, # Alif Wasla
    'ـ': 0, # Tatweel

    # Zusätzliche Varianten und Sonderzeichen
    'ة': 400, # Taa Marbuta
    'ؤ': 6,  # Waw mit Hamza darüber
    'ئ': 10, # Ya mit Hamza darüber
    'ء': 1,  # Hamza
    'ى': 10, # Alif Maqsurah
    'ٹ': 400, # Taa' marbuta goal
    'پ': 2,  # Pe (Persisch/Urdu)
    'چ': 3,  # Che (Persisch/Urdu)
    'ژ': 7,  # Zhe (Persisch/Urdu)
    'گ': 20, # Gaf (Persisch/Urdu)
    'ڭ': 20, # Ngaf (Kazakh, Uyghur, Uzbek, and in some Arabic dialects)
    'ں': 50, # Noon Ghunna (Persisch/Urdu)
    'ۀ': 5,  # Heh with Yeh above (Persisch/Urdu)
    'ے': 10, # Barree Yeh (Persisch/Urdu)
    '؋': 0,  # Afghani Sign (wird als Währungssymbol verwendet, nicht für Gematria relevant, aber hier zur Vollständigkeit aufgeführt)

    # Anmerkung: Das Währungssymbol und ähnliche Zeichen sind in einem Gematria-Kontext normalerweise nicht relevant,
    # werden aber der Vollständigkeit halber aufgeführt. Es gibt noch viele weitere spezifische Zeichen in erweiterten
    # arabischen Schriftsystemen (z.B. für andere Sprachen wie Persisch, Urdu, Pashto usw.), die hier nicht vollständig
    # abgedeckt sind.

    # Grund- und Schlussformen hebräischer Buchstaben

    'א': 1, 'ב': 2, 'ג': 3, 'ד': 4, 'ה': 5, 'ו': 6, 'ז': 7, 'ח': 8, 'ט': 9, 'י': 10,
    'כ': 20, 'ך': 500, 'ל': 30, 'מ': 40, 'ם': 600, 'נ': 50, 'ן': 700, 'ס': 60, 'ע': 70, 'פ': 80, 'ף': 800,
    'צ': 90, 'ץ': 900, 'ק': 100, 'ר': 200, 'ש': 300, 'ת': 400,

    # Griechische Buchstaben
    'α': 1, 'β': 2, 'γ': 3, 'δ': 4, 'ε': 5, 'ϝ': 6, 'ζ': 7, 'η': 8, 'θ': 9, 'ι': 10,
    'κ': 20, 'λ': 30, 'μ': 40, 'ν': 50, 'ξ': 60, 'ο': 70, 'π': 80, 'ϟ': 90, 'ρ': 100,
    'σ': 200, 'τ': 300, 'υ': 400, 'φ': 500, 'χ': 600, 'ψ': 700, 'ω': 800, 'ϡ': 900,

        # Griechische Großbuchstaben
    'Α': 1, 'Β': 2, 'Γ': 3, 'Δ': 4, 'Ε': 5, 'Ϝ': 6, 'Ζ': 7, 'Η': 8, 'Θ': 9, 'Ι': 10,
    'Κ': 20, 'Λ': 30, 'Μ': 40, 'Ν': 50, 'Ξ': 60, 'Ο': 70, 'Π': 80, 'Ϟ': 90, 'Ρ': 100,
    'Σ': 200, 'Τ': 300, 'Υ': 400, 'Φ': 500, 'Χ': 600, 'Ψ': 700, 'Ω': 800, 'Ϡ': 900,
    'σ': 200,  # Sigma
    'ς': 200,  # Final Sigma
    'ϲ': 200,  # Lunate Sigma (Greek)
    'Ϲ': 200,  # Uppercase Lunate Sigma (Greek)

        # Katapayadi System (Comprehensive with variants)
            'क': 1, 'ख': 2, 'ग': 3, 'घ': 4, 'ङ': 5,
            'च': 6, 'छ': 7, 'ज': 8, 'झ': 9, 'ञ': 0,  # Or placeholder for zero if appropriate
            'ट': 1, 'ठ': 2, 'ड': 3, 'ढ': 4, 'ण': 5,
            'त': 6, 'थ': 7, 'द': 8, 'ध': 9, 'न': 0, # Or placeholder for zero if appropriate
            'प': 1, 'फ': 2, 'ब': 3, 'भ': 4, 'म': 5,
            'य': 1, 'र': 2, 'ल': 3, 'व': 4, 'श': 5, 'ष': 6, 'स': 7, 'ह': 8,

            # Half forms (same values)
            'क्': 1, 'ख्': 2, 'ग्': 3, 'घ्': 4, 'ङ्': 5,
            'च्': 6, 'छ्': 7, 'ज्': 8, 'झ्': 9, 'ञ्': 0,
            'ट्': 1, 'ठ्': 2, 'ड्': 3, 'ढ्': 4, 'ण्': 5,
            'त्': 6, 'थ्': 7, 'द्': 8, 'ध्': 9, 'न्': 0,
            'प्': 1, 'फ्': 2, 'ब्': 3, 'भ्': 4, 'म्': 5,
            'य्': 1, 'र्': 2, 'ल्': 3, 'व्': 4, 'श्': 5, 'ष्': 6, 'स्': 7, 'ह्': 8,

            # Nukta forms (assuming same values - verify)
            'क़': 1, 'ख़': 2, 'ग़': 3, 'ज़': 8, 'ड़': 3, 'ढ़': 4, 'फ़': 2,

            # Kannada, Telugu, Malayalam equivalents
            'ಕ': 1, 'ಖ': 2, 'ಗ': 3, 'ಘ': 4, 'ಙ': 5, 'ಚ': 6, 'ಛ': 7, 'ಜ': 8, 'ಝ': 9, 'ಞ': 0,
            'ಟ': 1, 'ಠ': 2, 'ಡ': 3, 'ಢ': 4, 'ಣ': 5, 'ತ': 6, 'ಥ': 7, 'ದ': 8, 'ಧ': 9, 'ನ': 0,
            'ಪ': 1, 'ಫ': 2, 'ಬ': 3, 'ಭ': 4, 'ಮ': 5, 'ಯ': 1, 'ರ': 2, 'ಲ': 3, 'ವ': 4, 'ಶ': 5, 'ಷ': 6, 'ಸ': 7, 'ಹ': 8,

            # Malayalam
            'ക': 1, 'ഖ': 2, 'ഗ': 3, 'ഘ': 4, 'ങ': 5,
            'ച': 6, 'ഛ': 7, 'ജ': 8, 'ഝ': 9, 'ഞ': 0,
            'ട': 1, 'ഠ': 2, 'ഡ': 3, 'ഢ': 4, 'ണ': 5,
            'ത': 6, 'ഥ': 7, 'ദ': 8, 'ധ': 9, 'ന': 0,
            'പ': 1, 'ഫ': 2, 'ബ': 3, 'ഭ': 4, 'മ': 5,
            'യ': 1, 'ര': 2, 'ല': 3, 'വ': 4, 'ശ': 5, 'ഷ': 6, 'സ': 7, 'ഹ': 8,


    # Vokale (Svara)
        'अ': 0, 'आ': 0, 'इ': 0, 'ई': 0, 'उ': 0,
        'ऊ': 0, 'ऋ': 0, 'ॠ': 0, 'ऌ': 0, 'ॡ': 0,
        'ए': 0, 'ऐ': 0, 'ओ': 0, 'औ': 0,
    # Zusätzliche Zeichen
        'क्ष': 1, 'त्र': 6, 'ज्ञ': 8,

    # Anusvaras und Visargas
        'ं': 0, 'ः': 0,
    # Halante (Virama)
        '्': 0,
    # Ziffern
        '०': 0, '१': 1, '२': 2, '३': 3, '४': 4,
        '५': 5, '६': 6, '७': 7, '८': 8, '९': 9,
    # Sonderzeichen
        'ॐ': 0, # Om-Symbol
    # Vokalzeichen (Matra)
        'ा': 0, 'ि': 0, 'ी': 0, 'ु': 0, 'ू': 0,
        'ृ': 0, 'ॄ': 0, 'ॢ': 0, 'ॣ': 0, 'े': 0,
        'ै': 0, 'ो': 0, 'ौ': 0,
    # Zusätzliche Zeichen für vollständige Abdeckung
        'ॅ': 0, 'ॆ': 0, 'ॉ': 0, 'ॊ': 0, 'ऍ': 0,
        'ऎ': 0, 'ऑ': 0, 'ऒ': 0, 'ॎ': 0, 'ॏ': 0,
    # Vedische Erweiterungen
        'ᳵ': 0, 'ᳶ': 0, 'ॽ': 0,
    }

    # Stelle sicher, dass Diakritika entfernt werden, bevor auf das Wörterbuch zugegriffen wird
    letter_no_diacritics = strip_diacritics(letter)

    if letter_no_diacritics in values:
        return values[letter_no_diacritics.lower()]
    elif letter.strip() == "":  # Ignoriere Leerzeichen und leere Zeilen
        return 0
    else:
        # Gib eine spezifische Warnung aus, wenn das Zeichen unbekannt ist
        logger.info(f"Warnung: Unbekanntes Zeichen '{letter}' ignoriert.")
        return 0


def calculate_gematria(text):
    """Calculate the Gematria value of a given Hebrew text, ignoring spaces and non-Hebrew characters."""
    return sum(letter_to_value(letter) for letter in text if letter.strip() != "")

[File Ends] gematria.py

[File Begins] hindu.py
import logging
logger = logging.getLogger(__name__)

import json
import os
import re
from deep_translator import GoogleTranslator
from gematria import calculate_gematria
import math

def process_json_files(start=1, end=10, step=1, rounds="1", length=0, tlang="en", strip_spaces=True,
                       strip_in_braces=True, strip_diacritics=True, average_compile=False, translate=False):
    base_path = "texts/rigveda"
    translator = GoogleTranslator(source='ne', target=tlang)
    results = []

    for i in range(start, end + 1):
        file_name = f"{base_path}/rigveda_mandala_{i:02}.json"
        #file_name = f"{base_path}/mahabharata_book_{i:02}.json"
        try:
            with open(file_name, 'r', encoding='utf-8') as file:
                data = json.load(file)
                # Concatenate all suktas for the current book
                full_text = ""
                for sukta in data:
                    full_text += sukta["text"] + " "  # Add a space between suktas

                clean_text = full_text
                if strip_in_braces:
                    clean_text = re.sub(r"\[.*?\]", "", clean_text, flags=re.DOTALL)
                    
                if strip_diacritics:
                    clean_text = re.sub(r'[^\u0900-\u097F\s]', '', clean_text)  # Keep only Devanagari and whitespace
                    clean_text = re.sub(r'[\u0951-\u0954\u0964\u0965]+', '', clean_text)  # Remove pitch marks, Danda, Double Danda
                    clean_text = re.sub(r'[०१२३४५६७८९]+', '', clean_text)  # Remove Devanagari digits
                    clean_text = clean_text.replace(":", "")  # Remove colons
                    clean_text = clean_text.replace("?", "")  # Remove question marks
                    clean_text = clean_text.replace("!", "")  # Remove exclamation marks
                    # Add any other characters to remove here, e.g.:
                    clean_text = clean_text.replace("-", "") 
                    clean_text = clean_text.replace("'", "")
                    # ...


                clean_text = clean_text.replace("\n\n              ", " ")
                clean_text = clean_text.replace("\n", " ")
                clean_text = re.sub(r'\s+', ' ', clean_text)  # Condense multiple spaces


                if strip_spaces:
                    clean_text = clean_text.replace(" ", "")

                text_length = len(clean_text)

                selected_characters_per_round = {}
                for round_num in map(int, rounds.split(',')):
                    if not (round_num == 1 and step > text_length) and not (round_num == -1 and step > text_length):
                        if round_num > 0:
                            current_position = step - 1
                        else:
                            current_position = text_length - 1 if step == 1 else text_length - step

                        completed_rounds = 0
                        selected_characters = ""

                        while completed_rounds < abs(round_num):
                            selected_characters += clean_text[current_position % text_length]
                            current_position += step if round_num > 0 else -step

                            if (round_num > 0 and current_position >= text_length * (completed_rounds + 1)) or \
                                    (round_num < 0 and current_position < 0):
                                completed_rounds += 1

                        selected_characters_per_round[round_num] = selected_characters

                if average_compile and len(selected_characters_per_round) > 1:
                    result_text = ""
                    keys = sorted(selected_characters_per_round.keys())
                    for j in range(len(keys) - 1): # Changed i to j to avoid conflict with outer loop
                        result_text = average_gematria(selected_characters_per_round[keys[j]],
                                                        selected_characters_per_round[keys[j + 1]])
                else:
                    result_text = ''.join(selected_characters_per_round.values())

                if length != 0:
                    result_text = result_text[:length]

                translated_text = translator.translate(result_text) if result_text and translate else ""

                if result_text:
                    results.append({
                        "book": f"Rigveda {i}.",
                        "title": f"Mandala {i}",
                        "result_text": result_text,
                        "result_sum": calculate_gematria(result_text),
                        "translated_text": translated_text
                    })

        except (FileNotFoundError, json.JSONDecodeError, KeyError) as e:
            results.append({"error": f"Error processing {file_name}: {e}"})

    return results

[File Ends] hindu.py

[File Begins] populate_translations.py
import sqlite3
import logging
from deep_translator import GoogleTranslator, exceptions
from tqdm import tqdm
import threading
import time
from queue import Queue

# Constants
DATABASE_FILE = 'gematria.db'  # Use your actual database file name
BATCH_SIZE = 1000
NUM_THREADS = 10  # Number of parallel translation threads

# Set up logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# Initialize the translator
translator = GoogleTranslator(source='yi', target='en')
logging.info("Translator initialized.")

# Separate Queue and tqdm
translation_queue = Queue()  # Regular queue
translation_queue_tqdm = tqdm(total=0, dynamic_ncols=True, desc="Translation Queue")  # tqdm for the queue
total_translations_tqdm = tqdm(total=0, dynamic_ncols=True, desc="Total Translations") # tqdm for overall progress

# Lock for database access
db_lock = threading.Lock()
translations_completed = 0  # Counter for completed translations


def translate_and_store(phrase: str) -> str:
    """Translates a Hebrew phrase to English using Google Translate."""
    global translator
    max_retries = 3
    retries = 0
    while retries < max_retries:
        try:
            translation = translator.translate(phrase)
            return translation
        except (exceptions.TranslationNotFound, exceptions.NotValidPayload,
                exceptions.ServerException, exceptions.RequestError) as e:
            retries += 1
            logging.warning(f"Error translating phrase '{phrase}': {e}. Retrying... ({retries}/{max_retries})")
    logging.error(f"Failed to translate phrase '{phrase}' after {max_retries} retries.")
    return None


def translation_worker():
    """Worker thread to process translations from the queue."""
    global conn, translator, translation_queue, db_lock, translation_queue_tqdm, translations_completed, total_translations_tqdm

    while True:
        phrase = translation_queue.get()  # Get from the actual queue
        translation_queue_tqdm.update()  # Update the tqdm progress bar
        if phrase is None:  # Sentinel value to stop the thread
            break

        translation = translate_and_store(phrase)

        # Acquire the lock before any database interaction for this phrase
        with db_lock:
            with sqlite3.connect(DATABASE_FILE) as conn:
                cursor = conn.cursor()
                if translation is not None:
                    cursor.execute("UPDATE results SET translation = ? WHERE words = ?", (translation, phrase))
                    translations_completed += 1  # Increment the global counter
                    total_translations_tqdm.update()  # Update the overall progress bar
                conn.commit()

        translation_queue.task_done()


def populate_translations():
    """Populates translations for all Hebrew phrases in the database."""
    global conn, translator, translation_queue, translation_queue_tqdm, total_translations_tqdm

    with sqlite3.connect(DATABASE_FILE) as conn:
        cursor = conn.cursor()

        # Get the total count of distinct phrases needing translation
        cursor.execute("SELECT COUNT(DISTINCT words) FROM results WHERE translation IS NULL")
        total_phrases = cursor.fetchone()[0]

        logging.info(f"Found {total_phrases} distinct phrases to translate.")

        # Get distinct Hebrew phrases that need translation using a generator
        cursor.execute("SELECT DISTINCT words FROM results WHERE translation IS NULL")
        phrases_generator = (phrase for phrase, in cursor)  # Use a generator for tqdm

        # Set the total for both tqdm progress bars
        translation_queue_tqdm.total = total_phrases
        total_translations_tqdm.total = total_phrases

        # Build the translation queue first
        for phrase in phrases_generator:
            translation_queue.put(phrase)  # Put into the actual queue
            translation_queue_tqdm.update()  # Update tqdm progress bar

        # Close the translation queue tqdm after it's fully populated
        translation_queue_tqdm.close()

        # Start worker threads AFTER the queue is built
        threads = []
        for _ in range(NUM_THREADS):
            thread = threading.Thread(target=translation_worker)
            thread.start()
            threads.append(thread)

        # Wait for all tasks to be completed
        translation_queue.join()

        # Stop worker threads
        for _ in range(NUM_THREADS):
            translation_queue.put(None)  # Sentinel value to stop threads
        for thread in threads:
            thread.join()

        logging.info("All translations completed.")



def save_translations_periodically():
    """Saves translations to the database every minute."""
    while True:
        time.sleep(60)  # Wait for 1 minute
        logging.info("Saving translations to the database...")
        with db_lock:  # Acquire the lock before saving
            with sqlite3.connect(DATABASE_FILE) as conn:
                conn.commit()
        logging.info("Translations saved.")


if __name__ == "__main__":
    # Start the translation process in a separate thread
    translation_thread = threading.Thread(target=populate_translations)
    translation_thread.start()

    # Start the periodic saving thread
    save_thread = threading.Thread(target=save_translations_periodically)
    save_thread.start()

    # Keep the main thread alive
    while True:
        time.sleep(1)
[File Ends] populate_translations.py

[File Begins] quran.py
import logging
logger = logging.getLogger(__name__)

import json
import os
import re
from deep_translator import GoogleTranslator
from gematria import calculate_gematria
import math

# Hebrew gematria values for relevant characters
gematria_values = {
    'א': 1, 'ב': 2, 'ג': 3, 'ד': 4, 'ה': 5, 'ו': 6, 'ז': 7, 'ח': 8, 'ט': 9,
    'י': 10, 'כ': 20, 'ך': 500, 'ל': 30, 'מ': 40, 'ם': 600, 'נ': 50, 'ן': 700,
    'ס': 60, 'ע': 70, 'פ': 80, 'ף': 800, 'צ': 90, 'ץ': 900, 'ק': 100,
    'ר': 200, 'ש': 300, 'ת': 400
}

# Reverse dictionary for converting gematria values back to Hebrew characters
reverse_gematria_values = {v: k for k, v in gematria_values.items()}

# Function to convert a Hebrew string to its gematria values
def string_to_gematria(s):
    return [gematria_values.get(char, 0) for char in s]  # Handle characters not in the dictionary

# Function to convert a single gematria value to Hebrew characters
def gematria_to_string(value):
    result = []
    for val in sorted(reverse_gematria_values.keys(), reverse=True):
        while value >= val:
            result.append(reverse_gematria_values[val])
            value -= val
    return ''.join(result)

# Function to calculate the average gematria values of corresponding characters and convert them to Hebrew characters
def average_gematria(str1, str2):
    # Convert strings to gematria values
    gematria1 = string_to_gematria(str1)
    gematria2 = string_to_gematria(str2)

    # Handle cases where strings have different lengths by padding with 0s
    max_len = max(len(gematria1), len(gematria2))
    gematria1.extend([0] * (max_len - len(gematria1)))
    gematria2.extend([0] * (max_len - len(gematria2)))

    # Calculate the average of corresponding gematria values and apply math.ceil
    average_gematria_values = [math.ceil((g1 + g2) / 2) for g1, g2 in zip(gematria1, gematria2)]

    # Convert the average gematria values back to Hebrew characters
    return ''.join(gematria_to_string(val) for val in average_gematria_values)


def process_json_files(start=1, end=114, step=1, rounds="1", length=0, tlang="en", strip_spaces=True,
                       strip_in_braces=True, strip_diacritics=True, average_compile=False, translate=False):
    base_path = "texts/quran"
    translator = GoogleTranslator(source='ar', target=tlang)
    results = []

    for i in range(start, end + 1):
        file_name = f"{base_path}/{i:03}.json"  # Updated file name formatting
        try:
            with open(file_name, 'r', encoding='utf-8') as file:
                data = json.load(file)

                # Extract text from verses
                full_text = ""
                for verse_key, verse_text in data["verse"].items():
                    full_text += verse_text + " "

                full_text = full_text.replace("\ufeff", "")

                clean_text = full_text
                if strip_in_braces:
                    clean_text = re.sub(r"\[.*?\]", "", clean_text, flags=re.DOTALL)
                if strip_diacritics:
                    clean_text = re.sub(
                        r"[\u0610-\u061A\u064B-\u065F\u0670\u06D6-\u06DC\u06DF-\u06E4\u06E7\u06E8\u06EA-\u06ED]+", "",
                        clean_text)
                if strip_spaces:
                    clean_text = clean_text.replace(" ", "")
                else:
                    clean_text = clean_text.replace("  ", " ")
                    clean_text = clean_text.replace("  ", " ")
                    clean_text = clean_text.replace("  ", " ")

                text_length = len(clean_text)

                selected_characters_per_round = {}
                for round_num in map(int, rounds.split(',')):
                    # Handle cases where no characters should be selected
                    if not (round_num == 1 and step > text_length) and not (round_num == -1 and step > text_length):
                        # Corrected logic for negative rounds and step = 1
                        if round_num > 0:
                            current_position = step - 1
                        else:
                            current_position = text_length - 1 if step == 1 else text_length - step

                        completed_rounds = 0
                        selected_characters = ""

                        while completed_rounds < abs(round_num):
                            selected_characters += clean_text[current_position % text_length]

                            # Update current_position based on the sign of rounds
                            current_position += step if round_num > 0 else -step

                            if (round_num > 0 and current_position >= text_length * (completed_rounds + 1)) or \
                                    (round_num < 0 and current_position < 0):
                                completed_rounds += 1

                        selected_characters_per_round[round_num] = selected_characters

                if average_compile and len(selected_characters_per_round) > 1:
                    result_text = ""
                    keys = sorted(selected_characters_per_round.keys())
                    for i in range(len(keys) - 1):
                        result_text = average_gematria(selected_characters_per_round[keys[i]],
                                                       selected_characters_per_round[keys[i + 1]])
                else:
                    result_text = ''.join(selected_characters_per_round.values())

                if length != 0:
                    result_text = result_text[:length]

                translated_text = translator.translate(result_text) if result_text and translate else ""

                if result_text:  # Only append if result_text is not empty
                    results.append({
                        "book": f"Quran {i}.",
                        "title": data["name"],  # Use "name" instead of "title"
                        "result_text": result_text,
                        "result_sum": calculate_gematria(result_text),
                        "translated_text": translated_text
                    })

        except FileNotFoundError:
            results.append({"error": f"File {file_name} not found."})
        except json.JSONDecodeError as e:
            results.append({"error": f"File {file_name} could not be read as JSON: {e}"})
        except KeyError as e:
            results.append({"error": f"Expected key 'verse' is missing in {file_name}: {e}"})  # Updated key

    return results


# Tests
test_results = [
    #(process_json_files(0, 0, 21, rounds="3", length=0), "שרק"),
    #(process_json_files(0, 0, 22, rounds="1", length=0), "ת"), 
    #(process_json_files(0, 0, 22, rounds="3", length=0), "תתת"),
    #(process_json_files(0, 0, 23, rounds="3", length=0), "אבג"),
    #(process_json_files(0, 0, 11, rounds="1", length=0), "כת"),
    #(process_json_files(0, 0, 2, rounds="1", length=0), "בדוחילנעצרת"),
    #(process_json_files(0, 0, 23, rounds="1", length=0), None),  # Expect None, when no results
    #(process_json_files(0, 0, 23, rounds="-1", length=0), None),  # Expect None, when no results
    #(process_json_files(0, 0, 22, rounds="-1", length=0), "א"),
    #(process_json_files(0, 0, 22, rounds="-2", length=0), "אא"),
    #(process_json_files(0, 0, 1, rounds="-1", length=0), "תשרקצפעסנמלכיטחזוהדגבא"), # Reversed Hebrew alphabet
    #(process_json_files(0, 0, 1, rounds="1,-1", length=0), "אבגדהוזחטיכלמנסעפצקרשתתשרקצפעסנמלכיטחזוהדגבא"), # Combined rounds
    #(process_json_files(0, 0, 22, rounds="1,-1", length=0, average_compile=True), "רא"),  # average compile test (400+1) / 2 = math.ceil(200.5)=201=200+1="רא"
]

all_tests_passed = True
for result, expected in test_results:
    if expected is None:  # Check if no result is expected
        if not result:
            logger.info(f"Test passed: Expected no results, got no results.")
        else:
            logger.error(f"Test failed: Expected no results, but got: {result}")
            all_tests_passed = False
    else:
        # Check if result is not empty before accessing elements
        if result: 
            #result_text = result[0]['result_text']
            result_text = None
            if result_text == expected:
                logger.info(f"Test passed: Expected '{expected}', got '{result_text}'")
            else:
                logger.error(f"Test failed: Expected '{expected}', but got '{result_text}'")
                all_tests_passed = False
        else:
            logger.error(f"Test failed: Expected '{expected}', but got no results")
            all_tests_passed = False

if all_tests_passed:
    logger.info("All round tests passed.")

[File Ends] quran.py

[File Begins] requirements-all.txt
aiofiles==23.2.1
annotated-types==0.7.0
anyio==4.6.2.post1
beautifulsoup4==4.12.3
certifi==2024.8.30
charset-normalizer==3.4.0
click==8.1.7
deep-translator==1.11.4
exceptiongroup==1.2.2
fastapi==0.115.4
ffmpy==0.4.0
filelock==3.16.1
fsspec==2024.10.0
fuzzywuzzy==0.18.0
gradio==5.5.0
gradio_calendar==0.0.6
gradio_client==1.4.2
h11==0.14.0
httpcore==1.0.6
httpx==0.27.2
huggingface-hub==0.26.2
idna==3.10
inflect==7.4.0
Jinja2==3.1.4
Levenshtein==0.26.1
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
more-itertools==10.5.0
numpy==2.1.3
orjson==3.10.11
packaging==24.2
pandas==2.2.3
pillow==11.0.0
pydantic==2.9.2
pydantic_core==2.23.4
pydub==0.25.1
Pygments==2.18.0
python-dateutil==2.9.0.post0
python-Levenshtein==0.26.1
python-multipart==0.0.12
pytz==2024.2
PyYAML==6.0.2
RapidFuzz==3.10.1
requests==2.32.3
rich==13.9.4
ruff==0.7.3
safehttpx==0.1.1
semantic-version==2.10.0
shellingham==1.5.4
six==1.16.0
sniffio==1.3.1
soupsieve==2.6
starlette==0.41.2
tabulate==0.9.0
tomlkit==0.12.0
tqdm==4.67.0
transliterate==1.10.2
typeguard==4.4.1
typer==0.13.0
typing_extensions==4.12.2
tzdata==2024.2
urllib3==2.2.3
uvicorn==0.32.0
websockets==12.0

[File Ends] requirements-all.txt

[File Begins] requirements.txt
gradio==5.5.0
deep_translator==1.11.4
tabulate==0.9.0
gradio_calendar==0.0.6
inflect==7.4.0
fuzzywuzzy==0.18.0
python-Levenshtein==0.26.1
transliterate==1.10.2
[File Ends] requirements.txt

[File Begins] torah.py
import logging
logger = logging.getLogger(__name__)

import json
import os
import re
from deep_translator import GoogleTranslator
from gematria import calculate_gematria
import math

# Hebrew gematria values for relevant characters
gematria_values = {
    'א': 1, 'ב': 2, 'ג': 3, 'ד': 4, 'ה': 5, 'ו': 6, 'ז': 7, 'ח': 8, 'ט': 9,
    'י': 10, 'כ': 20, 'ך': 500, 'ל': 30, 'מ': 40, 'ם': 600, 'נ': 50, 'ן': 700,
    'ס': 60, 'ע': 70, 'פ': 80, 'ף': 800, 'צ': 90, 'ץ': 900, 'ק': 100,
    'ר': 200, 'ש': 300, 'ת': 400
}

# Reverse dictionary for converting gematria values back to Hebrew characters
reverse_gematria_values = {v: k for k, v in gematria_values.items()}

# Function to convert a Hebrew string to its gematria values
def string_to_gematria(s):
    return [gematria_values.get(char, 0) for char in s]  # Handle characters not in the dictionary

# Function to convert a single gematria value to Hebrew characters
def gematria_to_string(value):
    result = []
    for val in sorted(reverse_gematria_values.keys(), reverse=True):
        while value >= val:
            result.append(reverse_gematria_values[val])
            value -= val
    return ''.join(result)

# Function to calculate the average gematria values of corresponding characters and convert them to Hebrew characters
def average_gematria(str1, str2):
    # Convert strings to gematria values
    gematria1 = string_to_gematria(str1)
    gematria2 = string_to_gematria(str2)

    # Handle cases where strings have different lengths by padding with 0s
    max_len = max(len(gematria1), len(gematria2))
    gematria1.extend([0] * (max_len - len(gematria1)))
    gematria2.extend([0] * (max_len - len(gematria2)))

    # Calculate the average of corresponding gematria values and apply math.ceil
    average_gematria_values = [math.ceil((g1 + g2) / 2) for g1, g2 in zip(gematria1, gematria2)]

    # Convert the average gematria values back to Hebrew characters
    return ''.join(gematria_to_string(val) for val in average_gematria_values)

def process_json_files(start, end, step, rounds="1", length=0, tlang="en", strip_spaces=True, strip_in_braces=True, strip_diacritics=True, average_compile=False):
    base_path = "texts/torah"
    translator = GoogleTranslator(source='auto', target=tlang)
    results = []

    for i in range(start, end + 1):
        file_name = f"{base_path}/{i:02}.json"
        try:
            with open(file_name, 'r', encoding='utf-8') as file:
                data = json.load(file)
                text_blocks = data["text"]

                full_text = ""
                for block in text_blocks:
                    full_text += ' '.join(block)

                clean_text = full_text
                if strip_in_braces:
                    clean_text = re.sub(r"\[.*?\]", "", clean_text, flags=re.DOTALL)
                if strip_diacritics:
                    clean_text = re.sub(r"[^\u05D0-\u05EA ]+", "", clean_text)
                if strip_spaces:
                    clean_text = clean_text.replace(" ", "")
                else:
                    clean_text = clean_text.replace("  ", " ")
                    clean_text = clean_text.replace("  ", " ")
                    clean_text = clean_text.replace("  ", " ")

                text_length = len(clean_text)
                
                selected_characters_per_round = {}
                for round_num in map(int, rounds.split(',')):
                    # Handle cases where no characters should be selected
                    if not (round_num == 1 and step > text_length) and not (round_num == -1 and step > text_length):
                        # Corrected logic for negative rounds and step = 1
                        if round_num > 0:
                            current_position = step - 1 
                        else:
                            current_position = text_length - 1 if step == 1 else text_length - step

                        completed_rounds = 0
                        selected_characters = ""  

                        while completed_rounds < abs(round_num):
                            selected_characters += clean_text[current_position % text_length]

                            # Update current_position based on the sign of rounds
                            current_position += step if round_num > 0 else -step

                            if (round_num > 0 and current_position >= text_length * (completed_rounds + 1)) or \
                               (round_num < 0 and current_position < 0):
                                completed_rounds += 1

                        selected_characters_per_round[round_num] = selected_characters
                
                if average_compile and len(selected_characters_per_round) > 1:
                    result_text = ""
                    keys = sorted(selected_characters_per_round.keys())
                    for i in range(len(keys) - 1):
                        result_text = average_gematria(selected_characters_per_round[keys[i]], selected_characters_per_round[keys[i+1]])
                else:
                    result_text = ''.join(selected_characters_per_round.values())

                if length != 0:
                    result_text = result_text[:length]

                translated_text = translator.translate(result_text) if result_text else ""

                if result_text:  # Only append if result_text is not empty
                    results.append({
                        "book": f"Torah {i}.",
                        "title": data["title"],
                        "result_text": result_text,
                        "result_sum": calculate_gematria(result_text),
                        "translated_text": translated_text
                    })

        except FileNotFoundError:
            results.append({"error": f"File {file_name} not found."})
        except json.JSONDecodeError as e:
            results.append({"error": f"File {file_name} could not be read as JSON: {e}"})
        except KeyError as e:
            results.append({"error": f"Expected key 'text' is missing in {file_name}: {e}"})

    return results


# Tests
test_results = [
    #(process_json_files(0, 0, 21, rounds="3", length=0), "שרק"),
    #(process_json_files(0, 0, 22, rounds="1", length=0), "ת"), 
    #(process_json_files(0, 0, 22, rounds="3", length=0), "תתת"),
    #(process_json_files(0, 0, 23, rounds="3", length=0), "אבג"),
    #(process_json_files(0, 0, 11, rounds="1", length=0), "כת"),
    #(process_json_files(0, 0, 2, rounds="1", length=0), "בדוחילנעצרת"),
    #(process_json_files(0, 0, 23, rounds="1", length=0), None),  # Expect None, when no results
    #(process_json_files(0, 0, 23, rounds="-1", length=0), None),  # Expect None, when no results
    #(process_json_files(0, 0, 22, rounds="-1", length=0), "א"),
    #(process_json_files(0, 0, 22, rounds="-2", length=0), "אא"),
    #(process_json_files(0, 0, 1, rounds="-1", length=0), "תשרקצפעסנמלכיטחזוהדגבא"), # Reversed Hebrew alphabet
    #(process_json_files(0, 0, 1, rounds="1,-1", length=0), "אבגדהוזחטיכלמנסעפצקרשתתשרקצפעסנמלכיטחזוהדגבא"), # Combined rounds
    #(process_json_files(0, 0, 22, rounds="1,-1", length=0, average_compile=True), "רא"),  # average compile test (400+1) / 2 = math.ceil(200.5)=201=200+1="רא"
]

all_tests_passed = True
for result, expected in test_results:
    if expected is None:  # Check if no result is expected
        if not result:
            logger.info(f"Test passed: Expected no results, got no results.")
        else:
            logger.error(f"Test failed: Expected no results, but got: {result}")
            all_tests_passed = False
    else:
        # Check if result is not empty before accessing elements
        if result: 
            result_text = result[0]['result_text']
            if result_text == expected:
                logger.info(f"Test passed: Expected '{expected}', got '{result_text}'")
            else:
                logger.error(f"Test failed: Expected '{expected}', but got '{result_text}'")
                all_tests_passed = False
        else:
            logger.error(f"Test failed: Expected '{expected}', but got no results")
            all_tests_passed = False

if all_tests_passed:
    logger.info("All round tests passed.")

[File Ends] torah.py

[File Begins] translation_utils.py
# translation_utils.py

import logging
import sqlite3
from concurrent.futures import ThreadPoolExecutor
import functools
from deep_translator import GoogleTranslator, exceptions

# Constants
TRANSLATION_DATABASE_FILE = 'translation_database.db'
SUPPORTED_LANGUAGES = {"af", "sq", "am", "ar", "hy", "az", "eu", "be", "bn", "bs", "bg", "ca", "ceb", "ny", "zh-CN", "zh-TW", "co", "hr", "cs", "da", "nl", "en", "eo", "et", "tl", "fi", "fr", "fy", "gl", "ka", "de", "el", "gu", "ht", "ha", "haw", "iw", "hi", "hmn", "hu", "is", "ig", "id", "ga", "it", "ja", "jw", "kn", "kk", "km", "ko", "ku", "ky", "lo", "la", "lv", "lt", "lb", "mk", "mg", "ms", "ml", "mt", "mi", "mr", "mn", "my", "ne", "no", "ps", "fa", "pl", "pt", "pa", "ro", "ru", "sm", "gd", "sr", "st", "sn", "sd", "si", "sk", "sl", "so", "es", "su", "sw", "sv", "tg", "ta", "te", "th", "tr", "uk", "ur", "uz", "vi", "cy", "xh", "yi", "yo", "zu"}

# Initialize logger
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)


def create_translation_table():
    """Creates the translation table if it doesn't exist."""
    try:
        with sqlite3.connect(TRANSLATION_DATABASE_FILE) as conn:
            cursor = conn.cursor()
            cursor.execute('''
                CREATE TABLE IF NOT EXISTS translations (
                    phrase TEXT PRIMARY KEY
                )
            ''')

            # Dynamically add language columns
            cursor.execute("PRAGMA table_info(translations)")
            existing_columns = {col[1] for col in cursor.fetchall()}
            for lang_code in SUPPORTED_LANGUAGES:
                column_name = lang_code.replace('-', '_')
                if column_name == "is":  # Handle reserved keywords in SQLite
                    column_name = "is_"
                if column_name not in existing_columns:
                    try:
                        cursor.execute(f"ALTER TABLE translations ADD COLUMN `{column_name}` TEXT")
                        logger.info(f"Added column '{column_name}' to translations table.")
                    except sqlite3.OperationalError as e:
                        if "duplicate column name" in str(e).lower():
                            logger.debug(f"Column '{column_name}' already exists. Skipping.")
                        else:
                            logger.error(f"Error adding column '{column_name}': {e}")  # More specific error
            conn.commit()
    except Exception as e:  # Broad exception handling to catch any other potential issues
        logger.error(f"An unexpected error occurred in create_translation_table: {e}")


@functools.lru_cache(maxsize=1000)  # Use the correct decorator name
def translate_cached(text, target_language, source_language="auto"):  # Renamed to avoid conflicts
    """Translates text using Google Translate with caching."""
    if not text:
        return ""

    try:
        translator = GoogleTranslator(source=source_language, target=target_language)
        translated = translator.translate(text)
        return translated
    except exceptions.TranslationNotFound:
        logger.error(f"Translation not found for: {text}")
    except Exception as e:  # Catch generic exceptions
        logger.exception(f"Translation error: {e}")  # Log with traceback
    return None


def get_translation(phrase, target_language, source_language="auto"):
    """Retrieves a translation from the database or translates and stores it."""
    if target_language not in SUPPORTED_LANGUAGES:
        logger.error(f"Unsupported target language: {target_language}")
        return None, False  # Return None and False for failure

    try:
        with sqlite3.connect(TRANSLATION_DATABASE_FILE) as conn:
            cursor = conn.cursor()
            column_name = target_language.replace('-', '_')
            if column_name == "is":
                column_name = "is_"

            cursor.execute(f"SELECT `{column_name}` FROM translations WHERE phrase=?", (phrase,))
            result = cursor.fetchone()

            if result and result[0]:  # Check that the result is not empty and has a translated value
                return result[0], True

            translated_text = translate_cached(phrase, target_language, source_language)
            if translated_text:
                cursor.execute(f"""
                    INSERT INTO translations (phrase, `{column_name}`) VALUES (?, ?)
                    ON CONFLICT(phrase) DO UPDATE SET `{column_name}`=excluded.`{column_name}`
                """, (phrase, translated_text))

                conn.commit()
                return translated_text, True
            else:
                return None, False  # Explicitly return False when translation fails


    except sqlite3.Error as e:
        logger.error(f"Database error: {e}")
        return None, False # Return explicit failure indicator
    except Exception as e:
        logger.exception(f"Unexpected error in get_translation: {e}") # Generic Exception Catch-All
        return None, False  # Return explicit failure indicator


def batch_translate(phrases, target_language, source_language="auto"):
    """Translates multiple phrases concurrently."""
    phrases_to_translate = [phrase for phrase in phrases if phrase]

    with ThreadPoolExecutor() as executor:
        futures = [executor.submit(get_translation, phrase, target_language, source_language)
                   for phrase in phrases_to_translate]
        results = [future.result() for future in futures]

    translations = {phrase: translation for phrase, (translation, _) in zip(phrases_to_translate, results)}
    for phrase in phrases:
        if not phrase:
            translations[phrase] = None
    return translations

[File Ends] translation_utils.py

[File Begins] tripitaka.py
# TODO: Auto-tune by step / 2 , 4, 8, 16, 32, 64, so that the translation result has spaces
import logging
logger = logging.getLogger(__name__)

import json
import os
import re
from deep_translator import GoogleTranslator
from gematria import calculate_gematria
import math
import glob
import math

def process_json_files(start=1, end=52, step=1, rounds="1,-1", length=0, tlang="en", strip_spaces=True,
                       strip_in_braces=True, strip_diacritics=True, average_compile=False, translate=False):
    base_path = "texts/tripitaka"
    translator = GoogleTranslator(source='ne', target=tlang)  # Assuming Nepali for tripitaka as well
    results = []

    for i in range(start, end + 1):
        file_pattern = f"{base_path}/{i:02}*.json"  # Using glob for flexible file names
        for file_name in glob.glob(file_pattern):
            try:
                with open(file_name, 'r', encoding='utf-8') as file:
                    data = json.load(file)
                    full_text = ""
                    for gatha in data.get("gathas", []): # Accessing 'gathas' safely
                        full_text += gatha + " "

                    clean_text = full_text
                    if strip_in_braces:
                        clean_text = re.sub(r"\[.*?\]", "", clean_text, flags=re.DOTALL)

                    if strip_diacritics:
                        clean_text = re.sub(r'[^\u0900-\u097F\s]', '', clean_text)
                        clean_text = re.sub(r'[\u0951-\u0954\u0964\u0965]+', '', clean_text)
                        clean_text = re.sub(r'[०१२३४५६७८९]+', '', clean_text)
                        clean_text = clean_text.replace(":", "")
                        clean_text = clean_text.replace("?", "")
                        clean_text = clean_text.replace("!", "")
                        clean_text = clean_text.replace("-", "")
                        clean_text = clean_text.replace("'", "")


                    clean_text = clean_text.replace("\n\n              ", " ")
                    clean_text = clean_text.replace("\n", " ")
                    clean_text = re.sub(r'\s+', ' ', clean_text)

                    if strip_spaces:
                        clean_text = clean_text.replace(" ", "")

                    text_length = len(clean_text)
                    #step=math.ceil(step/64)
                    selected_characters_per_round = {}
                    for round_num in map(int, rounds.split(',')):
                        if not (round_num == 1 and step > text_length) and not (round_num == -1 and step > text_length):
                            if round_num > 0:
                                current_position = step - 1
                            else:
                                current_position = text_length - 1 if step == 1 else text_length - step

                            completed_rounds = 0
                            selected_characters = ""

                            while completed_rounds < abs(round_num):
                                selected_characters += clean_text[current_position % text_length]
                                current_position += step if round_num > 0 else -step

                                if (round_num > 0 and current_position >= text_length * (completed_rounds + 1)) or \
                                   (round_num < 0 and current_position < text_length * completed_rounds -1): # corrected condition here
                                    completed_rounds += 1

                            selected_characters_per_round[round_num] = selected_characters

                    if average_compile and len(selected_characters_per_round) > 1:
                        result_text = ""
                        keys = sorted(selected_characters_per_round.keys())
                        for j in range(len(keys) - 1): # Changed i to j to avoid conflict with outer loop
                            result_text = average_gematria(selected_characters_per_round[keys[j]],
                                                        selected_characters_per_round[keys[j + 1]])
                    else:
                        result_text = ''.join(selected_characters_per_round.values())

                    if length != 0:
                        result_text = result_text[:length]

                    translated_text = translator.translate(result_text) if result_text and translate else ""

                    if result_text:
                        results.append({
                            "book": f"Tripitaka {i}.",
                            "title": f'{data.get("title")} {data.get("book_name")} {data.get("chapter")}',
                            "result_text": result_text,
                            "result_sum": calculate_gematria(result_text),
                            "translated_text": translated_text
                        })

            except (FileNotFoundError, json.JSONDecodeError, KeyError) as e:
                results.append({"error": f"Error processing {file_name}: {e}"})

    return results

[File Ends] tripitaka.py

[File Begins] util.py
import json
import re

def process_json_files(start, end):
    """
    Processes JSON files containing Tanach text and returns a dictionary
    mapping book IDs to their data.

    Args:
        start: The starting book ID (inclusive).
        end: The ending book ID (inclusive).

    Returns:
        A dictionary where keys are book IDs and values are dictionaries
        containing 'title' and 'text' fields.
    """
    base_path = "texts"
    results = {}  # Use a dictionary to store results

    for i in range(start, end + 1):
        file_name = f"{base_path}/{i:02}.json"
        try:
            with open(file_name, 'r', encoding='utf-8') as file:
                data = json.load(file)
                if data:
                    # Store book ID as key and book data as value
                    results[i] = {"title": data.get("title", "No title"), "text": data.get("text", [])}

        except FileNotFoundError:
            logging.warning(f"File {file_name} not found.")
        except json.JSONDecodeError as e:
            logging.warning(f"File {file_name} could not be read as JSON: {e}")
        except KeyError as e:
            logging.warning(f"Expected key 'text' is missing in {file_name}: {e}")

    return results
[File Ends] util.py

[File Begins] utils.py
import logging
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

import inflect
from datetime import datetime, date
from deep_translator import GoogleTranslator

# Custom function to convert number to ordinal words
def number_to_ordinal_word(number):
    ordinal_dict = {
        1: "first", 2: "second", 3: "third", 4: "fourth", 5: "fifth",
        6: "sixth", 7: "seventh", 8: "eighth", 9: "ninth", 10: "tenth",
        11: "eleventh", 12: "twelfth", 13: "thirteenth", 14: "fourteenth",
        15: "fifteenth", 16: "sixteenth", 17: "seventeenth", 18: "eighteenth",
        19: "nineteenth", 20: "twentieth", 21: "twentyfirst", 22: "twentysecond",
        23: "twentythird", 24: "twentyfourth", 25: "twentyfifth",
        26: "twentysixth", 27: "twentyseventh", 28: "twentyeighth",
        29: "twentyninth", 30: "thirtieth", 31: "thirtyfirst"
    }
    return ordinal_dict.get(number, "")

def custom_normalize(text):
    mappings = {
        'ü': 'ue', 'ö': 'oe', 'ä': 'ae', 'ß': 'ss', 'Ü': 'Ue', 'Ö': 'Oe', 'Ä': 'Ae',
        'á': 'a', 'à': 'a', 'â': 'a', 'ã': 'a', 'å': 'aa', 'ā': 'a', 'ă': 'a', 'ą': 'a',
        'Á': 'A', 'À': 'A', 'Â': 'A', 'Ã': 'A', 'Å': 'Aa', 'Ā': 'A', 'Ă': 'A', 'Ą': 'A',
        'é': 'e', 'è': 'e', 'ê': 'e', 'ë': 'e', 'ē': 'e', 'ĕ': 'e', 'ė': 'e', 'ę': 'e', 'ě': 'e',
        'É': 'E', 'È': 'E', 'Ê': 'E', 'Ë': 'E', 'Ē': 'E', 'Ĕ': 'E', 'Ė': 'E', 'Ę': 'E', 'Ě': 'E',
        'í': 'i', 'ì': 'i', 'î': 'i', 'ï': 'i', 'ī': 'i', 'ĭ': 'i', 'į': 'i', 'ı': 'i',
        'Í': 'I', 'Ì': 'I', 'Î': 'I', 'Ï': 'I', 'Ī': 'I', 'Ĭ': 'I', 'Į': 'I', 'I': 'I',
        'ó': 'o', 'ò': 'o', 'ô': 'o', 'õ': 'o', 'ø': 'oe', 'ō': 'o', 'ŏ': 'o', 'ő': 'o',
        'Ó': 'O', 'Ò': 'O', 'Ô': 'O', 'Õ': 'O', 'Ø': 'Oe', 'Ō': 'O', 'Ŏ': 'O', 'Ő': 'O',
        'ú': 'u', 'ù': 'u', 'û': 'u', 'ū': 'u', 'ŭ': 'u', 'ů': 'u', 'ű': 'u', 'ų': 'u',
        'Ú': 'U', 'Ù': 'U', 'Û': 'U', 'Ü': 'Ue', 'Ū': 'U', 'Ŭ': 'U', 'Ů': 'U', 'Ű': 'U', 'Ų': 'U',
        'ç': 'c', 'ć': 'c', 'ĉ': 'c', 'ċ': 'c', 'č': 'c',
        'Ç': 'C', 'Ć': 'C', 'Ĉ': 'C', 'Ċ': 'C', 'Č': 'C',
        'ñ': 'n', 'ń': 'n', 'ņ': 'n', 'ň': 'n', 'ŋ': 'n',
        'Ñ': 'N', 'Ń': 'N', 'Ņ': 'N', 'Ň': 'N', 'Ŋ': 'N',
        'ý': 'y', 'ÿ': 'y', 'ŷ': 'y',
        'Ý': 'Y', 'Ÿ': 'Y', 'Ŷ': 'Y',
        'ž': 'zh', 'ź': 'z', 'ż': 'z',
        'Ž': 'Zh', 'Ź': 'Z', 'Ż': 'Z',
        'ð': 'd', 'Ð': 'D', 'þ': 'th', 'Þ': 'Th', 'ł': 'l', 'Ł': 'L', 'đ': 'd', 'Đ': 'D',
        'æ': 'ae', 'Æ': 'Ae', 'œ': 'oe', 'Œ': 'Oe',
        'ś': 's', 'ŝ': 's', 'ş': 's', 'š': 's',
        'Ś': 'S', 'Ŝ': 'S', 'Ş': 'S', 'Š': 'S',
        'ť': 't', 'ţ': 't', 'ŧ': 't', 'Ť': 'T', 'Ţ': 'T', 'Ŧ': 'T',
        'ŕ': 'r', 'ř': 'r', 'Ŕ': 'R', 'Ř': 'R',
        'ľ': 'l', 'ĺ': 'l', 'ļ': 'l', 'ŀ': 'l',
        'Ľ': 'L', 'Ĺ': 'L', 'Ļ': 'L', 'Ŀ': 'L',
        'ē': 'e', 'Ē': 'E',
        'ň': 'n', 'Ň': 'N',
        'ğ': 'g', 'Ğ': 'G',
        'ġ': 'g', 'Ġ': 'G',
        'ħ': 'h', 'Ħ': 'H',
        'ı': 'i', 'İ': 'I',
        'ĵ': 'j', 'Ĵ': 'J',
        'ķ': 'k', 'Ķ': 'K',
        'ļ': 'l', 'Ļ': 'L',
        'ņ': 'n', 'Ņ': 'N',
        'ŧ': 't', 'Ŧ': 'T',
        'ŭ': 'u', 'Ŭ': 'U'
    }
    for key, value in mappings.items():
        text = text.replace(key, value)
    return text




# Convert a date to words with an ordinal day
def date_to_words(date_obj):
    inf_engine = inflect.engine()

    if isinstance(date_obj, (date, datetime)):
        year = date_obj.year
        month = date_obj.strftime("%B") if hasattr(date_obj, 'month') else None
        day_ordinal = number_to_ordinal_word(date_obj.day) if hasattr(date_obj, 'day') else None
    elif isinstance(date_obj, str):
        try:
            # Attempt to parse full date first
            date_obj = datetime.strptime(date_obj, "%Y-%m-%d")
            year = date_obj.year
            month = date_obj.strftime("%B")
            day_ordinal = number_to_ordinal_word(date_obj.day)
        except ValueError:
            try:
                # Try year-month
                date_obj = datetime.strptime(date_obj, "%Y-%m")
                year = date_obj.year
                month = date_obj.strftime("%B")
                day_ordinal = None
            except ValueError:
                try:
                    # Try just year
                    date_obj = datetime.strptime(date_obj, "%Y")
                    year = date_obj.year
                    month = None
                    day_ordinal = None
                except ValueError:
                    raise ValueError("Invalid date format. Use YYYY-MM-DD, YYYY-MM, or YYYY.")
    else:
        raise TypeError("date_obj must be a datetime, date, or string object.")


    if 1900 <= year <= 1999:
        year_words = f"{inf_engine.number_to_words(year // 100, andword='')} hundred"
        if year % 100 != 0:
            year_words += f" {inf_engine.number_to_words(year % 100, andword='')}"
    else:
        year_words = inf_engine.number_to_words(year, andword='')
    year_formatted = year_words.replace(',', '')

    parts = []
    if day_ordinal:
        parts.append(day_ordinal)
    if month:
        parts.append(month)
    parts.append(year_formatted)

    return " ".join(parts)



def translate_date_to_words(date, lang='en'):
    """Converts a date to words in the specified language."""
    if date is None:
        return "No date selected"
    
    date_string = date.strftime("%Y-%m-%d")
    logger.info(f"Date string: {date_string}")
    
    date_in_words = date_to_words(date_string)
    logger.info(f"Date in words: {date_in_words}")
    
    translator = GoogleTranslator(source='auto', target=lang)
    translated_date_words = translator.translate(date_in_words)
    logger.info(f"Translated date words: {translated_date_words}")
    
    # Normalize the text if it contains any special characters
    translated_date_words = custom_normalize(translated_date_words)
    logger.info(f"Normalized date words: {translated_date_words}")
    
    return translated_date_words

[File Ends] utils.py


<-- File Content Ends


Instruction: Okay Echo, now you know the code. Please give me a function overview of the code, with headers and subheaders, in Markdown format. And in the following conversation, you stick to that main function overview, so long as it is feasible.