add files

.gitignore

@@ -0,0 +1,168 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+Pipfile
+Pipfile.lock
+
+data/
+bounding_box_images/

Dockerfile

@@ -1,7 +1,7 @@
 # read the doc: https://huggingface.co/docs/hub/spaces-sdks-docker
 # you will also find guides on how best to write your Dockerfile
 
-FROM python:3.9
+FROM python:3.11
 
 WORKDIR /code
 
@@ -9,6 +9,20 @@ COPY ./requirements.txt /code/requirements.txt
 
 RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
 
+# Install espeak
+RUN apt-get install -y espeak
+
 COPY . .
 
-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "7860"]
+RUN useradd -m -u 1000 user
+
+USER user
+
+ENV HOME=/home/user \
+	PATH=/home/user/.local/bin:$PATH
+
+WORKDIR $HOME/app
+
+COPY --chown=user . $HOME/app
+
+CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "7860"]

README.md

@@ -1,10 +1,83 @@
 ---
 title: Manga Translator
-emoji: 🌖
-colorFrom: red
-colorTo: red
+emoji: 📖
+colorFrom: pink
+colorTo: yellow
 sdk: docker
 pinned: false
 ---
 
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# Manga Translator
+
+- [Manga Translator](#manga-translator)
+  - [Introduction](#introduction)
+  - [Approach](#approach)
+    - [Data Collection](#data-collection)
+    - [Yolov8](#yolov8)
+    - [Manga-ocr](#manga-ocr)
+    - [Deep-translator](#deep-translator)
+  - [Server](#server)
+  - [Demo](#demo)
+
+## Introduction
+
+I love reading manga, and I can't wait for the next chapter of my favorite manga to be released. However, the newest chapters are usually in Japanese, and they are translated to English after some time. I want to read the newest chapters as soon as possible, so I decided to build a manga translator that can translate Japanese manga to English.
+
+## Approach
+
+I want to translate the text in the manga images from Japanese to English. I will first need to know where these speech bubbles are on the image. For this I will use `Yolov8` to detect the speech bubbles. Once I have the speech bubbles, I will use `manga-ocr` to extract the text from the speech bubbles. Finally, I will use `deep-translator` to translate the text from Japanese to English.
+
+![Manga Translator](./assets/MangaTranslator.png)
+
+### Data Collection
+
+This [dataset](https://universe.roboflow.com/speechbubbledetection-y9yz3/bubble-detection-gbjon/dataset/2#) contains over 8500 images of manga pages together with their annotations from Roboflow. I will use this dataset to train `Yolov8` to detect the speech bubbles in the manga images. To use this dataset with Yolov8, I will need to convert the annotations to the YOLO format, which is a text file containing the class label and the bounding box coordinates of the object in the image.
+
+This dataset is over 1.7GB in size, so I will need to download it to my local machine. The rest of the code should be run after the dataset has been downloaded and extracted in this directory.
+
+The dataset contains mostly English manga, but that is fine since I am only interested in the speech bubbles.
+
+### Yolov8
+
+`Yolov8` is a state-of-the-art, real-time object detection system [that I've used in the past before](https://github.com/Detopall/parking-lot-prediction). I will use `Yolov8` to detect the speech bubbles in the manga images.
+
+### Manga-ocr
+
+Optical character recognition for Japanese text, with the main focus being Japanese manga. This Python package is built and trained specifically for extracting text from manga images. This makes it perfect for extracting text from the speech bubbles in the manga images.
+
+### Deep-translator
+
+`Deep-translator` is a Python package that uses the Google Translate API to translate text from one language to another. I will use `deep-translator` to translate the text extracted from the manga images from Japanese to English.
+
+## Server
+
+I created a simple server and client using FastAPI. The server will receive the manga image from the client, detect the speech bubbles, extract the text from the speech bubbles, and translate the text from Japanese to English. The server will then send the translated text back to the client.
+
+To run the server, you will need to install the required packages. You can do this by running the following command:
+
+```bash
+pip install -r requirements.txt
+```
+
+You can then start the server by running the following command:
+
+```bash
+python app.py
+```
+
+The server will start running on `http://localhost:8000`. You can then send a POST request to `http://localhost:8000/predict` with the manga image as the request body.
+
+```json
+POST /predict
+{
+  "image": "base64_encoded_image"
+}
+```
+
+## Demo
+
+The following video is a screen recording of the client sending a manga image to the server, and the server detecting the speech bubbles, extracting the text, and translating the text from Japanese to English.
+
+[![Manga Translator](./assets/MangaTranslator.png)](https://www.youtube.com/watch?v=aEOpx1_CUGI)

fonts/GL-NovantiquaMinamoto.ttf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6dd6f8754f1e5ffd3c21a514c15b6bcb0cd5893a4cb60fcd51f795eb9167d4ce
+size 10225352

fonts/mangat.ttf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7de5680ca9a17be79d6b311c1010865e4352da94f3512f6f1738111381a59a26
+size 29964

model_creation/runs/detect/train5/args.yaml

@@ -0,0 +1,106 @@
+task: detect
+mode: train
+model: yolov8s.pt
+data: /content/data/data.yaml
+epochs: 5
+time: null
+patience: 100
+batch: 16
+imgsz: 640
+save: true
+save_period: -1
+cache: false
+device: null
+workers: 8
+project: null
+name: train5
+exist_ok: false
+pretrained: true
+optimizer: auto
+verbose: true
+seed: 0
+deterministic: true
+single_cls: false
+rect: false
+cos_lr: false
+close_mosaic: 10
+resume: false
+amp: true
+fraction: 1.0
+profile: false
+freeze: null
+multi_scale: false
+overlap_mask: true
+mask_ratio: 4
+dropout: 0.0
+val: true
+split: val
+save_json: false
+save_hybrid: false
+conf: null
+iou: 0.7
+max_det: 300
+half: false
+dnn: false
+plots: true
+source: null
+vid_stride: 1
+stream_buffer: false
+visualize: false
+augment: false
+agnostic_nms: false
+classes: null
+retina_masks: false
+embed: null
+show: false
+save_frames: false
+save_txt: false
+save_conf: false
+save_crop: false
+show_labels: true
+show_conf: true
+show_boxes: true
+line_width: null
+format: torchscript
+keras: false
+optimize: false
+int8: false
+dynamic: false
+simplify: false
+opset: null
+workspace: 4
+nms: false
+lr0: 0.01
+lrf: 0.01
+momentum: 0.937
+weight_decay: 0.0005
+warmup_epochs: 3.0
+warmup_momentum: 0.8
+warmup_bias_lr: 0.1
+box: 7.5
+cls: 0.5
+dfl: 1.5
+pose: 12.0
+kobj: 1.0
+label_smoothing: 0.0
+nbs: 64
+hsv_h: 0.015
+hsv_s: 0.7
+hsv_v: 0.4
+degrees: 0.0
+translate: 0.1
+scale: 0.5
+shear: 0.0
+perspective: 0.0
+flipud: 0.0
+fliplr: 0.5
+bgr: 0.0
+mosaic: 1.0
+mixup: 0.0
+copy_paste: 0.0
+auto_augment: randaugment
+erasing: 0.4
+crop_fraction: 1.0
+cfg: null
+tracker: botsort.yaml
+save_dir: runs/detect/train5

model_creation/runs/detect/train5/events.out.tfevents.1716832099.517a5a4e3ae1.2051.0

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71ffd4980af6bc63e2b5261b3bef21512ea589d084d037f2af7f528c267129cb
+size 189152

model_creation/runs/detect/train5/results.csv

@@ -0,0 +1,6 @@
+                  epoch,         train/box_loss,         train/cls_loss,         train/dfl_loss,   metrics/precision(B),      metrics/recall(B),       metrics/mAP50(B),    metrics/mAP50-95(B),           val/box_loss,           val/cls_loss,           val/dfl_loss,                 lr/pg0,                 lr/pg1,                 lr/pg2
+                      1,                0.89006,                0.67441,                 1.0306,                0.82238,                 0.8392,                0.85022,                0.60429,                0.70056,                0.38437,                0.96738,             0.00055456,             0.00055456,             0.00055456
+                      2,                0.84925,                0.51813,                 1.0105,                0.82923,                0.89142,                0.86099,                0.61496,                0.70351,                0.35903,                0.97034,              0.0008904,              0.0008904,              0.0008904
+                      3,                 0.8267,                0.48975,                 1.0046,                0.74549,                0.82015,                0.79615,                0.60803,                 0.6697,                0.34986,                0.95315,              0.0010062,              0.0010062,              0.0010062
+                      4,                 0.7923,                0.44796,                0.98622,                 0.8072,                0.93015,                0.88952,                 0.6569,                0.64922,                0.31112,                 0.9423,              0.0006768,              0.0006768,              0.0006768
+                      5,                0.75979,                0.41155,                0.97252,                0.87983,                 0.9133,                0.91496,                0.68457,                0.65444,                0.29384,                0.93771,             0.00034674,             0.00034674,             0.00034674

model_creation/runs/detect/train5/weights/best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e96681ca0f2777bbf9b81f9ec2c8c9098dc775ee525091491d6ec37e39f0140d
+size 22500057

model_creation/runs/detect/train5/weights/last.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:229ada749eb509008639fedd342fee7c851ce0d068af481a0e230863a665e373
+size 22500057

model_creation/yolov8s.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:268e5bb54c640c96c3510224833bc2eeacab4135c6deb41502156e39986b562d
+size 22573363

requirements.txt

@@ -0,0 +1,9 @@
+ipykernel==6.29.4
+pillow==10.3.0
+ultralytics==8.2.23
+manga-ocr==0.1.11
+googletrans==4.0.0-rc1
+deep-translator==1.11.4
+fastapi==0.111.0
+uvicorn==0.30.0
+

server.py

@@ -0,0 +1,83 @@
+"""
+This file contains the FastAPI application that serves the web interface and handles the API requests.
+"""
+
+import os
+import io
+import base64
+
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.staticfiles import StaticFiles
+from fastapi.templating import Jinja2Templates
+from starlette.requests import Request
+from PIL import Image
+import uvicorn
+from ultralytics import YOLO
+
+from utils.predict_bounding_boxes import predict_bounding_boxes
+from utils.manga_ocr import get_text_from_images
+from utils.translate_manga import translate_manga
+from utils.write_text_on_image import write_text
+
+# Load the object detection model
+best_model_path = "./model_creation/runs/detect/train5"
+object_detection_model = YOLO(os.path.join(best_model_path, "weights/best.pt"))
+
+app = FastAPI()
+
+# Add CORS middleware
+app.add_middleware(
+	CORSMiddleware,
+	allow_origins=["*"],
+	allow_methods=["*"],
+	allow_headers=["*"]
+)
+
+# Serve static files and templates
+app.mount("/static", StaticFiles(directory="static"), name="static")
+app.mount("/fonts", StaticFiles(directory="fonts"), name="fonts")
+templates = Jinja2Templates(directory="templates")
+
+@app.get("/")
+def home(request: Request):
+	return templates.TemplateResponse("index.html", {"request": request})
+
+
+@app.post("/predict")
+def predict(request: dict):
+	image = request["image"]
+
+	# Decode base64-encoded image
+	image = base64.b64decode(image)
+	image = Image.open(io.BytesIO(image))
+
+	# Save the image
+	image.save("image.jpg")
+
+	# Perform object detection
+	result = predict_bounding_boxes(object_detection_model, "image.jpg")
+
+	# Extract text from images
+	text_list = get_text_from_images("./bounding_box_images")
+
+	# Translate the manga
+	translated_text_list = translate_manga(text_list)
+
+	# Write translation text on image
+	translated_image = write_text(result, translated_text_list, "image.jpg")
+
+	# Convert the image to base64
+	buff = io.BytesIO()
+	translated_image.save(buff, format="JPEG")
+	img_str = base64.b64encode(buff.getvalue()).decode("utf-8")
+
+	# Clean up
+	os.remove("image.jpg")
+	os.remove("translated_image.png")
+
+	return {"image": img_str}
+
+
+if __name__ == '__main__':
+	uvicorn.run('app:app', host='localhost', port=8000, reload=True)

static/index.js

@@ -0,0 +1,57 @@
+"use strict";
+
+async function predict() {
+	const fileInput = document.getElementById('fileInput');
+	const translateButton = document.getElementById('translateButton');
+	const spinner = document.getElementById('spinner');
+	const inputImage = document.getElementById('inputImage');
+	const outputImage = document.getElementById('outputImage');
+	const downloadButton = document.getElementById('downloadButton');
+
+	downloadButton.style.display = 'none';
+
+	if (fileInput.files.length === 0) {
+		alert('Please select an image file.');
+		return;
+	}
+
+	const file = fileInput.files[0];
+	const reader = new FileReader();
+
+	reader.onload = function () {
+		const base64Image = reader.result.split(',')[1];
+		inputImage.src = `data:image/jpeg;base64,${base64Image}`;
+		inputImage.style.display = 'block';
+	};
+
+	reader.readAsDataURL(file);
+
+	translateButton.style.display = 'none';
+	spinner.style.display = 'block';
+
+	reader.onloadend = async function () {
+		const base64Image = reader.result.split(',')[1];
+
+		const response = await fetch('/predict', {
+			method: 'POST',
+			headers: {
+				'Content-Type': 'application/json'
+			},
+			body: JSON.stringify({ image: base64Image })
+		});
+
+		const result = await response.json();
+		if (response.status !== 200) {
+			alert(result.error);
+			return;
+		}
+		outputImage.src = `data:image/jpeg;base64,${result.image}`;
+		outputImage.style.display = 'block';
+		downloadButton.href = outputImage.src;
+		downloadButton.style.display = 'block';
+
+		translateButton.style.display = 'inline-block';
+		spinner.style.display = 'none';
+
+	};
+}

static/styles.css

@@ -0,0 +1,105 @@
+@font-face {
+    font-family: "MangaFont";
+    src: url("../fonts/mangat.ttf") format("truetype");
+}
+
+body {
+    font-family: "MangaFont", Arial, sans-serif;
+    text-align: center;
+    background-color: #f0f0f0;
+    margin: 0;
+    padding: 0;
+}
+
+header {
+    background-color: #4caf50;
+    color: white;
+    padding: 10px 0;
+}
+
+a {
+    color: white;
+    text-decoration: none;
+}
+
+.container {
+    padding: 20px;
+}
+
+input[type="file"] {
+    margin: 20px 0;
+    padding: 10px;
+    border: 2px solid #4caf50;
+    border-radius: 5px;
+    background-color: #fff;
+    cursor: pointer;
+    transition: border-color 0.3s;
+}
+
+input[type="file"]:hover {
+    border-color: #45a049;
+}
+
+button {
+    padding: 10px 20px;
+    background-color: #4caf50;
+    color: white;
+    border: none;
+    border-radius: 5px;
+    cursor: pointer;
+    font-size: 16px;
+    transition: background-color 0.3s, transform 0.3s;
+}
+
+button:hover {
+    background-color: #45a049;
+    transform: scale(1.05);
+}
+
+.spinner {
+    border: 16px solid #f3f3f3;
+    border-top: 16px solid #4caf50;
+    border-radius: 50%;
+    width: 50px;
+    height: 50px;
+    animation: spin 2s linear infinite;
+    margin: 20px auto;
+}
+
+@keyframes spin {
+    0% {
+        transform: rotate(0deg);
+    }
+    100% {
+        transform: rotate(360deg);
+    }
+}
+
+.images-container {
+    display: flex;
+    justify-content: space-around;
+    margin-top: 20px;
+}
+
+.image-wrapper {
+    width: 45%;
+}
+
+.image-wrapper h3 {
+    margin-bottom: 10px;
+}
+
+#fileInput::file-selector-button {
+    padding: 10px 20px;
+    background-color: #4caf50;
+    color: white;
+    border: none;
+    border-radius: 5px;
+    cursor: pointer;
+    transition: background-color 0.3s, transform 0.3s;
+}
+
+#fileInput::file-selector-button:hover {
+    background-color: #45a049;
+    transform: scale(1.05);
+}

templates/index.html

@@ -0,0 +1,48 @@
+<!DOCTYPE html>
+<html lang="en">
+	<head>
+		<meta charset="UTF-8" />
+		<meta name="viewport" content="width=device-width, initial-scale=1.0" />
+		<title>Manga Translator</title>
+		<link rel="stylesheet" href="/static/styles.css" />
+	</head>
+	<body>
+		<header>
+			<h1>Manga Translator</h1>
+		</header>
+		<div class="container">
+			<input type="file" id="fileInput" accept="image/*" />
+			<br />
+			<button id="translateButton" onclick="predict()">Translate</button>
+			<div id="spinner" class="spinner" style="display: none"></div>
+			<div class="images-container">
+				<div class="image-wrapper">
+					<h3>Original Image</h3>
+					<img
+						id="inputImage"
+						alt="Selected Manga"
+						style="max-width: 100%"
+					/>
+				</div>
+				<div class="image-wrapper">
+					<h3>Translated Image</h3>
+					<img
+						id="outputImage"
+						alt="Translated Manga"
+						style="max-width: 100%; display: none"
+					/>
+					<button>
+						<a
+							id="downloadButton"
+							href="#"
+							download="translated_manga.jpg"
+							style="display: none"
+							>Download Translated Image</a
+						>
+					</button>
+				</div>
+			</div>
+		</div>
+		<script src="/static/index.js"></script>
+	</body>
+</html>