Merge pull request #1 from AlexanderHMagno/main

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,259 @@
+---
+title: Object Segmentation
+emoji: 👁
+colorFrom: gray
+colorTo: pink
+sdk: gradio
+sdk_version: 5.22.0
+app_file: src/app.py
+pinned: false
+---
+
+# 3D Person Segmentation and Anaglyph Generation
+
+[![Open in Spaces](https://img.shields.io/badge/🤗-Open%20in%20Spaces-blue.svg)](https://huggingface.co/spaces/axelhortua/Object-segmentation)
+
+## Documentation Examples
+
+### Input and Segmentation
+![Input Image and Segmentation](public/images/documentation/input_segmentation.png)
+*Example of original input image and its segmentation mask*
+
+### Stereo Processing
+![Stereo Processing](public/images/documentation/stereo_process.png)
+*Demonstration of stereo pair generation with different interaxial distances*
+
+### Anaglyph Output
+![Anaglyph Result](public/images/documentation/anaglyph_output.png)
+*Final anaglyph output with red-cyan 3D effect*
+
+### Interface Overview
+![Gradio Interface](public/images/documentation/interface.png)
+*The Gradio web interface with all adjustment controls*
+
+## Lab Report
+
+### Introduction
+This project implements a sophisticated 3D image processing system that combines person segmentation with stereoscopic and anaglyph image generation. The main objectives were to:
+1. Accurately segment people from images using advanced AI models
+2. Generate stereoscopic 3D effects from 2D images
+3. Create red-cyan anaglyph images for 3D viewing
+4. Provide an interactive web interface for real-time processing
+5. Handle varying image sizes with intelligent mask alignment
+
+### Methodology
+
+#### Tools and Technologies Used
+- **SegFormer (nvidia/segformer-b0)**: State-of-the-art transformer-based model for semantic segmentation
+- **PyTorch**: Deep learning framework for running the SegFormer model
+- **OpenCV**: Image processing operations and mask refinement
+- **Gradio**: Web interface development
+- **NumPy**: Efficient array operations for image manipulation
+- **PIL (Python Imaging Library)**: Image loading and basic transformations
+
+#### Mask Processing Deep Dive
+
+The mask processing is a crucial component of our system, designed to handle various challenges in creating high-quality 3D effects:
+
+1. **Why Mask Resizing is Necessary**
+   - **Input Variability**: User-uploaded images come in different sizes and aspect ratios
+   - **Model Constraints**: SegFormer outputs masks at a fixed resolution (512x512)
+   - **Background Compatibility**: Backgrounds may have different dimensions than person images
+   - **3D Effect Quality**: Proper alignment is crucial for convincing stereoscopic effects
+
+2. **Mask Processing Pipeline**
+   ```
+   Original Image → SegFormer Segmentation → Initial Mask (512x512)
+         ↓
+   Resize to Match Background
+         ↓
+   Add Transparent Padding
+         ↓
+   Center Alignment
+         ↓
+   Final Processed Mask
+   ```
+
+3. **Technical Implementation**
+   ```python
+   # Pseudocode for mask processing
+   def process_mask(mask, background_size):
+       # Calculate padding dimensions
+       pad_top = (background_height - mask_height) // 2
+       pad_bottom = background_height - mask_height - pad_top
+       pad_left = (background_width - mask_width) // 2
+       pad_right = background_width - mask_width - pad_left
+       
+       # Add padding with transparency
+       padded_mask = np.pad(mask, 
+                           ((pad_top, pad_bottom), 
+                            (pad_left, pad_right), 
+                            (0,0)), 
+                           mode='constant')
+       
+       return padded_mask
+   ```
+
+#### Visual Process Explanation
+
+```
++----------------+     +----------------+     +----------------+
+|   Original     |     |   Segmented   |     |    Padded     |
+|    Image       | --> |     Mask      | --> |     Mask      |
+|  (Variable)    |     |   (512x512)   |     | (Background)  |
++----------------+     +----------------+     +----------------+
+         |                                           |
+         v                                           v
++----------------+     +----------------+     +----------------+
+|   Left View    |     |  Stereo Pair  |     |   Anaglyph    |
+|    Shifted     | --> |  Combined     | --> |    Output     |
+|                |     |               |     |               |
++----------------+     +----------------+     +----------------+
+```
+
+**Key Processing Steps Visualization:**
+
+1. **Mask Generation and Sizing:**
+   ```
+   +------------+    +-----------+    +-------------+
+   | Raw Image  |    | Raw Mask  |    | Sized Mask  |
+   |  ******   | -> | ########  | -> | ########    |
+   | *Image *  |    | #Mask  #  |    | #Mask    #  |
+   |  ******   |    | ########  |    | ########    |
+   +------------+    +-----------+    +-------------+
+   ```
+
+2. **Transparency Handling:**
+   ```
+   Original       Padded        Final
+   +----+        +------+      +------+
+   |####|        |      |      |  ##  |
+   |####|   ->   |####  |  ->  |######|
+   |####|        |####  |      |  ##  |
+   +----+        +------+      +------+
+   ```
+
+#### Implementation Steps
+
+1. **Person Segmentation**
+   - Utilized SegFormer model fine-tuned on ADE20K dataset
+   - Applied post-processing with erosion and Gaussian blur for mask refinement
+   - Implemented mask scaling and centering for various input sizes
+   - Added transparent padding for proper background integration
+
+2. **Mask Processing and Alignment**
+   - Implemented dynamic mask resizing to match background dimensions
+   - Added centered padding for smaller masks
+   - Preserved transparency in padded regions
+   - Ensured proper aspect ratio maintenance
+
+3. **Stereoscopic Processing**
+   - Created depth simulation through horizontal pixel shifting
+   - Implemented parallel view stereo pair generation
+   - Added configurable interaxial distance for 3D effect adjustment
+   - Enhanced alignment between stereo pairs with mask centering
+
+4. **Anaglyph Generation**
+   - Combined left and right eye views into red-cyan anaglyph
+   - Implemented color channel separation and recombination
+   - Added background image support with proper masking
+   - Improved blending between foreground and background
+
+5. **User Interface**
+   - Developed interactive web interface using Gradio
+   - Added real-time parameter adjustment capabilities
+   - Implemented support for custom background images
+   - Added size adjustment controls
+
+### Results
+
+The system produces three main outputs:
+1. Segmentation mask showing the isolated person with proper transparency
+2. Side-by-side stereo pair for parallel viewing with centered alignment
+3. Red-cyan anaglyph image for 3D glasses viewing
+
+Key Features:
+- Adjustable person size (10-200%)
+- Configurable interaxial distance (0-10 pixels)
+- Optional custom background support
+- Real-time processing and preview
+- Intelligent mask alignment and padding
+- Transparent background handling
+
+### Discussion
+
+#### Technical Challenges
+1. **Mask Alignment**: Ensuring proper alignment between segmentation masks and background images required careful consideration of image dimensions and aspect ratios.
+2. **Stereo Effect Quality**: Balancing the interaxial distance for comfortable viewing while maintaining the 3D effect.
+3. **Performance Optimization**: Efficient processing of large images while maintaining real-time interaction.
+4. **Transparency Handling**: Implementing proper transparency in padded regions while maintaining mask quality.
+5. **Size Adaptation**: Managing different input image sizes while preserving aspect ratios and alignment.
+
+#### Learning Outcomes
+- Deep understanding of stereoscopic image generation
+- Experience with state-of-the-art segmentation models
+- Practical knowledge of image processing techniques
+- Web interface development for ML applications
+- Advanced mask manipulation and alignment strategies
+
+### Conclusion
+
+This project successfully demonstrates the integration of modern AI-powered segmentation with classical stereoscopic image processing techniques. The system provides an accessible way to create 3D effects from regular 2D images, with robust handling of different image sizes and proper transparency management.
+
+#### Future Work
+- Implementation of depth-aware 3D effect generation
+- Support for video processing
+- Additional 3D viewing formats (side-by-side, over-under)
+- Enhanced background replacement options
+- Mobile device optimization
+- Advanced depth map generation
+- Multi-person segmentation support
+
+## Setup
+
+```bash
+pip install -r requirements.txt
+```
+
+## Usage
+
+```bash
+cd src
+python app.py
+```
+
+## Parameters
+
+- **Person Image**: Upload an image containing a person
+- **Background Image**: (Optional) Custom background image
+- **Interaxial Distance**: Adjust the 3D effect strength (0-10)
+- **Person Size**: Adjust the size of the person in the output (10-200%)
+
+## Output Types
+
+1. **Segmentation Mask**: Shows the isolated person with proper transparency
+2. **Stereo Pair**: Side-by-side stereo image for parallel viewing
+3. **Anaglyph**: Red-cyan 3D image viewable with anaglyph glasses
+
+## Technical Notes
+
+- **Mask Processing Details**:
+  - Initial mask is generated at 512x512 resolution
+  - Dynamic padding calculation: `pad = (background_size - mask_size) // 2`
+  - Transparency preservation using NumPy's constant padding mode
+  - Aspect ratio maintained through centered scaling
+  - Real-time size adjustments (10-200%) applied before padding
+
+- **Size Handling Algorithm**:
+  1. Calculate target dimensions based on background
+  2. Resize mask while maintaining aspect ratio
+  3. Add transparent padding to match background
+  4. Center the mask content
+  5. Apply any user-specified size adjustments
+
+- The system automatically handles different input image sizes
+- Masks are dynamically padded and centered for optimal alignment
+- Transparent regions are properly preserved in the final output
+- Background images are automatically scaled to match the person image
+- Real-time preview updates as parameters are adjusted
+

public/lab_report.txt

@@ -0,0 +1,131 @@
+3D Person Segmentation and Anaglyph Generation - Lab Report
+=================================================
+
+Introduction
+------------
+In this project, I developed a sophisticated 3D image processing system that combines modern AI-powered person segmentation with classical stereoscopic image processing. The main objectives were successfully accomplished:
+
+1. Implementation of accurate person segmentation using SegFormer AI model
+2. Creation of stereoscopic 3D effects from 2D images
+3. Generation of red-cyan anaglyph images for 3D viewing
+4. Development of an interactive web interface
+5. Implementation of intelligent mask alignment for varying image sizes
+
+The project is accessible at: https://huggingface.co/spaces/axelhortua/Object-segmentation
+
+Methodology
+-----------
+The implementation followed a systematic approach using various tools and technologies:
+
+1. Tools Selection:
+   - SegFormer (nvidia/segformer-b0) for semantic segmentation
+   - PyTorch for deep learning implementation
+   - OpenCV for image processing
+   - Gradio for web interface
+   - NumPy for array operations
+   - PIL for image handling
+
+2. Implementation Process:
+
+   a) Person Segmentation:
+      - Used SegFormer model fine-tuned on ADE20K dataset
+      - Applied post-processing with erosion and Gaussian blur
+      - Implemented dynamic mask scaling and centering
+
+   b) Mask Processing:
+      - Developed dynamic mask resizing system
+      - Implemented transparent padding
+      - Ensured proper aspect ratio maintenance
+      - Created centered alignment algorithm
+
+   c) Stereoscopic Processing:
+      - Implemented horizontal pixel shifting for depth simulation
+      - Created parallel view stereo pair generation
+      - Added configurable interaxial distance
+      - Enhanced stereo pair alignment
+
+   d) Anaglyph Generation:
+      - Implemented color channel separation
+      - Created background integration system
+      - Developed foreground-background blending
+      - Optimized 3D effect quality
+
+Results
+-------
+The system successfully produces three main outputs:
+
+1. Segmentation Mask:
+   - Clean person isolation
+   - Proper transparency handling
+   - Accurate edge detection
+   - Smooth mask transitions
+
+2. Stereo Pair:
+   - Side-by-side stereo image
+   - Configurable depth effect
+   - Proper alignment between pairs
+   - Maintained image quality
+
+3. Anaglyph Output:
+   - Red-cyan 3D image
+   - Adjustable 3D effect strength
+   - Clean color separation
+   - Minimal ghosting artifacts
+
+Key Features Achieved:
+- Person size adjustment (10-200%)
+- Interaxial distance control (0-10 pixels)
+- Custom background support
+- Real-time processing and preview
+- Intelligent mask alignment
+- Transparent background handling
+
+Discussion
+----------
+Technical Challenges Faced:
+
+1. Mask Alignment:
+   - Complex handling of different image dimensions
+   - Maintaining proper aspect ratios
+   - Ensuring consistent centering
+   - Handling edge cases
+
+2. Stereo Effect Quality:
+   - Balancing interaxial distance
+   - Minimizing visual artifacts
+   - Maintaining comfortable viewing experience
+   - Preserving image details
+
+3. Performance Optimization:
+   - Efficient large image processing
+   - Real-time interface responsiveness
+   - Memory management
+   - Processing speed optimization
+
+4. Transparency Handling:
+   - Proper alpha channel management
+   - Clean edge preservation
+   - Consistent transparency across operations
+   - Background integration
+
+Learning Outcomes:
+- Deep understanding of stereoscopic image generation
+- Practical experience with AI models
+- Advanced image processing techniques
+- Web interface development skills
+- Complex system integration experience
+
+Conclusion
+----------
+The project successfully demonstrates the integration of AI-powered segmentation with classical stereoscopic techniques. The system provides an accessible way to create 3D effects from regular 2D images, with robust handling of different image sizes and proper transparency management.
+
+Future Work:
+1. Implementation of depth-aware 3D effect generation
+2. Addition of video processing capabilities
+3. Support for additional 3D viewing formats
+4. Enhanced background replacement options
+5. Mobile device optimization
+6. Advanced depth map generation
+7. Multi-person segmentation support
+
+The project has laid a strong foundation for future developments in 3D image processing and demonstrates the potential of combining AI with traditional image processing techniques. 

requirements.txt

@@ -5,4 +5,4 @@ datasets
 opencv-python
 gradio
 numpy
-scikit-image
+imagehash

src/app.py

@@ -1,8 +1,8 @@
 import gradio as gr
 import numpy as np
 from PIL import Image
-from utils import load_model, segment_person, resize_image, split_stereo_image
-
+from utils import load_model, segment_person, resize_image, split_stereo_image,resize_image_to_width, resize_mask, resize_images
+from testing import get_image_names
 # Load model and processor once
 processor, model = load_model()
 
@@ -10,32 +10,55 @@ processor, model = load_model()
 default_bg = Image.new("RGB", (512, 512), color=(95, 147, 89))
 
 
-
-
-
 def generate_3d_outputs(person_img, background_img=None, shift_pixels=10,  person_size=100):
     # Resize images to match
-    image = resize_image(person_img, person_size)
-    background_img = background_img if background_img is not None else default_bg
 
+    background_img = background_img if background_img is not None else default_bg
 
     # Split background image into left and right halves
     leftBackground, rightBackground = split_stereo_image(Image.fromarray(background_img))
 
-    # Resize image to match background dimensions
+    ## Match person image to background image width
+    image = resize_image_to_width(person_img, leftBackground)
 
-    
-    image = Image.fromarray(np.array(image)).resize((leftBackground.shape[1], leftBackground.shape[0]))
     # Step 1: Segment person
     mask = segment_person(image, processor, model)
 
-    image_np = np.array(image)
+    # Resize mask based on person_size percentage
+    mask = resize_mask(person_size, mask)
+
+    # Resize image based on person_size percentage
+    image_np = resize_images(image, person_size)
+
+    # Apply mask to image
+    person_only = image_np * mask
+    person_segmentation = np.clip(person_only, 0, 255).astype(np.uint8)
 
+    # Resize mask and person_only to match background dimensions while preserving content
+    target_height, target_width = leftBackground.shape[:2]
+    current_height, current_width = mask.shape[:2]
+
+    # Calculate padding
+    pad_top = max(0, (target_height - current_height) // 2)
+    pad_bottom = max(0, target_height - current_height - pad_top)
+    pad_left = max(0, (target_width - current_width) // 2)
+    pad_right = max(0, target_width - current_width - pad_left)
+
+    # Pad mask and person_only arrays
+    mask = np.pad(mask, ((pad_top, pad_bottom), (pad_left, pad_right), (0,0)), mode='constant')    
+    person_only = np.pad(person_segmentation, ((pad_top, pad_bottom), (pad_left, pad_right), (0,0)), mode='constant')
+
+
+    # CROP MASK TO MATCH BACKGROUND DIMENSIONS FROM CENTER OF BACKGROUND
+    if(mask.shape[0] > target_height or mask.shape[1] > target_width):
+        mask = mask[mask.shape[0]//2-target_height//2:mask.shape[0]//2+target_height//2, mask.shape[1]//2-target_width//2:mask.shape[1]//2+target_width//2, :]
+        person_only = person_only[person_only.shape[0]//2-target_height//2:person_only.shape[0]//2+target_height//2, person_only.shape[1]//2-target_width//2:person_only.shape[1]//2+target_width//2, :]
+
+    # Convert background images to numpy arrays
     leftBackground_np = np.array(leftBackground)
     rightBackground_np = np.array(rightBackground)
 
-
-    person_only = image_np * mask
+    # Apply mask to background images
     leftBackground_only = leftBackground_np * (1 - mask)
     rightBackground_only = rightBackground_np * (1 - mask)
 
@@ -43,10 +66,9 @@ def generate_3d_outputs(person_img, background_img=None, shift_pixels=10,  perso
     person_left = np.roll(person_only, shift=-shift_pixels, axis=1)
     person_right = np.roll(person_only, shift=shift_pixels, axis=1)
     
-
+    
     left_eye = np.clip(person_right + leftBackground_only, 0, 255).astype(np.uint8)
     right_eye = np.clip(person_left + rightBackground_only, 0, 255).astype(np.uint8)
-    person_segmentation = np.clip(person_only, 0, 255).astype(np.uint8)
 
     # --- Combine left and right images side by side ---
     stereo_pair = np.concatenate([left_eye, right_eye], axis=1)
@@ -71,7 +93,7 @@ demo = gr.Interface(
     inputs=[
         gr.Image(label="Person Image"),
         gr.Image(label="Optional Background Image"),
-        gr.Slider(minimum=0, maximum=10, step=1, value=10, label="interaxial distance"),
+        gr.Slider(minimum=0, maximum=20, step=1, value=10, label="interaxial distance"),
         gr.Slider(minimum=10, maximum=200, step=10, value=100, label="Person Size %"),
   
     ],
@@ -80,6 +102,7 @@ demo = gr.Interface(
         gr.Image(label="Stereo_pair"),
         gr.Image(label="3D Anaglyph Image")
     ],
+    examples= get_image_names(),
     title="3D Person Segmentation Viewer",
     description="Upload a person photo and optionally a background image. Outputs anaglyph and stereo views."
 )

src/testing.py

@@ -1,4 +1,38 @@
-from app import create_anaglyph
-
 # Provide paths to your test images
-create_anaglyph("person.png", "bg.png", "test_anaglyph.png")
+import os
+import random
+
+def get_image_names():
+    # Get people images
+    people_dir = "public/images/people" 
+    # Get background images
+    background_dir = "public/images/backgrounds"
+
+
+    data = [
+      ['e.jpg', 'e.png', 10, 60],
+      ['f.jpg', 'f.png', 10, 60],
+      ['g.jpeg', 'g.png', 10, 40],      
+      ['a.jpg', 'a.png', 10, 130],
+      ['b.jpg', 'b.png', 10, 40],
+      ['c.jpg', 'c.png', 10, 60],
+      ['d.png', 'd.png', 10, 50]
+    ]
+
+    generate_testing_list = []
+
+    for i in range(len(data)):
+      generate_testing_list.append(
+        list((
+          os.path.join(people_dir, data[i][0]), 
+          os.path.join(background_dir, data[i][1]), 
+          data[i][2], 
+          data[i][3]
+          ))
+        )
+    
+    return generate_testing_list
+
+
+
+

src/utils.py

@@ -94,3 +94,61 @@ def split_stereo_image(image):
     else:
         return image, resize_image(image, 99)
     
+def resize_image_to_width(person_img, background_img):
+       # Resize image to match background dimensions
+    if (background_img.shape[1] > background_img.shape[0]):
+        width = background_img.shape[1]
+        img_array = np.array(person_img)
+        height = int(width * img_array.shape[0] / img_array.shape[1])
+        person_img = Image.fromarray(img_array).resize((width, height))
+        person_img = np.array(person_img)
+        image = Image.fromarray(person_img)
+    else:
+        height = background_img.shape[0]
+        img_array = np.array(person_img)
+        width = int(height * img_array.shape[1] / img_array.shape[0])
+        person_img = Image.fromarray(img_array).resize((width, height))
+        person_img = np.array(person_img)
+        image = Image.fromarray(person_img)
+
+
+    return image
+
+def resize_mask(person_size, mask):
+        
+    scale_factor = person_size / 100.0
+    mask_height, mask_width = mask.shape[:2]
+    new_height = int(mask_height * scale_factor)
+    new_width = int(mask_width * scale_factor)
+    
+    # Convert mask to PIL Image for resizing
+    mask_image = Image.fromarray((mask * 255).astype(np.uint8))
+    resized_mask = mask_image.resize((new_width, new_height))
+    
+    # Convert back to numpy and normalize to 0-1
+    mask = np.array(resized_mask).astype(np.float32) / 255.0
+    
+    # Add third channel dimension back if needed
+    if len(mask.shape) == 2:
+        mask = np.stack([mask] * 3, axis=-1)
+
+    return mask
+
+def resize_images(image, person_size):
+    image_np = np.array(image)
+    # Resize image based on person_size percentage
+
+    scale_factor = person_size / 100.0
+    img_height, img_width = image_np.shape[:2]
+    new_height = int(img_height * scale_factor)
+    new_width = int(img_width * scale_factor)
+    
+    # Convert image to PIL Image for resizing
+    image_pil = Image.fromarray(image_np)
+    resized_image = image_pil.resize((new_width, new_height))
+    
+    # Convert back to numpy
+    image = resized_image
+    image_np = np.array(image)
+
+    return image_np