README.md

---
title: README
emoji: 🍫
colorFrom: yellow
colorTo: green
sdk: static
pinned: false
---

<p align="center">
    <img src="https://raw.githubusercontent.com/ml6team/fondant/main/docs/art/fondant_banner.svg" height="250px"/>
</p>
<p align="center">
    <i>Large-scale data processing made easy and reusable</i>
    <br>
    <a href="https://fondant.readthedocs.io/en/stable/"><strong>Explore the docs »</strong></a>
    <br>
    <br>
    <a href="https://discord.gg/HnTdWhydGp"><img alt="Discord" src="https://dcbadge.vercel.app/api/server/HnTdWhydGp?style=flat-square"></a>
    <a href="https://pypi.org/project/fondant/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/fondant?color=brightgreen&style=flat-square"></a>
    <a href="https://fondant.readthedocs.io/en/latest/license/"><img alt="License" src="https://img.shields.io/github/license/ml6team/fondant?style=flat-square&color=brightgreen"></a>
    <a href="https://github.com/ml6team/fondant/actions/workflows/pipeline.yaml"><img alt="GitHub Workflow Status" src="https://img.shields.io/github/actions/workflow/status/ml6team/fondant/pipeline.yaml?style=flat-square"></a>
    <a href="https://coveralls.io/github/ml6team/fondant?branch=main"><img alt="Coveralls" src="https://img.shields.io/coverallsCoverage/github/ml6team/fondant?style=flat-square"></a>
</p>

![image/svg](https://raw.githubusercontent.com/ml6team/fondant/main/docs/art/fondant_banner.svg)
<p align="center">
    <i>Large-scale data processing made easy and reusable</i>
    <br>
    <a href="https://fondant.readthedocs.io/en/stable/"><strong>Explore the docs »</strong></a>
    <br>
    <br>
    <a href="https://discord.gg/HnTdWhydGp"><img alt="Discord" src="https://dcbadge.vercel.app/api/server/HnTdWhydGp?style=flat-square"></a>
    <a href="https://pypi.org/project/fondant/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/fondant?color=brightgreen&style=flat-square"></a>
    <a href="https://fondant.readthedocs.io/en/latest/license/"><img alt="License" src="https://img.shields.io/github/license/ml6team/fondant?style=flat-square&color=brightgreen"></a>
    <a href="https://github.com/ml6team/fondant/actions/workflows/pipeline.yaml"><img alt="GitHub Workflow Status" src="https://img.shields.io/github/actions/workflow/status/ml6team/fondant/pipeline.yaml?style=flat-square"></a>
    <a href="https://coveralls.io/github/ml6team/fondant?branch=main"><img alt="Coveralls" src="https://img.shields.io/coverallsCoverage/github/ml6team/fondant?style=flat-square"></a>
</p>
---
🍫**Fondant is an open-source framework that aims to simplify and speed up large-scale data processing by making 
containerized components reusable across pipelines and execution environments and shareable within the community.**\
It offers:
- 🔧 Plug ‘n’ play composable pipelines for creating datasets for
    - AI image generation model fine-tuning (Stable Diffusion, ControlNet)
    - Large language model fine-tuning (LLaMA, Falcon)
    - Code generation model fine-tuning (StarCoder)
- 🧱 Library of off-the-shelf reusable components for
    - Extracting data from public sources such as Common Crawl, LAION, ...
    - Filtering on 
        - Content, e.g. language, visual style, topic, format, aesthetics, etc.
        - Context, e.g. copyright license, origin
        - Metadata
    - Removal of unwanted data such as toxic, NSFW or generated content
    - Removal of unwanted data patterns such as societal bias
    - Transforming data (resizing, cropping, reformatting, …)
    - Tuning the data for model performance (normalization, deduplication, …)
    - Enriching data (captioning, metadata generation, synthetics, …)
    - Transparency, auditability, compliance
- 📖 🖼️ 🎞️ ♾️ Out of the box multimodal capabilities: text, images, video, etc.
- 🐍 Standardized, Python/Pandas-based way of creating custom components
- 🏭 Production-ready, scalable deployment
- ☁️ Multi-cloud integrations

## 🪤 Why Fondant?

In the age of Foundation Models, control over your data is key and building pipelines
for large-scale data processing is costly, especially when they require advanced
machine learning-based operations. This need not be the case, however, if processing
components would be reusable and exchangeable and pipelines were easily composable. 
Realizing this is the main vision behind Fondant.

<p align="right">(<a href="#chocolate_bar-fondant">back to top</a>)</p>