Reorganized docs for docusaurus publish (#860)

### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update

README.md

@@ -34,11 +34,11 @@
 - 2024-05-15 Integrates OpenAI GPT-4o.
 - 2024-05-08 Integrates LLM DeepSeek-V2.
 - 2024-04-26 Adds file management.
-- 2024-04-19 Supports conversation API ([detail](./docs/conversation_api.md)).
+- 2024-04-19 Supports conversation API ([detail](./docs/references/api.md)).
 - 2024-04-16 Integrates an embedding model 'bce-embedding-base_v1' from [BCEmbedding](https://github.com/netease-youdao/BCEmbedding), and [FastEmbed](https://github.com/qdrant/fastembed), which is designed specifically for light and speedy embedding.
-- 2024-04-11 Supports [Xinference](./docs/xinference.md) for local LLM deployment.
+- 2024-04-11 Supports [Xinference](./docs/guides/deploy_local_llm.md) for local LLM deployment.
 - 2024-04-10 Adds a new layout recognition model for analyzing legal documents.
-- 2024-04-08 Supports [Ollama](./docs/ollama.md) for local LLM deployment.
+- 2024-04-08 Supports [Ollama](./docs/guides/deploy_local_llm.md) for local LLM deployment.
 - 2024-04-07 Supports Chinese UI.
 
 ## 🌟 Key Features
@@ -87,7 +87,7 @@
 
 ### 🚀 Start up the server
 
-1. Ensure `vm.max_map_count` >= 262144 ([more](./docs/max_map_count.md)):
+1. Ensure `vm.max_map_count` >= 262144 ([more](./docs/guides/max_map_count.md)):
 
    > To check the value of `vm.max_map_count`:
    >
@@ -154,7 +154,7 @@
    > With default settings, you only need to enter `http://IP_OF_YOUR_MACHINE` (**sans** port number) as the default HTTP serving port `80` can be omitted when using the default configurations.
 6. In [service_conf.yaml](./docker/service_conf.yaml), select the desired LLM factory in `user_default_llm` and update the `API_KEY` field with the corresponding API key.
 
-   > See [./docs/llm_api_key_setup.md](./docs/llm_api_key_setup.md) for more information.
+   > See [./docs/guides/llm_api_key_setup.md](./docs/guides/llm_api_key_setup.md) for more information.
 
    _The show is now on!_
 
@@ -288,7 +288,7 @@ To launch the service from source:
 ## 📚 Documentation
 
 - [Quickstart](./docs/quickstart.md)
-- [FAQ](./docs/faq.md)
+- [FAQ](./docs/references/faq.md)
 
 ## 📜 Roadmap
 
@@ -301,4 +301,4 @@ See the [RAGFlow Roadmap 2024](https://github.com/infiniflow/ragflow/issues/162)
 
 ## 🙌 Contributing
 
-RAGFlow flourishes via open-source collaboration. In this spirit, we embrace diverse contributions from the community. If you would like to be a part, review our [Contribution Guidelines](https://github.com/infiniflow/ragflow/blob/main/docs/CONTRIBUTING.md) first.
+RAGFlow flourishes via open-source collaboration. In this spirit, we embrace diverse contributions from the community. If you would like to be a part, review our [Contribution Guidelines](./docs/references/CONTRIBUTING.md) first.

README_ja.md

@@ -34,12 +34,12 @@
 - 2024-05-15 OpenAI GPT-4oを統合しました。
 - 2024-05-08 LLM DeepSeek-V2を統合しました。
 - 2024-04-26 「ファイル管理」機能を追加しました。
-- 2024-04-19 会話 API をサポートします ([詳細](./docs/conversation_api.md))。
+- 2024-04-19 会話 API をサポートします ([詳細](./docs/references/api.md))。
 - 2024-04-16 [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) から埋め込みモデル「bce-embedding-base_v1」を追加します。
 - 2024-04-16 [FastEmbed](https://github.com/qdrant/fastembed) は、軽量かつ高速な埋め込み用に設計されています。
-- 2024-04-11 ローカル LLM デプロイメント用に [Xinference](./docs/xinference.md) をサポートします。
+- 2024-04-11 ローカル LLM デプロイメント用に [Xinference](./docs/guides/deploy_local_llm.md) をサポートします。
 - 2024-04-10 メソッド「Laws」に新しいレイアウト認識モデルを追加します。
-- 2024-04-08 [Ollama](./docs/ollama.md) を使用した大規模モデルのローカライズされたデプロイメントをサポートします。
+- 2024-04-08 [Ollama](./docs/guides/deploy_local_llm.md) を使用した大規模モデルのローカライズされたデプロイメントをサポートします。
 - 2024-04-07 中国語インターフェースをサポートします。
 
 
@@ -89,7 +89,7 @@
 
 ### 🚀 サーバーを起動
 
-1. `vm.max_map_count` >= 262144 であることを確認する【[もっと](./docs/max_map_count.md)】:
+1. `vm.max_map_count` >= 262144 であることを確認する【[もっと](./docs/guides/max_map_count.md)】:
 
    > `vm.max_map_count` の値をチェックするには:
    >
@@ -155,7 +155,7 @@
    > デフォルトの設定を使用する場合、デフォルトの HTTP サービングポート `80` は省略できるので、与えられたシナリオでは、`http://IP_OF_YOUR_MACHINE`（ポート番号は省略）だけを入力すればよい。
 6. [service_conf.yaml](./docker/service_conf.yaml) で、`user_default_llm` で希望の LLM ファクトリを選択し、`API_KEY` フィールドを対応する API キーで更新する。
 
-   > 詳しくは [./docs/llm_api_key_setup.md](./docs/llm_api_key_setup.md) を参照してください。
+   > 詳しくは [./docs/guides/llm_api_key_setup.md](./docs/guides/llm_api_key_setup.md) を参照してください。
 
    _これで初期設定完了！ショーの開幕です！_
 
@@ -255,7 +255,7 @@ $ bash ./entrypoint.sh
 ## 📚 ドキュメンテーション
 
 - [Quickstart](./docs/quickstart.md)
-- [FAQ](./docs/faq.md)
+- [FAQ](./docs/references/faq.md)
 
 ## 📜 ロードマップ
 
@@ -268,4 +268,4 @@ $ bash ./entrypoint.sh
 
 ## 🙌 コントリビュート
 
-RAGFlow はオープンソースのコラボレーションによって発展してきました。この精神に基づき、私たちはコミュニティからの多様なコントリビュートを受け入れています。 参加を希望される方は、まず[コントリビューションガイド](https://github.com/infiniflow/ragflow/blob/main/docs/CONTRIBUTING.md)をご覧ください。
+RAGFlow はオープンソースのコラボレーションによって発展してきました。この精神に基づき、私たちはコミュニティからの多様なコントリビュートを受け入れています。 参加を希望される方は、まず[コントリビューションガイド](./docs/references/CONTRIBUTING.md)をご覧ください。

README_zh.md

@@ -34,11 +34,11 @@
 - 2024-05-15 集成大模型 OpenAI GPT-4o。
 - 2024-05-08 集成大模型 DeepSeek。
 - 2024-04-26 增添了'文件管理'功能。
-- 2024-04-19 支持对话 API ([更多](./docs/conversation_api.md))。
+- 2024-04-19 支持对话 API ([更多](./docs/references/api.md))。
 - 2024-04-16 集成嵌入模型 [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) 和 专为轻型和高速嵌入而设计的 [FastEmbed](https://github.com/qdrant/fastembed)。
-- 2024-04-11 支持用 [Xinference](./docs/xinference.md) 本地化部署大模型。
+- 2024-04-11 支持用 [Xinference](./docs/guides/deploy_local_llm.md) 本地化部署大模型。
 - 2024-04-10 为‘Laws’版面分析增加了底层模型。
-- 2024-04-08 支持用 [Ollama](./docs/ollama.md) 本地化部署大模型。
+- 2024-04-08 支持用 [Ollama](./docs/guides/deploy_local_llm.md) 本地化部署大模型。
 - 2024-04-07 支持中文界面。
 
 ## 🌟 主要功能
@@ -87,7 +87,7 @@
 
 ### 🚀 启动服务器
 
-1. 确保 `vm.max_map_count` 不小于 262144 【[更多](./docs/max_map_count.md)】：
+1. 确保 `vm.max_map_count` 不小于 262144 【[更多](./docs/guides/max_map_count.md)】：
 
    > 如需确认 `vm.max_map_count` 的大小：
    >
@@ -153,7 +153,7 @@
    > 上面这个例子中，您只需输入 http://IP_OF_YOUR_MACHINE 即可：未改动过配置则无需输入端口（默认的 HTTP 服务端口 80）。
 6. 在 [service_conf.yaml](./docker/service_conf.yaml) 文件的 `user_default_llm` 栏配置 LLM factory，并在 `API_KEY` 栏填写和你选择的大模型相对应的 API key。
 
-   > 详见 [./docs/llm_api_key_setup.md](./docs/llm_api_key_setup.md)。
+   > 详见 [./docs/guides/llm_api_key_setup.md](./docs/guides/llm_api_key_setup.md)。
 
    _好戏开始，接着奏乐接着舞！_
 
@@ -274,7 +274,7 @@ $ systemctl start nginx
 ## 📚 技术文档
 
 - [Quickstart](./docs/quickstart.md)
-- [FAQ](./docs/faq.md)
+- [FAQ](./docs/references/faq.md)
 
 ## 📜 路线图
 
@@ -287,7 +287,7 @@ $ systemctl start nginx
 
 ## 🙌 贡献指南
 
-RAGFlow 只有通过开源协作才能蓬勃发展。秉持这一精神,我们欢迎来自社区的各种贡献。如果您有意参与其中,请查阅我们的[贡献者指南](https://github.com/infiniflow/ragflow/blob/main/docs/CONTRIBUTING.md) 。
+RAGFlow 只有通过开源协作才能蓬勃发展。秉持这一精神,我们欢迎来自社区的各种贡献。如果您有意参与其中,请查阅我们的[贡献者指南](./docs/references/CONTRIBUTING.md) 。
 
 ## 👥 加入社区
 

docs/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Get Started",
+  "position": 1,
+  "link": {
+    "type": "generated-index",
+    "description": "RAGFlow Quick Start"
+  }
+}

docs/guides/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "User Guides",
+  "position": 2,
+  "link": {
+    "type": "generated-index",
+    "description": "RAGFlow User Guides"
+  }
+}

docs/{configure_knowledge_base.md → guides/configure_knowledge_base.md}

@@ -1,3 +1,8 @@
+---
+sidebar_position: 1
+slug: /configure_knowledge_base
+---
+
 # Configure a knowledge base
 
 Knowledge base, hallucination-free chat, and file management are three pillars of RAGFlow. RAGFlow's AI chats are based on knowledge bases. Each of RAGFlow's knowledge bases serves as a knowledge source, *parsing* files uploaded from your local machine and file references generated in **File Management** into the real 'knowledge' for future AI chats. This guide demonstrates some basic usages of the knowledge base feature, covering the following topics:
@@ -118,7 +123,7 @@ RAGFlow uses multiple recall of both full-text search and vector search in its c
 
 ## Search for knowledge base
 
-As of RAGFlow v0.5.0, the search feature is still in a rudimentary form, supporting only knowledge base search by name.
+As of RAGFlow v0.6.0, the search feature is still in a rudimentary form, supporting only knowledge base search by name.
 
 ![search knowledge base](https://github.com/infiniflow/ragflow/assets/93570324/836ae94c-2438-42be-879e-c7ad2a59693e)
 

docs/{xinference.md → guides/deploy_local_llm.md}

@@ -1,43 +1,75 @@
-# Xinference
-
-<div align="center" style="margin-top:20px;margin-bottom:20px;">
-<img src="https://github.com/infiniflow/ragflow/assets/12318111/2c5e86a7-807b-4d29-bd2b-f73fb1018866" width="130"/>
-</div>
-
-Xorbits Inference([Xinference](https://github.com/xorbitsai/inference)) empowers you to unleash the full potential of cutting-edge AI models. 
-
-## Install
-
-- [pip install "xinference[all]"](https://inference.readthedocs.io/en/latest/getting_started/installation.html)
-- [Docker](https://inference.readthedocs.io/en/latest/getting_started/using_docker_image.html)
-
-To start a local instance of Xinference, run the following command:
-```bash
-$ xinference-local --host 0.0.0.0 --port 9997
-```
-## Launch Xinference
-
-Decide which LLM you want to deploy ([here's a list for supported LLM](https://inference.readthedocs.io/en/latest/models/builtin/)), say, **mistral**.
-Execute the following command to launch the model, remember to replace ${quantization} with your chosen quantization method from the options listed above:
-```bash
-$ xinference launch -u mistral --model-name mistral-v0.1 --size-in-billions 7 --model-format pytorch --quantization ${quantization}
-```
-
-## Use Xinference in RAGFlow
-
-- Go to 'Settings > Model Providers > Models to be added > Xinference'.
-    
-<div align="center" style="margin-top:20px;margin-bottom:20px;">
-<img src="https://github.com/infiniflow/ragflow/assets/12318111/bcbf4d7a-ade6-44c7-ad5f-0a92c8a73789" width="1300"/>
-</div>
-
-> Base URL: Enter the base URL where the Xinference service is accessible, like, `http://<your-xinference-endpoint-domain>:9997/v1`.
-
-- Use Xinference Models.
-
-<div align="center" style="margin-top:20px;margin-bottom:20px;">
-<img src="https://github.com/infiniflow/ragflow/assets/12318111/b01fcb6f-47c9-4777-82e0-f1e947ed615a" width="530"/>
-</div>
-<div align="center" style="margin-top:20px;margin-bottom:20px;">
-<img src="https://github.com/infiniflow/ragflow/assets/12318111/1763dcd1-044f-438d-badd-9729f5b3a144" width="530"/>
-</div>
+---
+sidebar_position: 5
+slug: /deploy_local_llm
+---
+
+# Deploy a local LLM
+
+RAGFlow supports deploying LLMs locally using Ollama or Xinference.
+
+## Ollama
+
+One-click deployment of local LLMs, that is [Ollama](https://github.com/ollama/ollama).
+
+### Install
+
+- [Ollama on Linux](https://github.com/ollama/ollama/blob/main/docs/linux.md)
+- [Ollama Windows Preview](https://github.com/ollama/ollama/blob/main/docs/windows.md)
+- [Docker](https://hub.docker.com/r/ollama/ollama)
+
+### Launch Ollama
+
+Decide which LLM you want to deploy ([here's a list for supported LLM](https://ollama.com/library)), say, **mistral**:
+```bash
+$ ollama run mistral
+```
+Or,
+```bash
+$ docker exec -it ollama ollama run mistral
+```
+
+### Use Ollama in RAGFlow
+
+- Go to 'Settings > Model Providers > Models to be added > Ollama'.
+    
+![](https://github.com/infiniflow/ragflow/assets/12318111/a9df198a-226d-4f30-b8d7-829f00256d46)
+
+> Base URL: Enter the base URL where the Ollama service is accessible, like, `http://<your-ollama-endpoint-domain>:11434`.
+
+- Use Ollama Models.
+
+![](https://github.com/infiniflow/ragflow/assets/12318111/60ff384e-5013-41ff-a573-9a543d237fd3)
+
+## Xinference
+
+Xorbits Inference([Xinference](https://github.com/xorbitsai/inference)) empowers you to unleash the full potential of cutting-edge AI models. 
+
+### Install
+
+- [pip install "xinference[all]"](https://inference.readthedocs.io/en/latest/getting_started/installation.html)
+- [Docker](https://inference.readthedocs.io/en/latest/getting_started/using_docker_image.html)
+
+To start a local instance of Xinference, run the following command:
+```bash
+$ xinference-local --host 0.0.0.0 --port 9997
+```
+### Launch Xinference
+
+Decide which LLM you want to deploy ([here's a list for supported LLM](https://inference.readthedocs.io/en/latest/models/builtin/)), say, **mistral**.
+Execute the following command to launch the model, remember to replace ${quantization} with your chosen quantization method from the options listed above:
+```bash
+$ xinference launch -u mistral --model-name mistral-v0.1 --size-in-billions 7 --model-format pytorch --quantization ${quantization}
+```
+
+### Use Xinference in RAGFlow
+
+- Go to 'Settings > Model Providers > Models to be added > Xinference'.
+    
+![](https://github.com/infiniflow/ragflow/assets/12318111/bcbf4d7a-ade6-44c7-ad5f-0a92c8a73789)
+
+> Base URL: Enter the base URL where the Xinference service is accessible, like, `http://<your-xinference-endpoint-domain>:9997/v1`.
+
+- Use Xinference Models.
+
+![](https://github.com/infiniflow/ragflow/assets/12318111/b01fcb6f-47c9-4777-82e0-f1e947ed615a)
+![](https://github.com/infiniflow/ragflow/assets/12318111/1763dcd1-044f-438d-badd-9729f5b3a144)

docs/{llm_api_key_setup.md → guides/llm_api_key_setup.md}

@@ -1,19 +1,25 @@
-
-## Set Before Starting The System
-
-In **user_default_llm** of [service_conf.yaml](./docker/service_conf.yaml), you need to specify LLM factory and your own _API_KEY_. 
-RagFlow supports the flowing LLM factory, and with more coming in the pipeline:
-
-> [OpenAI](https://platform.openai.com/login?launch), [Tongyi-Qianwen](https://dashscope.console.aliyun.com/model), 
-> [ZHIPU-AI](https://open.bigmodel.cn/), [Moonshot](https://platform.moonshot.cn/docs)
-
-After sign in these LLM suppliers, create your own API-Key, they all have a certain amount of free quota.
-
-## After Starting The System
-
-You can also set API-Key in **User Setting** as following:
-
-<div align="center" style="margin-top:20px;margin-bottom:20px;">
-<img src="https://github.com/infiniflow/ragflow/assets/12318111/e4e4066c-e964-45ff-bd56-c3fc7fb18bd3" width="1000"/>
-</div>
-
+---
+sidebar_position: 4
+slug: /llm_api_key_setup
+---
+
+# Set your LLM API key
+
+You have two ways to input your LLM API key. 
+
+## Before Starting The System
+
+In **user_default_llm** of [service_conf.yaml](./docker/service_conf.yaml), you need to specify LLM factory and your own _API_KEY_. 
+RagFlow supports the flowing LLM factory, and with more coming in the pipeline:
+
+> [OpenAI](https://platform.openai.com/login?launch), [Tongyi-Qianwen](https://dashscope.console.aliyun.com/model), 
+> [ZHIPU-AI](https://open.bigmodel.cn/), [Moonshot](https://platform.moonshot.cn/docs)
+
+After sign in these LLM suppliers, create your own API-Key, they all have a certain amount of free quota.
+
+## After Starting The System
+
+You can also set API-Key in **User Setting** as following:
+
+![](https://github.com/infiniflow/ragflow/assets/12318111/e4e4066c-e964-45ff-bd56-c3fc7fb18bd3)
+

docs/{manage_files.md → guides/manage_files.md}

@@ -1,3 +1,8 @@
+---
+sidebar_position: 3
+slug: /manage_files
+---
+
 #  Manage files
 
 Knowledge base, hallucination-free chat, and file management are three pillars of RAGFlow. RAGFlow's file management allows you to upload files individually or in bulk. You can then link an uploaded file to multiple target knowledge bases. This guide showcases some basic usages of the file management feature. 
@@ -40,11 +45,11 @@ You can link your file to one knowledge base or multiple knowledge bases at one
 
 ## Move file to specified folder
 
-As of RAGFlow v0.5.0, this feature is *not* available.
+As of RAGFlow v0.6.0, this feature is *not* available.
 
 ## Search files or folders
 
-As of RAGFlow v0.5.0, the search feature is still in a rudimentary form, supporting only file and folder search in the current directory by name (files or folders in the child directory will not be retrieved).
+As of RAGFlow v0.6.0, the search feature is still in a rudimentary form, supporting only file and folder search in the current directory by name (files or folders in the child directory will not be retrieved).
 
 ![search file](https://github.com/infiniflow/ragflow/assets/93570324/77ffc2e5-bd80-4ed1-841f-068e664efffe)
 
@@ -76,4 +81,4 @@ RAGFlow's file management allows you to download an uploaded file:
 
 ![download_file](https://github.com/infiniflow/ragflow/assets/93570324/cf3b297f-7d9b-4522-bf5f-4f45743e4ed5)
 
-> As of RAGFlow v0.5.0, bulk download is not supported, nor can you download an entire folder. 
+> As of RAGFlow v0.6.0, bulk download is not supported, nor can you download an entire folder. 

docs/{max_map_count.md → guides/max_map_count.md}

@@ -1,66 +1,71 @@
-# Set vm.max_map_count to at least 262144
-
-## Linux
-
-To check the value of `vm.max_map_count`:
-
-```bash
-$ sysctl vm.max_map_count
-```
-
-Reset `vm.max_map_count` to a value at least 262144 if it is not.
-
-```bash
-# In this case, we set it to 262144:
-$ sudo sysctl -w vm.max_map_count=262144
-```
-
-This change will be reset after a system reboot. To ensure your change remains permanent, add or update the `vm.max_map_count` value in **/etc/sysctl.conf** accordingly:
-
-```bash
-vm.max_map_count=262144
-```
-
-## Mac
-
-```bash
-$ screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
-$ sysctl -w vm.max_map_count=262144
-```
-To exit the screen session, type Ctrl a d.
-
-## Windows and macOS with Docker Desktop
-
-The vm.max_map_count setting must be set via docker-machine:
-
-```bash
-$ docker-machine ssh
-$ sudo sysctl -w vm.max_map_count=262144
-```
-
-## Windows with Docker Desktop WSL 2 backend
-
-To manually set it every time you reboot, you must run the following commands in a command prompt or PowerShell window every time you restart Docker:
-
-```bash
-$ wsl -d docker-desktop -u root
-$ sysctl -w vm.max_map_count=262144
-```
-If you are on these versions of WSL and you do not want to have to run those commands every time you restart Docker, you can globally change every WSL distribution with this setting by modifying your %USERPROFILE%\.wslconfig as follows:
-
-```bash
-[wsl2]
-kernelCommandLine = "sysctl.vm.max_map_count=262144"
-```
-This will cause all WSL2 VMs to have that setting assigned when they start.
-
-If you are on Windows 11, or Windows 10 version 22H2 and have installed the Microsoft Store version of WSL, you can modify the /etc/sysctl.conf within the "docker-desktop" WSL distribution, perhaps with commands like this:
-
-```bash
-$ wsl -d docker-desktop -u root
-$ vi /etc/sysctl.conf
-```
-and appending a line which reads:
-```bash
-vm.max_map_count = 262144
+---
+sidebar_position: 7
+slug: /max_map_count
+---
+
+# Update vm.max_map_count
+
+## Linux
+
+To check the value of `vm.max_map_count`:
+
+```bash
+$ sysctl vm.max_map_count
+```
+
+Reset `vm.max_map_count` to a value at least 262144 if it is not.
+
+```bash
+# In this case, we set it to 262144:
+$ sudo sysctl -w vm.max_map_count=262144
+```
+
+This change will be reset after a system reboot. To ensure your change remains permanent, add or update the `vm.max_map_count` value in **/etc/sysctl.conf** accordingly:
+
+```bash
+vm.max_map_count=262144
+```
+
+## Mac
+
+```bash
+$ screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
+$ sysctl -w vm.max_map_count=262144
+```
+To exit the screen session, type Ctrl a d.
+
+## Windows and macOS with Docker Desktop
+
+The vm.max_map_count setting must be set via docker-machine:
+
+```bash
+$ docker-machine ssh
+$ sudo sysctl -w vm.max_map_count=262144
+```
+
+## Windows with Docker Desktop WSL 2 backend
+
+To manually set it every time you reboot, you must run the following commands in a command prompt or PowerShell window every time you restart Docker:
+
+```bash
+$ wsl -d docker-desktop -u root
+$ sysctl -w vm.max_map_count=262144
+```
+If you are on these versions of WSL and you do not want to have to run those commands every time you restart Docker, you can globally change every WSL distribution with this setting by modifying your %USERPROFILE%\.wslconfig as follows:
+
+```bash
+[wsl2]
+kernelCommandLine = "sysctl.vm.max_map_count=262144"
+```
+This will cause all WSL2 VMs to have that setting assigned when they start.
+
+If you are on Windows 11, or Windows 10 version 22H2 and have installed the Microsoft Store version of WSL, you can modify the /etc/sysctl.conf within the "docker-desktop" WSL distribution, perhaps with commands like this:
+
+```bash
+$ wsl -d docker-desktop -u root
+$ vi /etc/sysctl.conf
+```
+and appending a line which reads:
+```bash
+vm.max_map_count = 262144
 ```

docs/{start_chat.md → guides/start_chat.md}

@@ -1,3 +1,8 @@
+---
+sidebar_position: 2
+slug: /start_chat
+---
+
 # Start an AI chat
 
 Knowledge base, hallucination-free chat, and file management are three pillars of RAGFlow. Chats in RAGFlow are based on a particular knowledge base or multiple knowledge bases. Once you have created your knowledge base and finished file parsing, you can go ahead and start an AI conversation. 

docs/ollama.md

@@ -1,40 +0,0 @@
-# Ollama
-
-<div align="center" style="margin-top:20px;margin-bottom:20px;">
-<img src="https://github.com/infiniflow/ragflow/assets/12318111/2019e7ee-1e8a-412e-9349-11bbf702e549" width="130"/>
-</div>
-
-One-click deployment of local LLMs, that is [Ollama](https://github.com/ollama/ollama).
-
-## Install
-
-- [Ollama on Linux](https://github.com/ollama/ollama/blob/main/docs/linux.md)
-- [Ollama Windows Preview](https://github.com/ollama/ollama/blob/main/docs/windows.md)
-- [Docker](https://hub.docker.com/r/ollama/ollama)
-
-## Launch Ollama
-
-Decide which LLM you want to deploy ([here's a list for supported LLM](https://ollama.com/library)), say, **mistral**:
-```bash
-$ ollama run mistral
-```
-Or,
-```bash
-$ docker exec -it ollama ollama run mistral
-```
-
-## Use Ollama in RAGFlow
-
-- Go to 'Settings > Model Providers > Models to be added > Ollama'.
-    
-<div align="center" style="margin-top:20px;margin-bottom:20px;">
-<img src="https://github.com/infiniflow/ragflow/assets/12318111/a9df198a-226d-4f30-b8d7-829f00256d46" width="1300"/>
-</div>
-
-> Base URL: Enter the base URL where the Ollama service is accessible, like, `http://<your-ollama-endpoint-domain>:11434`.
-
-- Use Ollama Models.
-
-<div align="center" style="margin-top:20px;margin-bottom:20px;">
-<img src="https://github.com/infiniflow/ragflow/assets/12318111/60ff384e-5013-41ff-a573-9a543d237fd3" width="530"/>
-</div>

docs/quickstart.md

@@ -1,4 +1,9 @@
-# Quickstart
+---
+sidebar_position: 1
+slug: /
+---
+
+# Quick start
 
 RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. When integrated with LLMs, it is capable of providing truthful question-answering capabilities, backed by well-founded citations from various complex formatted data.
 
@@ -20,7 +25,7 @@ This quick start guide describes a general process from:
 
 ## Start up the server
 
-1. Ensure `vm.max_map_count` >= 262144 ([more](./docs/max_map_count.md)):
+1. Ensure `vm.max_map_count` >= 262144:
 
    > To check the value of `vm.max_map_count`:
    >

docs/{CONTRIBUTING.md → references/CONTRIBUTING.md}

@@ -1,52 +1,52 @@
----
-sidebar_position: 0
-slug: /contribution_guidelines
----
-
-# Contribution Guidelines
-
-Thanks for wanting to contribute to RAGFlow. This document offers guidlines and major considerations for submitting your contributions. 
-
-- To report a bug, file a [GitHub issue](https://github.com/infiniflow/ragflow/issues/new/choose) with us.
-- For further questions, you can explore existing discussions or initiate a new one in [Discussions](https://github.com/orgs/infiniflow/discussions).
-
-
-## What you can contribute
-
-The list below mentions some contributions you can make, but it is not a complete list.
-
-- Proposing or implementing new features
-- Fixing a bug
-- Adding test cases or demos
-- Posting a blog or tutorial
-- Updates to existing documents, codes, or annotations.
-- Suggesting more user-friendly error codes
-
-## File a pull request (PR)
-
-### General workflow
-
-1. Fork our GitHub repository.
-2. Clone your fork to your local machine: 
-`git clone [email protected]:<yourname>/ragflow.git`
-3. Create a local branch: 
-`git checkout -b my-branch`
-4. Provide sufficient information in your commit message
-`git commit -m 'Provide sufficient info in your commit message'`
-5. Commit changes to your local branch, and push to GitHub: (include necessary commit message)
-`git push origin my-branch.`
-6. Submit a pull request for review.
-
-### Before filing a PR
-
-- Consider splitting a large PR into multiple smaller, standalone PRs to keep a traceable development history. 
-- Ensure that your PR addresses just one issue, or keep any unrelated changes small.
-- Add test cases when contributing new features. They demonstrate that your code functions correctly and protect against potential issues from future changes.
-### Describing your PR 
-
-- Ensure that your PR title is concise and clear, providing all the required information.
-- Refer to a corresponding GitHub issue in your PR description if applicable. 
-- Include sufficient design details for *breaking changes* or *API changes* in your description.
-
-### Reviewing & merging a PR
+---
+sidebar_position: 0
+slug: /contribution_guidelines
+---
+
+# Contribution guidelines
+
+Thanks for wanting to contribute to RAGFlow. This document offers guidlines and major considerations for submitting your contributions. 
+
+- To report a bug, file a [GitHub issue](https://github.com/infiniflow/ragflow/issues/new/choose) with us.
+- For further questions, you can explore existing discussions or initiate a new one in [Discussions](https://github.com/orgs/infiniflow/discussions).
+
+
+## What you can contribute
+
+The list below mentions some contributions you can make, but it is not a complete list.
+
+- Proposing or implementing new features
+- Fixing a bug
+- Adding test cases or demos
+- Posting a blog or tutorial
+- Updates to existing documents, codes, or annotations.
+- Suggesting more user-friendly error codes
+
+## File a pull request (PR)
+
+### General workflow
+
+1. Fork our GitHub repository.
+2. Clone your fork to your local machine: 
+`git clone [email protected]:<yourname>/ragflow.git`
+3. Create a local branch: 
+`git checkout -b my-branch`
+4. Provide sufficient information in your commit message
+`git commit -m 'Provide sufficient info in your commit message'`
+5. Commit changes to your local branch, and push to GitHub: (include necessary commit message)
+`git push origin my-branch.`
+6. Submit a pull request for review.
+
+### Before filing a PR
+
+- Consider splitting a large PR into multiple smaller, standalone PRs to keep a traceable development history. 
+- Ensure that your PR addresses just one issue, or keep any unrelated changes small.
+- Add test cases when contributing new features. They demonstrate that your code functions correctly and protect against potential issues from future changes.
+### Describing your PR 
+
+- Ensure that your PR title is concise and clear, providing all the required information.
+- Refer to a corresponding GitHub issue in your PR description if applicable. 
+- Include sufficient design details for *breaking changes* or *API changes* in your description.
+
+### Reviewing & merging a PR
 - Ensure that your PR passes all Continuous Integration (CI) tests before merging it.

docs/references/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "References",
+  "position": 1,
+  "link": {
+    "type": "generated-index",
+    "description": "RAGFlow References"
+  }
+}

docs/{conversation_api.md → references/api.md}

@@ -1,403 +1,406 @@
-# Conversation API Instruction
-
-<div align="center" style="margin-top:20px;margin-bottom:20px;">
-<img src="https://github.com/infiniflow/ragflow/assets/12318111/df0dcc3d-789a-44f7-89f1-7a5f044ab729" width="830"/>
-</div>
-
-## Base URL
-```buildoutcfg
-https://demo.ragflow.io/v1/
-```
-
-## Authorization
-
-All the APIs are authorized with API-Key. Please keep it safe and private. Don't reveal it in any way from the front-end. 
-The API-Key should put in the header of request:
-```buildoutcfg
-Authorization: Bearer {API_KEY}
-```
-
-## Start a conversation
-
-This should be called whenever there's new user coming to chat.
-### Path: /api/new_conversation
-### Method: GET
-### Parameter:
-
-| name | type | optional | description|
-|------|-------|----|----|
-| user_id| string | No | It's for identifying user in order to search and calculate statistics.|
-
-### Response 
-```json
-{
-    "data": {
-        "create_date": "Fri, 12 Apr 2024 17:26:21 GMT",
-        "create_time": 1712913981857,
-        "dialog_id": "4f0a2e4cb9af11ee9ba20aef05f5e94f",
-        "duration": 0.0,
-        "id": "b9b2e098f8ae11ee9f45fa163e197198",
-        "message": [
-            {
-                "content": "Hi, I'm your assistant, can I help you?",
-                "role": "assistant"
-            }
-        ],
-        "reference": [],
-        "tokens": 0,
-        "update_date": "Fri, 12 Apr 2024 17:26:21 GMT",
-        "update_time": 1712913981857,
-        "user_id": "kevinhu"
-    },
-    "retcode": 0,
-    "retmsg": "success"
-}
-```
-> data['id'] in response should be stored and will be used in every round of following conversation.
-
-## Get history of a conversation
-
-### Path: /api/conversation/\<id\>
-### Method: GET
-### Response 
-```json
-{
-    "data": {
-        "create_date": "Mon, 01 Apr 2024 09:28:42 GMT",
-        "create_time": 1711934922220,
-        "dialog_id": "df4a4916d7bd11eeaa650242ac180006",
-        "id": "2cae30fcefc711ee94140242ac180006",
-        "message": [
-            {
-                "content": "Hi! I'm your assistant, what can I do for you?",
-                "role": "assistant"
-            },
-            {
-                "content": "What's the vit score for GPT-4?",
-                "role": "user"
-            },
-            {
-                "content": "The ViT Score for GPT-4 in the zero-shot scenario is 0.5058, and in the few-shot scenario, it is 0.6480. ##0$$",
-                "role": "assistant"
-            },
-            {
-                "content": "How is the nvlink topology like?",
-                "role": "user"
-            },
-            {
-                "content": "NVLink topology refers to the arrangement of connections between GPUs using NVIDIA's NVLink technology. Correct NVLink topology for NVIDIA A100 cards involves connecting one GPU to another through a series of NVLink bridges ##0$$. Each of the three attached bridges spans two PCIe slots, and for optimal performance and balanced bridge topology, all three NVLink bridges should be used when connecting two adjacent A100 cards.\n\nHere's a summary of the correct and incorrect topologies:\n\n- **Correct**: Both GPUs are connected via all three NVLink bridges, ensuring full bandwidth and proper communication.\n- **Incorrect**: Not using all three bridges or having an uneven connection configuration would result in suboptimal performance.\n\nIt's also important to note that for multi-CPU systems, both A100 cards in a bridged pair should be within the same CPU domain, unless each CPU has a single A100 PCIe card, in which case they can be bridged together.",
-                "role": "assistant"
-            }
-        ],
-        "user_id": "user name",
-        "reference": [
-            {
-                "chunks": [
-                    {
-                        "chunk_id": "d0bc7892c3ec4aeac071544fd56730a8",
-                        "content_ltks": "tabl 1:openagi task-solv perform under differ set for three closed-sourc llm . boldfac denot the highest score under each learn schema . metric gpt-3.5-turbo claude-2 gpt-4 zero few zero few zero few clip score 0.0 0.0 0.0 0.2543 0.0 0.3055 bert score 0.1914 0.3820 0.2111 0.5038 0.2076 0.6307 vit score 0.2437 0.7497 0.4082 0.5416 0.5058 0.6480 overal 0.1450 0.3772 0.2064 0.4332 0.2378 0.5281",
-                        "content_with_weight": "<table><caption>Table 1: OpenAGI task-solving performances under different settings for three closed-source LLMs. Boldface denotes the highest score under each learning schema.</caption>\n<tr><th  rowspan=2 >Metrics</th><th  >GPT-3.5-turbo</th><th></th><th  >Claude-2</th><th  >GPT-4</th></tr>\n<tr><th  >Zero</th><th  >Few</th><th  >Zero Few</th><th  >Zero Few</th></tr>\n<tr><td  >CLIP Score</td><td  >0.0</td><td  >0.0</td><td  >0.0 0.2543</td><td  >0.0 0.3055</td></tr>\n<tr><td  >BERT Score</td><td  >0.1914</td><td  >0.3820</td><td  >0.2111 0.5038</td><td  >0.2076 0.6307</td></tr>\n<tr><td  >ViT Score</td><td  >0.2437</td><td  >0.7497</td><td  >0.4082 0.5416</td><td  >0.5058 0.6480</td></tr>\n<tr><td  >Overall</td><td  >0.1450</td><td  >0.3772</td><td  >0.2064 0.4332</td><td  >0.2378 0.5281</td></tr>\n</table>",
-                        "doc_id": "c790da40ea8911ee928e0242ac180005",
-                        "docnm_kwd": "OpenAGI When LLM Meets Domain Experts.pdf",
-                        "img_id": "afab9fdad6e511eebdb20242ac180006-d0bc7892c3ec4aeac071544fd56730a8",
-                        "important_kwd": [],
-                        "kb_id": "afab9fdad6e511eebdb20242ac180006",
-                        "positions": [
-                            [
-                                9.0,
-                                159.9383341471354,
-                                472.1773274739583,
-                                223.58013916015625,
-                                307.86692301432294
-                            ]
-                        ],
-                        "similarity": 0.7310340654129031,
-                        "term_similarity": 0.7671974387781668,
-                        "vector_similarity": 0.40556370512552886
-                    },
-                    {
-                        "chunk_id": "7e2345d440383b756670e1b0f43a7007",
-                        "content_ltks": "5.5 experiment analysi the main experiment result are tabul in tab . 1 and 2 , showcas the result for closed-sourc and open-sourc llm , respect . the overal perform is calcul a the averag of cllp 8 bert and vit score . here , onli the task descript of the benchmark task are fed into llm(addit inform , such a the input prompt and llm\u2019output , is provid in fig . a.4 and a.5 in supplementari). broadli speak , closed-sourc llm demonstr superior perform on openagi task , with gpt-4 lead the pack under both zero-and few-shot scenario . in the open-sourc categori , llama-2-13b take the lead , consist post top result across variou learn schema--the perform possibl influenc by it larger model size . notabl , open-sourc llm significantli benefit from the tune method , particularli fine-tun and\u2019rltf . these method mark notic enhanc for flan-t5-larg , vicuna-7b , and llama-2-13b when compar with zero-shot and few-shot learn schema . in fact , each of these open-sourc model hit it pinnacl under the rltf approach . conclus , with rltf tune , the perform of llama-2-13b approach that of gpt-3.5 , illustr it potenti .",
-                        "content_with_weight": "5.5 Experimental Analysis\nThe main experimental results are tabulated in Tab. 1 and 2, showcasing the results for closed-source and open-source LLMs, respectively. The overall performance is calculated as the average of CLlP\n8\nBERT and ViT scores. Here, only the task descriptions of the benchmark tasks are fed into LLMs (additional information, such as the input prompt and LLMs\u2019 outputs, is provided in Fig. A.4 and A.5 in supplementary). Broadly speaking, closed-source LLMs demonstrate superior performance on OpenAGI tasks, with GPT-4 leading the pack under both zero- and few-shot scenarios. In the open-source category, LLaMA-2-13B takes the lead, consistently posting top results across various learning schema--the performance possibly influenced by its larger model size. Notably, open-source LLMs significantly benefit from the tuning methods, particularly Fine-tuning and\u2019 RLTF. These methods mark noticeable enhancements for Flan-T5-Large, Vicuna-7B, and LLaMA-2-13B when compared with zero-shot and few-shot learning schema. In fact, each of these open-source models hits its pinnacle under the RLTF approach. Conclusively, with RLTF tuning, the performance of LLaMA-2-13B approaches that of GPT-3.5, illustrating its potential.",
-                        "doc_id": "c790da40ea8911ee928e0242ac180005",
-                        "docnm_kwd": "OpenAGI When LLM Meets Domain Experts.pdf",
-                        "img_id": "afab9fdad6e511eebdb20242ac180006-7e2345d440383b756670e1b0f43a7007",
-                        "important_kwd": [],
-                        "kb_id": "afab9fdad6e511eebdb20242ac180006",
-                        "positions": [
-                            [
-                                8.0,
-                                107.3,
-                                508.90000000000003,
-                                686.3,
-                                697.0
-                            ],
-                        ],
-                        "similarity": 0.6691508616357027,
-                        "term_similarity": 0.6999011754270821,
-                        "vector_similarity": 0.39239803751328806
-                    },
-                ],
-                "doc_aggs": [
-                    {
-                        "count": 8,
-                        "doc_id": "c790da40ea8911ee928e0242ac180005",
-                        "doc_name": "OpenAGI When LLM Meets Domain Experts.pdf"
-                    }
-                ],
-                "total": 8
-            },
-            {
-                "chunks": [
-                    {
-                        "chunk_id": "8c11a1edddb21ad2ae0c43b4a5dcfa62",
-                        "content_ltks": "nvlink bridg support nvidia\u00aenvlink\u00aei a high-spe point-to-point peer transfer connect , where one gpu can transfer data to and receiv data from one other gpu . the nvidia a100 card support nvlink bridg connect with a singl adjac a100 card . each of the three attach bridg span two pcie slot . to function correctli a well a to provid peak bridg bandwidth , bridg connect with an adjac a100 card must incorpor all three nvlink bridg . wherev an adjac pair of a100 card exist in the server , for best bridg perform and balanc bridg topolog , the a100 pair should be bridg . figur 4 illustr correct and incorrect a100 nvlink connect topolog . nvlink topolog\u2013top view figur 4. correct incorrect correct incorrect for system that featur multipl cpu , both a100 card of a bridg card pair should be within the same cpu domain\u2014that is , under the same cpu\u2019s topolog . ensur thi benefit workload applic perform . the onli except is for dual cpu system wherein each cpu ha a singl a100 pcie card under it;in that case , the two a100 pcie card in the system may be bridg togeth . a100 nvlink speed and bandwidth are given in the follow tabl . tabl 5. a100 nvlink speed and bandwidth paramet valu total nvlink bridg support by nvidia a100 3 total nvlink rx and tx lane support 96 data rate per nvidia a100 nvlink lane(each direct)50 gbp total maximum nvlink bandwidth 600 gbyte per second pb-10137-001_v03|8 nvidia a100 40gb pcie gpu acceler",
-                        "content_with_weight": "NVLink Bridge Support\nNVIDIA\u00aeNVLink\u00aeis a high-speed point-to-point peer transfer connection, where one GPU can transfer data to and receive data from one other GPU. The NVIDIA A100 card supports NVLink bridge connection with a single adjacent A100 card.\nEach of the three attached bridges spans two PCIe slots. To function correctly as well as to provide peak bridge bandwidth, bridge connection with an adjacent A100 card must incorporate all three NVLink bridges. Wherever an adjacent pair of A100 cards exists in the server, for best bridging performance and balanced bridge topology, the A100 pair should be bridged. Figure 4 illustrates correct and incorrect A100 NVLink connection topologies.\nNVLink Topology \u2013Top Views \nFigure 4. \nCORRECT \nINCORRECT \nCORRECT \nINCORRECT \nFor systems that feature multiple CPUs, both A100 cards of a bridged card pair should be within the same CPU domain\u2014that is, under the same CPU\u2019s topology. Ensuring this benefits workload application performance. The only exception is for dual CPU systems wherein each CPU has a single A100 PCIe card under it; in that case, the two A100 PCIe cards in the system may be bridged together.\nA100 NVLink speed and bandwidth are given in the following table.\n<table><caption>Table 5. A100 NVLink Speed and Bandwidth </caption>\n<tr><th  >Parameter </th><th  >Value </th></tr>\n<tr><td  >Total NVLink bridges supported by NVIDIA A100 </td><td  >3 </td></tr>\n<tr><td  >Total NVLink Rx and Tx lanes supported </td><td  >96 </td></tr>\n<tr><td  >Data rate per NVIDIA A100 NVLink lane (each direction)</td><td  >50 Gbps </td></tr>\n<tr><td  >Total maximum NVLink bandwidth</td><td  >600 Gbytes per second </td></tr>\n</table>\nPB-10137-001_v03 |8\nNVIDIA A100 40GB PCIe GPU Accelerator",
-                        "doc_id": "806d1ed0ea9311ee860a0242ac180005",
-                        "docnm_kwd": "A100-PCIE-Prduct-Brief.pdf",
-                        "img_id": "afab9fdad6e511eebdb20242ac180006-8c11a1edddb21ad2ae0c43b4a5dcfa62",
-                        "important_kwd": [],
-                        "kb_id": "afab9fdad6e511eebdb20242ac180006",
-                        "positions": [
-                            [
-                                12.0,
-                                84.0,
-                                541.3,
-                                76.7,
-                                96.7
-                            ],
-                        ],
-                        "similarity": 0.3200748779905588,
-                        "term_similarity": 0.3082244010114718,
-                        "vector_similarity": 0.42672917080234146
-                    },
-                ],
-                "doc_aggs": [
-                    {
-                        "count": 1,
-                        "doc_id": "806d1ed0ea9311ee860a0242ac180005",
-                        "doc_name": "A100-PCIE-Prduct-Brief.pdf"
-                    }
-                ],
-                "total": 3
-            }
-        ],
-        "update_date": "Tue, 02 Apr 2024 09:07:49 GMT",
-        "update_time": 1712020069421
-    },
-    "retcode": 0,
-    "retmsg": "success"
-}
-```
-
-- **message**: All the chat history in it.
-    - role: user or assistant
-    - content: the text content of user or assistant. The citations are in format like: ##0$$. The number in the middle indicate which part in data.reference.chunks it refers to.
-    
-- **user_id**: This is set by the caller.
-- **reference**: Every item in it refer to the corresponding message in data.message whose role is assistant. 
-    - chunks
-        - content_with_weight: The content of chunk.
-        - docnm_kwd: the document name.
-        - img_id: the image id of the chunk. It is an optional field only for PDF/pptx/picture. And accessed by 'GET' /document/get/\<id\>.
-        - positions: [page_number, [upleft corner(x, y)], [right bottom(x, y)]], the chunk position, only for PDF.
-        - similarity: the hybrid similarity.
-        - term_similarity: keyword simimlarity
-        - vector_similarity: embedding similarity
-    - doc_aggs:
-        - doc_id: the document can be accessed by 'GET' /document/get/\<id\>
-        - doc_name: the file name
-        - count: the chunk number hit in this document.
-    
-## Chat
-
-This will be called to get the answer to users' questions.
-
-### Path: /api/completion
-### Method: POST
-### Parameter:
-
-| name | type | optional | description|
-|------|-------|----|----|
-| conversation_id| string | No | This is from calling /new_conversation.|
-| messages| json | No | The latest question, such as `[{"role": "user", "content": "How are you doing!"}]`|
-| quote | bool | Yes | Default: true |
-| stream | bool | Yes | Default: true |
-| doc_ids | string | Yes | Document IDs which is delimited by comma, like `c790da40ea8911ee928e0242ac180005,c790da40ea8911ee928e0242ac180005`. The retrieved content is limited in these documents. |
-
-### Response 
-```json
-{
-    "data": {
-      "answer": "The ViT Score for GPT-4 in the zero-shot scenario is 0.5058, and in the few-shot scenario, it is 0.6480. ##0$$",
-      "reference": {
-        "chunks": [
-          {
-            "chunk_id": "d0bc7892c3ec4aeac071544fd56730a8",
-            "content_ltks": "tabl 1:openagi task-solv perform under differ set for three closed-sourc llm . boldfac denot the highest score under each learn schema . metric gpt-3.5-turbo claude-2 gpt-4 zero few zero few zero few clip score 0.0 0.0 0.0 0.2543 0.0 0.3055 bert score 0.1914 0.3820 0.2111 0.5038 0.2076 0.6307 vit score 0.2437 0.7497 0.4082 0.5416 0.5058 0.6480 overal 0.1450 0.3772 0.2064 0.4332 0.2378 0.5281",
-            "content_with_weight": "<table><caption>Table 1: OpenAGI task-solving performances under different settings for three closed-source LLMs. Boldface denotes the highest score under each learning schema.</caption>\n<tr><th  rowspan=2 >Metrics</th><th  >GPT-3.5-turbo</th><th></th><th  >Claude-2</th><th  >GPT-4</th></tr>\n<tr><th  >Zero</th><th  >Few</th><th  >Zero Few</th><th  >Zero Few</th></tr>\n<tr><td  >CLIP Score</td><td  >0.0</td><td  >0.0</td><td  >0.0 0.2543</td><td  >0.0 0.3055</td></tr>\n<tr><td  >BERT Score</td><td  >0.1914</td><td  >0.3820</td><td  >0.2111 0.5038</td><td  >0.2076 0.6307</td></tr>\n<tr><td  >ViT Score</td><td  >0.2437</td><td  >0.7497</td><td  >0.4082 0.5416</td><td  >0.5058 0.6480</td></tr>\n<tr><td  >Overall</td><td  >0.1450</td><td  >0.3772</td><td  >0.2064 0.4332</td><td  >0.2378 0.5281</td></tr>\n</table>",
-            "doc_id": "c790da40ea8911ee928e0242ac180005",
-            "docnm_kwd": "OpenAGI When LLM Meets Domain Experts.pdf",
-            "img_id": "afab9fdad6e511eebdb20242ac180006-d0bc7892c3ec4aeac071544fd56730a8",
-            "important_kwd": [],
-            "kb_id": "afab9fdad6e511eebdb20242ac180006",
-            "positions": [
-              [
-                9.0,
-                159.9383341471354,
-                472.1773274739583,
-                223.58013916015625,
-                307.86692301432294
-              ]
-            ],
-            "similarity": 0.7310340654129031,
-            "term_similarity": 0.7671974387781668,
-            "vector_similarity": 0.40556370512552886
-          },
-          {
-            "chunk_id": "7e2345d440383b756670e1b0f43a7007",
-            "content_ltks": "5.5 experiment analysi the main experiment result are tabul in tab . 1 and 2 , showcas the result for closed-sourc and open-sourc llm , respect . the overal perform is calcul a the averag of cllp 8 bert and vit score . here , onli the task descript of the benchmark task are fed into llm(addit inform , such a the input prompt and llm\u2019output , is provid in fig . a.4 and a.5 in supplementari). broadli speak , closed-sourc llm demonstr superior perform on openagi task , with gpt-4 lead the pack under both zero-and few-shot scenario . in the open-sourc categori , llama-2-13b take the lead , consist post top result across variou learn schema--the perform possibl influenc by it larger model size . notabl , open-sourc llm significantli benefit from the tune method , particularli fine-tun and\u2019rltf . these method mark notic enhanc for flan-t5-larg , vicuna-7b , and llama-2-13b when compar with zero-shot and few-shot learn schema . in fact , each of these open-sourc model hit it pinnacl under the rltf approach . conclus , with rltf tune , the perform of llama-2-13b approach that of gpt-3.5 , illustr it potenti .",
-            "content_with_weight": "5.5 Experimental Analysis\nThe main experimental results are tabulated in Tab. 1 and 2, showcasing the results for closed-source and open-source LLMs, respectively. The overall performance is calculated as the average of CLlP\n8\nBERT and ViT scores. Here, only the task descriptions of the benchmark tasks are fed into LLMs (additional information, such as the input prompt and LLMs\u2019 outputs, is provided in Fig. A.4 and A.5 in supplementary). Broadly speaking, closed-source LLMs demonstrate superior performance on OpenAGI tasks, with GPT-4 leading the pack under both zero- and few-shot scenarios. In the open-source category, LLaMA-2-13B takes the lead, consistently posting top results across various learning schema--the performance possibly influenced by its larger model size. Notably, open-source LLMs significantly benefit from the tuning methods, particularly Fine-tuning and\u2019 RLTF. These methods mark noticeable enhancements for Flan-T5-Large, Vicuna-7B, and LLaMA-2-13B when compared with zero-shot and few-shot learning schema. In fact, each of these open-source models hits its pinnacle under the RLTF approach. Conclusively, with RLTF tuning, the performance of LLaMA-2-13B approaches that of GPT-3.5, illustrating its potential.",
-            "doc_id": "c790da40ea8911ee928e0242ac180005",
-            "docnm_kwd": "OpenAGI When LLM Meets Domain Experts.pdf",
-            "img_id": "afab9fdad6e511eebdb20242ac180006-7e2345d440383b756670e1b0f43a7007",
-            "important_kwd": [],
-            "kb_id": "afab9fdad6e511eebdb20242ac180006",
-            "positions": [
-              [
-                8.0,
-                107.3,
-                508.90000000000003,
-                686.3,
-                697.0
-              ]
-            ],
-            "similarity": 0.6691508616357027,
-            "term_similarity": 0.6999011754270821,
-            "vector_similarity": 0.39239803751328806
-          }
-        ],
-        "doc_aggs": {
-          "OpenAGI When LLM Meets Domain Experts.pdf": 4
-        },
-        "total": 8
-      }
-    },
-    "retcode": 0,
-    "retmsg": "success"
-}
-```
-
-- **answer**: The replay of the chat bot.
-- **reference**: 
-    - chunks: Every item in it refer to the corresponding message in answer. 
-        - content_with_weight: The content of chunk.
-        - docnm_kwd: the document name.
-        - img_id: the image id of the chunk. It is an optional field only for PDF/pptx/picture. And accessed by 'GET' /document/get/\<id\>.
-        - positions: [page_number, [upleft corner(x, y)], [right bottom(x, y)]], the chunk position, only for PDF.
-        - similarity: the hybrid similarity.
-        - term_similarity: keyword simimlarity
-        - vector_similarity: embedding similarity
-    - doc_aggs:
-        - doc_id: the document can be accessed by 'GET' /document/get/\<id\>
-        - doc_name: the file name
-        - count: the chunk number hit in this document.
-    
-## Get document content or image
-
-This is usually used when display content of citation.
-### Path: /api/document/get/\<id\>
-### Method: GET
-
-## Upload file
-
-This is usually used when upload a file to.
-### Path: /api/document/upload/
-### Method: POST
-
-### Parameter:
-
-| name      | type   | optional | description                                             |
-|-----------|--------|----------|---------------------------------------------------------|
-| file      | file   | No       | Upload file.                                            |
-| kb_name   | string | No       | Choose the upload knowledge base name.                  |
-| parser_id | string | Yes      | Choose the parsing method.                              |
-| run       | string | Yes      | Parsing will start automatically when the value is "1". |
-
-### Response 
-```json
-{
-    "data": {
-        "chunk_num": 0,
-        "create_date": "Thu, 25 Apr 2024 14:30:06 GMT",
-        "create_time": 1714026606921,
-        "created_by": "553ec818fd5711ee8ea63043d7ed348e",
-        "id": "41e9324602cd11ef9f5f3043d7ed348e",
-        "kb_id": "06802686c0a311ee85d6246e9694c130",
-        "location": "readme.txt",
-        "name": "readme.txt",
-        "parser_config": {
-            "field_map": {
-            },
-            "pages": [
-                [
-                    0,
-                    1000000
-                ]
-            ]
-        },
-        "parser_id": "general",
-        "process_begin_at": null,
-        "process_duation": 0.0,
-        "progress": 0.0,
-        "progress_msg": "",
-        "run": "0",
-        "size": 929,
-        "source_type": "local",
-        "status": "1",
-        "thumbnail": null,
-        "token_num": 0,
-        "type": "doc",
-        "update_date": "Thu, 25 Apr 2024 14:30:06 GMT",
-        "update_time": 1714026606921
-    },
-    "retcode": 0,
-    "retmsg": "success"
-}
-
-```
-
-## Get document chunks
-
-Get the chunks of the document based on doc_name or doc_id.
-### Path: /api/list_chunks/
-### Method: POST
-
-### Parameter:
-
-| Name     | Type   | Optional | Description                     |
-|----------|--------|----------|---------------------------------|
-| `doc_name` | string | Yes      | The name of the document in the knowledge base. It must not be empty if `doc_id` is not set.|
-| `doc_id`   | string | Yes      | The ID of the document in the knowledge base. It must not be empty if `doc_name` is not set.|
-
-
-### Response 
-```json
-{
-    "data": [
-        {
-            "content": "Figure 14: Per-request neural-net processingof RL-Cache.\n103\n(sn)\nCPU\n 102\nGPU\n8101\n100\n8\n16 64 256 1K\n4K",
-            "doc_name": "RL-Cache.pdf",
-            "img_id": "0335167613f011ef91240242ac120006-b46c3524952f82dbe061ce9b123f2211"
-        },
-        {
-            "content": "4.3 ProcessingOverheadof RL-CacheACKNOWLEDGMENTSThis section evaluates how e￿ectively our RL-Cache implemen-tation leverages modern multi-core CPUs and GPUs to keep the per-request neural-net processing overhead low. Figure 14 depictsThis researchwas supported inpart by the Regional Government of Madrid (grant P2018/TCS-4499, EdgeData-CM)andU.S. National Science Foundation (grants CNS-1763617 andCNS-1717179).REFERENCES",
-            "doc_name": "RL-Cache.pdf",
-            "img_id": "0335167613f011ef91240242ac120006-d4c12c43938eb55d2d8278eea0d7e6d7"
-        }
-    ],
-    "retcode": 0,
-    "retmsg": "success"
-}
-
-```
+---
+sidebar_position: 1
+slug: /api
+---
+
+# API reference
+
+![](https://github.com/infiniflow/ragflow/assets/12318111/df0dcc3d-789a-44f7-89f1-7a5f044ab729)
+
+## Base URL
+```
+https://demo.ragflow.io/v1/
+```
+
+## Authorization
+
+All the APIs are authorized with API-Key. Please keep it safe and private. Don't reveal it in any way from the front-end. 
+The API-Key should put in the header of request:
+```buildoutcfg
+Authorization: Bearer {API_KEY}
+```
+
+## Start a conversation
+
+This should be called whenever there's new user coming to chat.
+### Path: /api/new_conversation
+### Method: GET
+### Parameter:
+
+| name | type | optional | description|
+|------|-------|----|----|
+| user_id| string | No | It's for identifying user in order to search and calculate statistics.|
+
+### Response 
+```json
+{
+    "data": {
+        "create_date": "Fri, 12 Apr 2024 17:26:21 GMT",
+        "create_time": 1712913981857,
+        "dialog_id": "4f0a2e4cb9af11ee9ba20aef05f5e94f",
+        "duration": 0.0,
+        "id": "b9b2e098f8ae11ee9f45fa163e197198",
+        "message": [
+            {
+                "content": "Hi, I'm your assistant, can I help you?",
+                "role": "assistant"
+            }
+        ],
+        "reference": [],
+        "tokens": 0,
+        "update_date": "Fri, 12 Apr 2024 17:26:21 GMT",
+        "update_time": 1712913981857,
+        "user_id": "kevinhu"
+    },
+    "retcode": 0,
+    "retmsg": "success"
+}
+```
+> data['id'] in response should be stored and will be used in every round of following conversation.
+
+## Get history of a conversation
+
+### Path: /api/conversation/\<id\>
+### Method: GET
+### Response 
+```json
+{
+    "data": {
+        "create_date": "Mon, 01 Apr 2024 09:28:42 GMT",
+        "create_time": 1711934922220,
+        "dialog_id": "df4a4916d7bd11eeaa650242ac180006",
+        "id": "2cae30fcefc711ee94140242ac180006",
+        "message": [
+            {
+                "content": "Hi! I'm your assistant, what can I do for you?",
+                "role": "assistant"
+            },
+            {
+                "content": "What's the vit score for GPT-4?",
+                "role": "user"
+            },
+            {
+                "content": "The ViT Score for GPT-4 in the zero-shot scenario is 0.5058, and in the few-shot scenario, it is 0.6480. ##0$$",
+                "role": "assistant"
+            },
+            {
+                "content": "How is the nvlink topology like?",
+                "role": "user"
+            },
+            {
+                "content": "NVLink topology refers to the arrangement of connections between GPUs using NVIDIA's NVLink technology. Correct NVLink topology for NVIDIA A100 cards involves connecting one GPU to another through a series of NVLink bridges ##0$$. Each of the three attached bridges spans two PCIe slots, and for optimal performance and balanced bridge topology, all three NVLink bridges should be used when connecting two adjacent A100 cards.\n\nHere's a summary of the correct and incorrect topologies:\n\n- **Correct**: Both GPUs are connected via all three NVLink bridges, ensuring full bandwidth and proper communication.\n- **Incorrect**: Not using all three bridges or having an uneven connection configuration would result in suboptimal performance.\n\nIt's also important to note that for multi-CPU systems, both A100 cards in a bridged pair should be within the same CPU domain, unless each CPU has a single A100 PCIe card, in which case they can be bridged together.",
+                "role": "assistant"
+            }
+        ],
+        "user_id": "user name",
+        "reference": [
+            {
+                "chunks": [
+                    {
+                        "chunk_id": "d0bc7892c3ec4aeac071544fd56730a8",
+                        "content_ltks": "tabl 1:openagi task-solv perform under differ set for three closed-sourc llm . boldfac denot the highest score under each learn schema . metric gpt-3.5-turbo claude-2 gpt-4 zero few zero few zero few clip score 0.0 0.0 0.0 0.2543 0.0 0.3055 bert score 0.1914 0.3820 0.2111 0.5038 0.2076 0.6307 vit score 0.2437 0.7497 0.4082 0.5416 0.5058 0.6480 overal 0.1450 0.3772 0.2064 0.4332 0.2378 0.5281",
+                        "content_with_weight": "<table><caption>Table 1: OpenAGI task-solving performances under different settings for three closed-source LLMs. Boldface denotes the highest score under each learning schema.</caption>\n<tr><th  rowspan=2 >Metrics</th><th  >GPT-3.5-turbo</th><th></th><th  >Claude-2</th><th  >GPT-4</th></tr>\n<tr><th  >Zero</th><th  >Few</th><th  >Zero Few</th><th  >Zero Few</th></tr>\n<tr><td  >CLIP Score</td><td  >0.0</td><td  >0.0</td><td  >0.0 0.2543</td><td  >0.0 0.3055</td></tr>\n<tr><td  >BERT Score</td><td  >0.1914</td><td  >0.3820</td><td  >0.2111 0.5038</td><td  >0.2076 0.6307</td></tr>\n<tr><td  >ViT Score</td><td  >0.2437</td><td  >0.7497</td><td  >0.4082 0.5416</td><td  >0.5058 0.6480</td></tr>\n<tr><td  >Overall</td><td  >0.1450</td><td  >0.3772</td><td  >0.2064 0.4332</td><td  >0.2378 0.5281</td></tr>\n</table>",
+                        "doc_id": "c790da40ea8911ee928e0242ac180005",
+                        "docnm_kwd": "OpenAGI When LLM Meets Domain Experts.pdf",
+                        "img_id": "afab9fdad6e511eebdb20242ac180006-d0bc7892c3ec4aeac071544fd56730a8",
+                        "important_kwd": [],
+                        "kb_id": "afab9fdad6e511eebdb20242ac180006",
+                        "positions": [
+                            [
+                                9.0,
+                                159.9383341471354,
+                                472.1773274739583,
+                                223.58013916015625,
+                                307.86692301432294
+                            ]
+                        ],
+                        "similarity": 0.7310340654129031,
+                        "term_similarity": 0.7671974387781668,
+                        "vector_similarity": 0.40556370512552886
+                    },
+                    {
+                        "chunk_id": "7e2345d440383b756670e1b0f43a7007",
+                        "content_ltks": "5.5 experiment analysi the main experiment result are tabul in tab . 1 and 2 , showcas the result for closed-sourc and open-sourc llm , respect . the overal perform is calcul a the averag of cllp 8 bert and vit score . here , onli the task descript of the benchmark task are fed into llm(addit inform , such a the input prompt and llm\u2019output , is provid in fig . a.4 and a.5 in supplementari). broadli speak , closed-sourc llm demonstr superior perform on openagi task , with gpt-4 lead the pack under both zero-and few-shot scenario . in the open-sourc categori , llama-2-13b take the lead , consist post top result across variou learn schema--the perform possibl influenc by it larger model size . notabl , open-sourc llm significantli benefit from the tune method , particularli fine-tun and\u2019rltf . these method mark notic enhanc for flan-t5-larg , vicuna-7b , and llama-2-13b when compar with zero-shot and few-shot learn schema . in fact , each of these open-sourc model hit it pinnacl under the rltf approach . conclus , with rltf tune , the perform of llama-2-13b approach that of gpt-3.5 , illustr it potenti .",
+                        "content_with_weight": "5.5 Experimental Analysis\nThe main experimental results are tabulated in Tab. 1 and 2, showcasing the results for closed-source and open-source LLMs, respectively. The overall performance is calculated as the average of CLlP\n8\nBERT and ViT scores. Here, only the task descriptions of the benchmark tasks are fed into LLMs (additional information, such as the input prompt and LLMs\u2019 outputs, is provided in Fig. A.4 and A.5 in supplementary). Broadly speaking, closed-source LLMs demonstrate superior performance on OpenAGI tasks, with GPT-4 leading the pack under both zero- and few-shot scenarios. In the open-source category, LLaMA-2-13B takes the lead, consistently posting top results across various learning schema--the performance possibly influenced by its larger model size. Notably, open-source LLMs significantly benefit from the tuning methods, particularly Fine-tuning and\u2019 RLTF. These methods mark noticeable enhancements for Flan-T5-Large, Vicuna-7B, and LLaMA-2-13B when compared with zero-shot and few-shot learning schema. In fact, each of these open-source models hits its pinnacle under the RLTF approach. Conclusively, with RLTF tuning, the performance of LLaMA-2-13B approaches that of GPT-3.5, illustrating its potential.",
+                        "doc_id": "c790da40ea8911ee928e0242ac180005",
+                        "docnm_kwd": "OpenAGI When LLM Meets Domain Experts.pdf",
+                        "img_id": "afab9fdad6e511eebdb20242ac180006-7e2345d440383b756670e1b0f43a7007",
+                        "important_kwd": [],
+                        "kb_id": "afab9fdad6e511eebdb20242ac180006",
+                        "positions": [
+                            [
+                                8.0,
+                                107.3,
+                                508.90000000000003,
+                                686.3,
+                                697.0
+                            ],
+                        ],
+                        "similarity": 0.6691508616357027,
+                        "term_similarity": 0.6999011754270821,
+                        "vector_similarity": 0.39239803751328806
+                    },
+                ],
+                "doc_aggs": [
+                    {
+                        "count": 8,
+                        "doc_id": "c790da40ea8911ee928e0242ac180005",
+                        "doc_name": "OpenAGI When LLM Meets Domain Experts.pdf"
+                    }
+                ],
+                "total": 8
+            },
+            {
+                "chunks": [
+                    {
+                        "chunk_id": "8c11a1edddb21ad2ae0c43b4a5dcfa62",
+                        "content_ltks": "nvlink bridg support nvidia\u00aenvlink\u00aei a high-spe point-to-point peer transfer connect , where one gpu can transfer data to and receiv data from one other gpu . the nvidia a100 card support nvlink bridg connect with a singl adjac a100 card . each of the three attach bridg span two pcie slot . to function correctli a well a to provid peak bridg bandwidth , bridg connect with an adjac a100 card must incorpor all three nvlink bridg . wherev an adjac pair of a100 card exist in the server , for best bridg perform and balanc bridg topolog , the a100 pair should be bridg . figur 4 illustr correct and incorrect a100 nvlink connect topolog . nvlink topolog\u2013top view figur 4. correct incorrect correct incorrect for system that featur multipl cpu , both a100 card of a bridg card pair should be within the same cpu domain\u2014that is , under the same cpu\u2019s topolog . ensur thi benefit workload applic perform . the onli except is for dual cpu system wherein each cpu ha a singl a100 pcie card under it;in that case , the two a100 pcie card in the system may be bridg togeth . a100 nvlink speed and bandwidth are given in the follow tabl . tabl 5. a100 nvlink speed and bandwidth paramet valu total nvlink bridg support by nvidia a100 3 total nvlink rx and tx lane support 96 data rate per nvidia a100 nvlink lane(each direct)50 gbp total maximum nvlink bandwidth 600 gbyte per second pb-10137-001_v03|8 nvidia a100 40gb pcie gpu acceler",
+                        "content_with_weight": "NVLink Bridge Support\nNVIDIA\u00aeNVLink\u00aeis a high-speed point-to-point peer transfer connection, where one GPU can transfer data to and receive data from one other GPU. The NVIDIA A100 card supports NVLink bridge connection with a single adjacent A100 card.\nEach of the three attached bridges spans two PCIe slots. To function correctly as well as to provide peak bridge bandwidth, bridge connection with an adjacent A100 card must incorporate all three NVLink bridges. Wherever an adjacent pair of A100 cards exists in the server, for best bridging performance and balanced bridge topology, the A100 pair should be bridged. Figure 4 illustrates correct and incorrect A100 NVLink connection topologies.\nNVLink Topology \u2013Top Views \nFigure 4. \nCORRECT \nINCORRECT \nCORRECT \nINCORRECT \nFor systems that feature multiple CPUs, both A100 cards of a bridged card pair should be within the same CPU domain\u2014that is, under the same CPU\u2019s topology. Ensuring this benefits workload application performance. The only exception is for dual CPU systems wherein each CPU has a single A100 PCIe card under it; in that case, the two A100 PCIe cards in the system may be bridged together.\nA100 NVLink speed and bandwidth are given in the following table.\n<table><caption>Table 5. A100 NVLink Speed and Bandwidth </caption>\n<tr><th  >Parameter </th><th  >Value </th></tr>\n<tr><td  >Total NVLink bridges supported by NVIDIA A100 </td><td  >3 </td></tr>\n<tr><td  >Total NVLink Rx and Tx lanes supported </td><td  >96 </td></tr>\n<tr><td  >Data rate per NVIDIA A100 NVLink lane (each direction)</td><td  >50 Gbps </td></tr>\n<tr><td  >Total maximum NVLink bandwidth</td><td  >600 Gbytes per second </td></tr>\n</table>\nPB-10137-001_v03 |8\nNVIDIA A100 40GB PCIe GPU Accelerator",
+                        "doc_id": "806d1ed0ea9311ee860a0242ac180005",
+                        "docnm_kwd": "A100-PCIE-Prduct-Brief.pdf",
+                        "img_id": "afab9fdad6e511eebdb20242ac180006-8c11a1edddb21ad2ae0c43b4a5dcfa62",
+                        "important_kwd": [],
+                        "kb_id": "afab9fdad6e511eebdb20242ac180006",
+                        "positions": [
+                            [
+                                12.0,
+                                84.0,
+                                541.3,
+                                76.7,
+                                96.7
+                            ],
+                        ],
+                        "similarity": 0.3200748779905588,
+                        "term_similarity": 0.3082244010114718,
+                        "vector_similarity": 0.42672917080234146
+                    },
+                ],
+                "doc_aggs": [
+                    {
+                        "count": 1,
+                        "doc_id": "806d1ed0ea9311ee860a0242ac180005",
+                        "doc_name": "A100-PCIE-Prduct-Brief.pdf"
+                    }
+                ],
+                "total": 3
+            }
+        ],
+        "update_date": "Tue, 02 Apr 2024 09:07:49 GMT",
+        "update_time": 1712020069421
+    },
+    "retcode": 0,
+    "retmsg": "success"
+}
+```
+
+- **message**: All the chat history in it.
+    - role: user or assistant
+    - content: the text content of user or assistant. The citations are in format like: ##0$$. The number in the middle indicate which part in data.reference.chunks it refers to.
+    
+- **user_id**: This is set by the caller.
+- **reference**: Every item in it refer to the corresponding message in data.message whose role is assistant. 
+    - chunks
+        - content_with_weight: The content of chunk.
+        - docnm_kwd: the document name.
+        - img_id: the image id of the chunk. It is an optional field only for PDF/pptx/picture. And accessed by 'GET' /document/get/\<id\>.
+        - positions: [page_number, [upleft corner(x, y)], [right bottom(x, y)]], the chunk position, only for PDF.
+        - similarity: the hybrid similarity.
+        - term_similarity: keyword simimlarity
+        - vector_similarity: embedding similarity
+    - doc_aggs:
+        - doc_id: the document can be accessed by 'GET' /document/get/\<id\>
+        - doc_name: the file name
+        - count: the chunk number hit in this document.
+    
+## Chat
+
+This will be called to get the answer to users' questions.
+
+### Path: /api/completion
+### Method: POST
+### Parameter:
+
+| name | type | optional | description|
+|------|-------|----|----|
+| conversation_id| string | No | This is from calling /new_conversation.|
+| messages| json | No | The latest question, such as `[{"role": "user", "content": "How are you doing!"}]`|
+| quote | bool | Yes | Default: true |
+| stream | bool | Yes | Default: true |
+| doc_ids | string | Yes | Document IDs which is delimited by comma, like `c790da40ea8911ee928e0242ac180005,c790da40ea8911ee928e0242ac180005`. The retrieved content is limited in these documents. |
+
+### Response 
+```json
+{
+    "data": {
+      "answer": "The ViT Score for GPT-4 in the zero-shot scenario is 0.5058, and in the few-shot scenario, it is 0.6480. ##0$$",
+      "reference": {
+        "chunks": [
+          {
+            "chunk_id": "d0bc7892c3ec4aeac071544fd56730a8",
+            "content_ltks": "tabl 1:openagi task-solv perform under differ set for three closed-sourc llm . boldfac denot the highest score under each learn schema . metric gpt-3.5-turbo claude-2 gpt-4 zero few zero few zero few clip score 0.0 0.0 0.0 0.2543 0.0 0.3055 bert score 0.1914 0.3820 0.2111 0.5038 0.2076 0.6307 vit score 0.2437 0.7497 0.4082 0.5416 0.5058 0.6480 overal 0.1450 0.3772 0.2064 0.4332 0.2378 0.5281",
+            "content_with_weight": "<table><caption>Table 1: OpenAGI task-solving performances under different settings for three closed-source LLMs. Boldface denotes the highest score under each learning schema.</caption>\n<tr><th  rowspan=2 >Metrics</th><th  >GPT-3.5-turbo</th><th></th><th  >Claude-2</th><th  >GPT-4</th></tr>\n<tr><th  >Zero</th><th  >Few</th><th  >Zero Few</th><th  >Zero Few</th></tr>\n<tr><td  >CLIP Score</td><td  >0.0</td><td  >0.0</td><td  >0.0 0.2543</td><td  >0.0 0.3055</td></tr>\n<tr><td  >BERT Score</td><td  >0.1914</td><td  >0.3820</td><td  >0.2111 0.5038</td><td  >0.2076 0.6307</td></tr>\n<tr><td  >ViT Score</td><td  >0.2437</td><td  >0.7497</td><td  >0.4082 0.5416</td><td  >0.5058 0.6480</td></tr>\n<tr><td  >Overall</td><td  >0.1450</td><td  >0.3772</td><td  >0.2064 0.4332</td><td  >0.2378 0.5281</td></tr>\n</table>",
+            "doc_id": "c790da40ea8911ee928e0242ac180005",
+            "docnm_kwd": "OpenAGI When LLM Meets Domain Experts.pdf",
+            "img_id": "afab9fdad6e511eebdb20242ac180006-d0bc7892c3ec4aeac071544fd56730a8",
+            "important_kwd": [],
+            "kb_id": "afab9fdad6e511eebdb20242ac180006",
+            "positions": [
+              [
+                9.0,
+                159.9383341471354,
+                472.1773274739583,
+                223.58013916015625,
+                307.86692301432294
+              ]
+            ],
+            "similarity": 0.7310340654129031,
+            "term_similarity": 0.7671974387781668,
+            "vector_similarity": 0.40556370512552886
+          },
+          {
+            "chunk_id": "7e2345d440383b756670e1b0f43a7007",
+            "content_ltks": "5.5 experiment analysi the main experiment result are tabul in tab . 1 and 2 , showcas the result for closed-sourc and open-sourc llm , respect . the overal perform is calcul a the averag of cllp 8 bert and vit score . here , onli the task descript of the benchmark task are fed into llm(addit inform , such a the input prompt and llm\u2019output , is provid in fig . a.4 and a.5 in supplementari). broadli speak , closed-sourc llm demonstr superior perform on openagi task , with gpt-4 lead the pack under both zero-and few-shot scenario . in the open-sourc categori , llama-2-13b take the lead , consist post top result across variou learn schema--the perform possibl influenc by it larger model size . notabl , open-sourc llm significantli benefit from the tune method , particularli fine-tun and\u2019rltf . these method mark notic enhanc for flan-t5-larg , vicuna-7b , and llama-2-13b when compar with zero-shot and few-shot learn schema . in fact , each of these open-sourc model hit it pinnacl under the rltf approach . conclus , with rltf tune , the perform of llama-2-13b approach that of gpt-3.5 , illustr it potenti .",
+            "content_with_weight": "5.5 Experimental Analysis\nThe main experimental results are tabulated in Tab. 1 and 2, showcasing the results for closed-source and open-source LLMs, respectively. The overall performance is calculated as the average of CLlP\n8\nBERT and ViT scores. Here, only the task descriptions of the benchmark tasks are fed into LLMs (additional information, such as the input prompt and LLMs\u2019 outputs, is provided in Fig. A.4 and A.5 in supplementary). Broadly speaking, closed-source LLMs demonstrate superior performance on OpenAGI tasks, with GPT-4 leading the pack under both zero- and few-shot scenarios. In the open-source category, LLaMA-2-13B takes the lead, consistently posting top results across various learning schema--the performance possibly influenced by its larger model size. Notably, open-source LLMs significantly benefit from the tuning methods, particularly Fine-tuning and\u2019 RLTF. These methods mark noticeable enhancements for Flan-T5-Large, Vicuna-7B, and LLaMA-2-13B when compared with zero-shot and few-shot learning schema. In fact, each of these open-source models hits its pinnacle under the RLTF approach. Conclusively, with RLTF tuning, the performance of LLaMA-2-13B approaches that of GPT-3.5, illustrating its potential.",
+            "doc_id": "c790da40ea8911ee928e0242ac180005",
+            "docnm_kwd": "OpenAGI When LLM Meets Domain Experts.pdf",
+            "img_id": "afab9fdad6e511eebdb20242ac180006-7e2345d440383b756670e1b0f43a7007",
+            "important_kwd": [],
+            "kb_id": "afab9fdad6e511eebdb20242ac180006",
+            "positions": [
+              [
+                8.0,
+                107.3,
+                508.90000000000003,
+                686.3,
+                697.0
+              ]
+            ],
+            "similarity": 0.6691508616357027,
+            "term_similarity": 0.6999011754270821,
+            "vector_similarity": 0.39239803751328806
+          }
+        ],
+        "doc_aggs": {
+          "OpenAGI When LLM Meets Domain Experts.pdf": 4
+        },
+        "total": 8
+      }
+    },
+    "retcode": 0,
+    "retmsg": "success"
+}
+```
+
+- **answer**: The replay of the chat bot.
+- **reference**: 
+    - chunks: Every item in it refer to the corresponding message in answer. 
+        - content_with_weight: The content of chunk.
+        - docnm_kwd: the document name.
+        - img_id: the image id of the chunk. It is an optional field only for PDF/pptx/picture. And accessed by 'GET' /document/get/\<id\>.
+        - positions: [page_number, [upleft corner(x, y)], [right bottom(x, y)]], the chunk position, only for PDF.
+        - similarity: the hybrid similarity.
+        - term_similarity: keyword simimlarity
+        - vector_similarity: embedding similarity
+    - doc_aggs:
+        - doc_id: the document can be accessed by 'GET' /document/get/\<id\>
+        - doc_name: the file name
+        - count: the chunk number hit in this document.
+    
+## Get document content or image
+
+This is usually used when display content of citation.
+### Path: /api/document/get/\<id\>
+### Method: GET
+
+## Upload file
+
+This is usually used when upload a file to.
+### Path: /api/document/upload/
+### Method: POST
+
+### Parameter:
+
+| name      | type   | optional | description                                             |
+|-----------|--------|----------|---------------------------------------------------------|
+| file      | file   | No       | Upload file.                                            |
+| kb_name   | string | No       | Choose the upload knowledge base name.                  |
+| parser_id | string | Yes      | Choose the parsing method.                              |
+| run       | string | Yes      | Parsing will start automatically when the value is "1". |
+
+### Response 
+```json
+{
+    "data": {
+        "chunk_num": 0,
+        "create_date": "Thu, 25 Apr 2024 14:30:06 GMT",
+        "create_time": 1714026606921,
+        "created_by": "553ec818fd5711ee8ea63043d7ed348e",
+        "id": "41e9324602cd11ef9f5f3043d7ed348e",
+        "kb_id": "06802686c0a311ee85d6246e9694c130",
+        "location": "readme.txt",
+        "name": "readme.txt",
+        "parser_config": {
+            "field_map": {
+            },
+            "pages": [
+                [
+                    0,
+                    1000000
+                ]
+            ]
+        },
+        "parser_id": "general",
+        "process_begin_at": null,
+        "process_duation": 0.0,
+        "progress": 0.0,
+        "progress_msg": "",
+        "run": "0",
+        "size": 929,
+        "source_type": "local",
+        "status": "1",
+        "thumbnail": null,
+        "token_num": 0,
+        "type": "doc",
+        "update_date": "Thu, 25 Apr 2024 14:30:06 GMT",
+        "update_time": 1714026606921
+    },
+    "retcode": 0,
+    "retmsg": "success"
+}
+
+```
+
+## Get document chunks
+
+Get the chunks of the document based on doc_name or doc_id.
+### Path: /api/list_chunks/
+### Method: POST
+
+### Parameter:
+
+| Name     | Type   | Optional | Description                     |
+|----------|--------|----------|---------------------------------|
+| `doc_name` | string | Yes      | The name of the document in the knowledge base. It must not be empty if `doc_id` is not set.|
+| `doc_id`   | string | Yes      | The ID of the document in the knowledge base. It must not be empty if `doc_name` is not set.|
+
+
+### Response 
+```json
+{
+    "data": [
+        {
+            "content": "Figure 14: Per-request neural-net processingof RL-Cache.\n103\n(sn)\nCPU\n 102\nGPU\n8101\n100\n8\n16 64 256 1K\n4K",
+            "doc_name": "RL-Cache.pdf",
+            "img_id": "0335167613f011ef91240242ac120006-b46c3524952f82dbe061ce9b123f2211"
+        },
+        {
+            "content": "4.3 ProcessingOverheadof RL-CacheACKNOWLEDGMENTSThis section evaluates how e￿ectively our RL-Cache implemen-tation leverages modern multi-core CPUs and GPUs to keep the per-request neural-net processing overhead low. Figure 14 depictsThis researchwas supported inpart by the Regional Government of Madrid (grant P2018/TCS-4499, EdgeData-CM)andU.S. National Science Foundation (grants CNS-1763617 andCNS-1717179).REFERENCES",
+            "doc_name": "RL-Cache.pdf",
+            "img_id": "0335167613f011ef91240242ac120006-d4c12c43938eb55d2d8278eea0d7e6d7"
+        }
+    ],
+    "retcode": 0,
+    "retmsg": "success"
+}
+
+```

docs/{faq.md → references/faq.md}

@@ -1,4 +1,9 @@
-# Frequently Asked Questions
+---
+sidebar_position: 3
+slug: /faq
+---
+
+# Frequently asked questions
 
 ## General
 
@@ -31,7 +36,7 @@ Currently, we only support x86 CPU and Nvidia GPU.
 
 ### 2. Do you offer an API for integration with third-party applications?
 
-The corresponding APIs are now available. See the [Conversation API](./conversation_api.md) for more information. 
+The corresponding APIs are now available. See the [RAGFlow API Reference](./api.md) for more information. 
 
 ### 3. Do you support stream output?
 
@@ -186,14 +191,12 @@ Parsing requests have to wait in queue due to limited server resources. We are c
 
 If your RAGFlow is deployed *locally*, try the following: 
 
-1. Click the red cross icon next to **Parsing Status** and refresh the file parsing process. 
-2. If the issue still persists, try the following: 
-   - check the log of your RAGFlow server to see if it is running properly:
-   ```bash
-   docker logs -f ragflow-server
-   ```
-   - Check if the **task_executor.py** process exists.
-   - Check if your RAGFlow server can access hf-mirror.com or huggingface.com.
+1. Check the log of your RAGFlow server to see if it is running properly:
+```bash
+docker logs -f ragflow-server
+```
+2. Check if the **task_executor.py** process exists.
+3. Check if your RAGFlow server can access hf-mirror.com or huggingface.com.
 
 #### 4.5 Why does my pdf parsing stall near completion, while the log does not show any error?
 
@@ -356,7 +359,7 @@ You limit what the system responds to what you specify in **Empty response** if
 
 ### 4. How to run RAGFlow with a locally deployed LLM?
 
-You can use Ollama to deploy local LLM. See [here](https://github.com/infiniflow/ragflow/blob/main/docs/ollama.md) for more information. 
+You can use Ollama to deploy local LLM. See [here](https://github.com/infiniflow/ragflow/blob/main/docs/guides/deploy_local_llm.md) for more information. 
 
 ### 5. How to link up ragflow and ollama servers?
 

web/src/pages/chat/chat-overview-modal/index.tsx

@@ -88,7 +88,7 @@ const ChatOverviewModal = ({
               <Button onClick={showApiKeyModal}>{t('apiKey')}</Button>
               <a
                 href={
-                  'https://github.com/infiniflow/ragflow/blob/main/docs/conversation_api.md'
+                  'https://github.com/infiniflow/ragflow/blob/main/docs/references/api.md'
                 }
                 target="_blank"
                 rel="noreferrer"