File size: 5,088 Bytes
caffddc
60ed594
dc9e27a
 
 
caffddc
dc9e27a
caffddc
 
 
 
dc9e27a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
---
title: MindSearch
emoji: ๐Ÿ“Š
colorFrom: purple
colorTo: yellow
sdk: gradio
sdk_version: 5.7.1
app_file: app.py
pinned: false
---

<div id="top"></div>

<div align="center">

<img src="assets/logo.svg" style="width: 50%; height: auto;">

[๐Ÿ“ƒ Paper](https://arxiv.org/abs/2407.20183) | [๐Ÿ’ป Demo](https://internlm-chat.intern-ai.org.cn/)

English | [็ฎ€ไฝ“ไธญๆ–‡](README_zh-CN.md)

<https://github.com/user-attachments/assets/44ffe4b9-be26-4b93-a77b-02fed16e33fe>

</div>
</p>

## โœจ MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

## ๐Ÿ“… Changelog

- 2024/11/05: ๐Ÿฅณ MindSearch is now deployed on Puyu! ๐Ÿ‘‰ [Try it](https://internlm-chat.intern-ai.org.cn/) ๐Ÿ‘ˆ
  -  Refactored the agent module based on [Lagent v0.5](https://github.com/InternLM/lagent) for better performance in concurrency.
  -  Improved the UI to embody the simultaneous multi-query search.


## โšฝ๏ธ Build Your Own MindSearch

### Step1: Dependencies Installation

```bash
git clone https://github.com/InternLM/MindSearch
cd MindSearch
pip install -r requirements.txt
```

### Step2: Setup Environment Variables

Before setting up the API, you need to configure environment variables. Rename the `.env.example` file to `.env` and fill in the required values.

```bash
mv .env.example .env
# Open .env and add your keys and model configurations
```

### Step3: Setup MindSearch API

Setup FastAPI Server.

```bash
python -m mindsearch.app --lang en --model_format internlm_silicon --search_engine DuckDuckGoSearch --asy 
```

- `--lang`: language of the model, `en` for English and `cn` for Chinese.
- `--model_format`: format of the model.
  - `internlm_server` for InternLM2.5-7b-chat with local server. (InternLM2.5-7b-chat has been better optimized for Chinese.)
  - `gpt4` for GPT4.
    if you want to use other models, please modify [models](./mindsearch/agent/models.py)
- `--search_engine`: Search engine.
  - `DuckDuckGoSearch` for search engine for DuckDuckGo.
  - `BingSearch` for Bing search engine.
  - `BraveSearch` for Brave search web api engine.
  - `GoogleSearch` for Google Serper web search api engine.
  - `TencentSearch` for Tencent search api engine.
  
  Please set your Web Search engine API key as the `WEB_SEARCH_API_KEY` environment variable unless you are using `DuckDuckGo`, or `TencentSearch` that requires secret id as `TENCENT_SEARCH_SECRET_ID` and secret key as `TENCENT_SEARCH_SECRET_KEY`.
- `--asy`: deploy asynchronous agents.

### Step4: Setup MindSearch Frontend

Providing following frontend interfaces,

- React

First configurate the backend URL for Vite proxy.

```bash
HOST="127.0.0.1"  # modify as you need
PORT=8002
sed -i -r "s/target:\s*\"\"/target: \"${HOST}:${PORT}\"/" frontend/React/vite.config.ts
```

```bash
# Install Node.js and npm
# for Ubuntu
sudo apt install nodejs npm

# for windows
# download from https://nodejs.org/zh-cn/download/prebuilt-installer

# Install dependencies

cd frontend/React
npm install
npm start
```

Details can be found in [React](./frontend/React/README.md)

- Gradio

```bash
python frontend/mindsearch_gradio.py
```

- Streamlit

```bash
streamlit run frontend/mindsearch_streamlit.py
```

## ๐ŸŒ Change Web Search API

To use a different type of web search API, modify the `searcher_type` attribute in the `searcher_cfg` located in `mindsearch/agent/__init__.py`. Currently supported web search APIs include:

- `GoogleSearch`
- `DuckDuckGoSearch`
- `BraveSearch`
- `BingSearch`
- `TencentSearch`

For example, to change to the Brave Search API, you would configure it as follows:

```python
BingBrowser(
    searcher_type='BraveSearch',
    topk=2,
    api_key=os.environ.get('BRAVE_API_KEY', 'YOUR BRAVE API')
)
```

## ๐Ÿž Using the Backend Without Frontend

For users who prefer to interact with the backend directly, use the `backend_example.py` script. This script demonstrates how to send a query to the backend and process the response.

```bash
python backend_example.py
```

Make sure you have set up the environment variables and the backend is running before executing the script.

## ๐Ÿž Debug Locally

```bash
python -m mindsearch.terminal
```

## ๐Ÿ“ License

This project is released under the [Apache 2.0 license](LICENSE).

## Citation

If you find this project useful in your research, please consider cite:

```
@article{chen2024mindsearch,
  title={MindSearch: Mimicking Human Minds Elicits Deep AI Searcher},
  author={Chen, Zehui and Liu, Kuikun and Wang, Qiuchen and Liu, Jiangning and Zhang, Wenwei and Chen, Kai and Zhao, Feng},
  journal={arXiv preprint arXiv:2407.20183},
  year={2024}
}
```

## Our Projects

Explore our additional research on large language models, focusing on LLM agents.

- [Lagent](https://github.com/InternLM/lagent): A lightweight framework for building LLM-based agents
- [AgentFLAN](https://github.com/InternLM/Agent-FLAN): An innovative approach for constructing and training with high-quality agent datasets (ACL 2024 Findings)
- [T-Eval](https://github.com/open-compass/T-Eval): A Fine-grained tool utilization evaluation benchmark (ACL 2024)