Spaces:
Runtime error
Runtime error
File size: 3,568 Bytes
21e639d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 |
# **批踢踢爬蟲 ptt-crawler**
### 學術研究用途，請勿不當使用。
This project scrapes the post details from the website [PTT](https://term.ptt.cc/), and writes the scraped items to csv files.
| author | alias |title | date | ip | city | country | ups | downs | comments | url |
|----|----|----|----|----|----|----|----|----|----|----|
| jason789780 | majiLove | \[請益\] google問題的精確與方向 | 2022-09-06 10:39:42 | 223.137.68.113 | Yilan | Taiwan | 9 | 0 | 29 | https://www.ptt.cc/bbs/Soft_Job/M.1662431984.A.A3F.html |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
## **說明**
### 1. Installation
1. Python version
* `python >= 3.9`
2. Clone repository
```bash
git clone [email protected]:lopentu/nlp_web.git
```
3. Install Requirement
```bash
cd nlp_web/assignments/ptt-crawler && pip install -r requirements.txt
```
### 2. 使用方式
1. Commands
```
scrapy crawl ptt -a boards=BOARDS [-a scrap_all=BOOLEAN]
[-a index_from=NUMBER -a index_to=NUMBER]
[-a since=YEAR] [-a data_dir=PATH]
[-a ip_cache=BOOLEAN]
positional arguments:
-a boards=BOARDS ptt board name (e.g. Soft_Job)
-a index_from=NUMBER -a index_to=NUMBER html index number from a ptt board
-a scrap_all=BOOLEAN scrap all posts if true
-a since=YEAR scrap all posts from a given year
-a data_dir=PATH output file path (default: ./data)
-a ip_cache=BOOLEAN enable redis service to cache ip if true
```
> If you enable `ip_cache`, please make sure you have Redis on your local machine. Otherwise, you can use `docker-compose` to run the crawler.
* Crawl all the posts of a board:
```bash
scrapy crawl ptt -a boards=Soft_Job -a scrap_all=True
```
* Crawl all the posts of a board from a year in the past:
```bash
scrapy crawl ptt -a boards=Soft_Job -a since=2020
```
* Crawl the posts of a board based on html indexes:
```bash
scrapy crawl ptt -a boards=Soft_Job -a index_from=1722 -a index_to=1723
```
> Please make sure the number of `index_from` is greater than `index_to`.
* Crawl the posts of multiple boards. For example:
```bash
scrapy crawl ptt -a boards=Soft_Job,Gossiping -a index_from=1722 -a index_to=1723
```
>Note: the comma in the argument `boards` cannot have spaces. It cannot be `boards=Soft_Job, Baseball` or `boards=["Soft_Job", "Baseball"]`.
### 3. 使用 Docker
A Docker setup is provided for the crawler.
To run the crawler, go to the `docker-compose.yml` file to edit the command:
```yaml
version: "3"
services:
scraptt:
build: .
environment:
- PYTHON_ENV=production
depends_on:
- redis
links:
- redis
# define your crawler here!
command: bash -c "scrapy crawl ptt -a boards=Soft_Job,Gossiping -a index_from=1722 -a index_to=1723 -a ip_cache=True"
```
> Feel free to change the command as long as it follows the command format as stated above.
Now start the crawler:
```bash
docker-compose up
```
We suggest setting the argument `ip_cache` to `true` when you run the crawler via Docker. The reason is that the crawler will connect to Redis container, and there is no need for you to have Redis on your local machine. Most importantly, the performance of the crawler will be increased!
## Contact
If you have any suggestion or question, please do not hesitate to email us at [email protected] or [email protected]
|