|
--- |
|
title: SearXNG Web Search |
|
emoji: 🌍 |
|
colorFrom: yellow |
|
colorTo: purple |
|
sdk: gradio |
|
sdk_version: 4.44.1 |
|
app_file: app.py |
|
pinned: true |
|
license: apache-2.0 |
|
short_description: Web Search AI |
|
--- |
|
|
|
An example chatbot using [Gradio](https://gradio.app), [`huggingface_hub`](https://huggingface.co/docs/huggingface_hub/v0.22.2/en/index), and the [Hugging Face Inference API](https://huggingface.co/docs/api-inference/index). |
|
|
|
# Web Scraper for Financial News with Sentinel AI |
|
|
|
## Table of Contents |
|
1. [Overview](#overview) |
|
2. [Core Components](#core-components) |
|
- [Search Engine Integration](#search-engine-integration) |
|
- [AI Models Integration](#ai-models-integration) |
|
- [Content Processing](#content-processing) |
|
3. [Key Features](#key-features) |
|
- [Intelligent Query Processing](#intelligent-query-processing) |
|
- [Content Analysis](#content-analysis) |
|
- [Search Optimization](#search-optimization) |
|
4. [Architecture](#architecture) |
|
- [User Interface (UI)](#user-interface-ui) |
|
- [Query Processing](#query-processing) |
|
- [Search Engine](#search-engine) |
|
- [Content Analysis](#content-analysis) |
|
- [Ranking System](#ranking-system) |
|
- [Response Generation](#response-generation) |
|
- [Core Classes](#core-classes) |
|
5. [Main Functions](#main-functions) |
|
6. [API Integration](#api-integration) |
|
8. [Advanced Parameters](#advanced-parameters) |
|
|
|
## 1. Overview |
|
This application is a sophisticated web scraper and AI-powered chat interface specifically designed for financial news analysis. It combines web scraping capabilities with multiple Language Learning Models (LLMs) to provide intelligent, context-aware responses to user queries about financial information. |
|
|
|
## 2. Core Components |
|
|
|
### Search Engine Integration |
|
- Uses SearXNG as the primary search meta-engine |
|
- Supports multiple search engines (Google, Bing, DuckDuckGo, etc.) |
|
- Implements custom retry mechanisms and timeout handling |
|
|
|
### AI Models Integration |
|
- Supports multiple LLM providers: Hugging Face (Mistral-Small-Instruct), Groq (Llama-3.1-70b), Mistral AI (Open-Mistral-Nemo) |
|
- Implements semantic similarity using Sentence-Transformer |
|
|
|
### Content Processing |
|
- PDF processing with PyPDF2 |
|
- Web content scraping with Newspaper3k |
|
- BM25 ranking algorithm implementation |
|
- Document deduplication and relevance assessment |
|
|
|
## 3. Key Features |
|
|
|
### Intelligent Query Processing |
|
- Query type determination (knowledge base vs. web search) |
|
- Query rephrasing for optimal search results |
|
- Entity recognition |
|
- Time-aware query modification |
|
|
|
### Content Analysis |
|
- Relevance assessment |
|
- Content summarization |
|
- Semantic similarity comparison |
|
- Document deduplication |
|
- Priority-based content ranking |
|
|
|
### Search Optimization |
|
- Custom retry mechanism |
|
- Rate limiting |
|
- Error handling |
|
- Content filtering and validation |
|
|
|
## 4. Architecture |
|
|
|
### User Interface (UI) |
|
- You start by interacting with a Gradio Chat Interface. |
|
|
|
### Query Processing |
|
- Your query is sent to the Query Analysis (QA) section. |
|
- The system then determines the type of query (DT). |
|
- If it's a type that can use a Knowledge Base, it generates an AI response (KB). |
|
- If it requires web searching, it rephrases the query (QR) for web search. |
|
- The system extracts the entity domain (ED) from the rephrased query. |
|
|
|
### Search Engine |
|
- The extracted entity domain is sent to the SearXNG Search Engine (SE). |
|
- The search engine returns the search results (SR). |
|
|
|
### Content Analysis |
|
- The search results are processed by web scraping (WS). |
|
- If the content is in PDF format, it is scraped using PDF Scraping (PDF). |
|
- If in HTML format, it's scraped using Newspaper3k Scraping (NEWS). |
|
- Relevant content is summarized (DS) and checked for uniqueness (UC). |
|
|
|
### Ranking System |
|
- Content is ranked (DR) based on: |
|
- **BM25 Scoring (BM):** A scoring method to rank documents. |
|
- **Semantic Similarity (SS):** How similar the content is to the query. |
|
- The scores are combined (CS) to produce a final ranking (FR). |
|
|
|
### Response Generation |
|
- The final ranking is summarized again (FS) to create a final summary. |
|
- The AI-generated response (KB) and final summary (FS) are combined to form the final response. |
|
|
|
### Completion |
|
- The final response is sent back to the Gradio Chat Interface (UI) for you to see. |
|
|
|
### Core Classes |
|
- **BM25:** Custom implementation for document ranking |
|
- **Search and Scrape Pipeline:** Handles query processing, web search, content scraping, document analysis, and content summarization. |
|
|
|
## 5. Main Functions |
|
- **`determine_query_type(query, chat_history, llm_client)`**: Determines whether to use knowledge base or web search based on context. |
|
- **`search_and_scrape(query, chat_history, ...)`**: Main function for web search and content aggregation. |
|
- **`rerank_documents_with_priority(query, documents, entity_domain, ...)`**: Hybrid ranking using BM25 and semantic similarity. |
|
- **`llm_summarize(json_input, model, temperature)`**: Generates summaries using the specified LLM and handles citation and formatting. |
|
|
|
## 6. API Integration |
|
- **Required API Keys**: Hugging Face, Groq, Mistral, SearXNG |
|
- **Environment Variables Setup**: Use dotenv to load environment variables |
|
|
|
## 8. Advanced Parameters |
|
|
|
| **Parameter** | **Description** | **Range/Options** | **Default** | **Usage** | |
|
|--------------------------|---------------------------------------------------------------|--------------------------------------------|---------------|---------------------------------------------------------------| |
|
| **Number of Results** | Number of search results retrieved. | 5 to 20 | 5 | Controls number of links/articles fetched from web searches. | |
|
| **Maximum Characters** | Limits characters per document processed. | 500 to 10,000 | 3000 | Truncates long documents, focusing on relevant information. | |
|
| **Time Range** | Specifies the time period for search results. | day, week, month, year | month | Filters results based on recent or historical data. | |
|
| **Language Selection** | Filters search results by language. | `en`, `fr`, `es`, etc. | `en` | Retrieves content in a specified language. | |
|
| **LLM Temperature** | Controls randomness in responses from LLM. | 0.0 to 1.0 | 0.2 | Low values for factual responses; higher for creative ones. | |
|
| **Search Engines** | Specifies search engines used for scraping. | Google, Bing, DuckDuckGo, etc. | All engines | Choose specific search engines for better or private results. | |
|
| **Safe Search Level** | Filters explicit/inappropriate content. | 0: No filter, 1: Moderate, 2: Strict | 2 (Strict) | Ensures family-friendly or professional content. | |
|
| **Model Selection** | Chooses the LLM for summaries or responses. | Mistral, GPT-4, Groq | Varies | Select models based on performance or speed. | |
|
| **PDF Processing Toggle** | Enables/disables PDF document processing. | `True` (process) or `False` (skip) | `False` | Processes PDFs, useful for reports but may slow down speed. | |
