Revise README to enhance clarity and detail about Inkling's purpose and functionality

README.md

@@ -16,54 +16,122 @@ models:
 - nomadicsynth/research-compass-arxiv-abstracts-embedding-model
 ---
 
-# Inkling: AI-assisted research discovery
+# Inkling: Bridging the Unconnected in Scientific Literature  
 
-![Inkling](https://huggingface.co/spaces/nomadicsynth/inkling/resolve/main/inkling-logo.png)
+![Inkling](https://huggingface.co/spaces/nomadicsynth/inkling/resolve/main/inkling-logo.png)  
 
-[**Inkling**](https://nomadicsynth-research-compass.hf.space) is an AI-assisted tool that helps you discover meaningful connections between research papers — the kind of links a domain expert might spot, if they had time to read everything.
+**Inkling** is an experimental bridge-finding engine for scientific literature, built to uncover *latent connections* between research papers—relationships that are obvious in hindsight but buried under the sheer volume of modern research. It’s inspired by the work of **Don R. Swanson**, the visionary who discovered the link between *fish oil* and *Raynaud’s syndrome* using nothing but manual literature analysis. Today, we call this approach **undiscovered public knowledge**—and Inkling is our attempt to automate it with modern NLP.  
 
-Rather than relying on superficial similarity or shared keywords, Inkling is trained to recognize **reasoning-based relationships** between papers. It evaluates conceptual, methodological, and application-level connections — even across disciplines — and surfaces links that may be overlooked due to the sheer scale of the research landscape.
+---
+
+## The Problem: Lost in the Literature  
+
+The scientific literature is growing exponentially, but human researchers can only read so much. As Sabine Hossenfelder explained in her 2024 YouTube video ["AIs Predict Research Results Without Doing Research"](https://www.youtube.com/watch?v=Qgrl3JSWWDE), even experts miss critical connections because no one has time to read everything. Swanson’s 1986 discovery of the fish oil–Raynaud’s link was a wake-up call: the knowledge existed in plain sight, but the papers were siloed. Inkling is our attempt to fix that.  
+
+---
+
+## The Vision: A Bridge-Finding Machine  
+
+Inkling isn’t just a search engine. It’s a **hypothesis generator**. It learns to recognize *intermediate concepts* that connect seemingly unrelated papers—like Swanson’s "blood viscosity" bridge. The model is built to:  
+
+- **Find indirect links** between papers that don’t cite each other.  
+- **Surface connections** that feel obvious once explained but are buried in the noise.  
+- **Scale** to the entire arXiv corpus and beyond.  
+
+---
+
+## How It Works  
+
+### Model Architecture  
+
+- **Base Model**: A `SentenceTransformer` using **Llama-7B** as its base (with frozen weights) and a dense embedding head.  
+- **Training**:  
+  - v1: Trained on a synthetic dataset of randomly paired papers, rated for conceptual overlap.  
+  - v2 (in progress): Focused on *bridge detection*, using prompts to explicitly identify intermediate concepts (e.g., "What connects these two papers?").  
+- **Embedding Strategy**:  
+  - Dense vector representations of abstracts.  
+  - FAISS for fast approximate nearest-neighbor search.  
+
+### Dataset Philosophy  
 
-This demo uses the first prototype of the model, trained on a dataset of **10,000+ rated abstract pairs**, built from a larger pool of arXiv triplets. The system will continue to improve with feedback and will be released alongside the dataset for public research.
+- v1: Random paper pairs rated for generic "relevance" (too broad, limited bridge detection).  
+- v2: Focus on **explicit bridge extraction** using LLM-generated triplets (e.g., "Paper A → Bridge Concept → Paper B").  
 
 ---
 
-## What it does
+## The Inspiration  
 
-- Accepts a research abstract, idea, or question
-- Searches for papers with **deep, contextual relevance**
-- Highlights key conceptual links and application overlaps
-- Offers reasoning-based analysis between selected papers
-- Gathers user feedback to improve the model over time
+This project was born from a **nerd-sniping moment** after watching Sabine Hossenfelder’s video on AI’s ability to predict neuroscience results without experiments. That led to three key influences:  
+
+### 1. **Swanson’s "Undiscovered Public Knowledge"**  
+
+Swanson’s 1986 paper showed that the fish oil–Raynaud’s link existed in the literature for decades—it just took a human to connect the dots. Inkling automates this process.  
+
+### 2. **Tshitoyan et al. (2019): Word Embeddings in Materials Science**  
+
+Their work demonstrated that unsupervised embeddings could predict future material discoveries from latent knowledge. Inkling applies this idea to *conceptual bridges* in all scientific fields.  
+
+### 3. **Luo et al. (2024): LLMs Beat Human Experts**  
+
+This study showed that a 7B LLM (like Mistral) could outperform neuroscientists in predicting experimental outcomes. Inkling leverages this power to find connections even domain experts might miss.  
 
 ---
 
-## Background and Motivation
+## What It Can Do (and What’s Next)  
 
-Scientific progress often depends on connecting ideas across papers, fields, and years of literature. But with the volume of research growing exponentially, it's increasingly difficult for any one person — or even a team — to stay on top of it all. As a result, valuable connections between papers often go unnoticed simply because the right expert never read both.
+### Current Capabilities  
 
-In 2024, Luo et al. published a landmark study in *Nature Human Behaviour* showing that **large language models (LLMs) can outperform human experts** in predicting the results of neuroscience experiments by integrating knowledge across the scientific literature. Their model, **BrainGPT**, demonstrated how tuning a general-purpose LLM (like Mistral-7B) on domain-specific data could synthesize insights that surpass human forecasting ability. Notably, the authors found that models as small as 7B parameters performed well — an insight that influenced the foundation for this project.
+- Embed arXiv abstracts into dense vectors.  
+- Search for papers with conceptual overlap (50% relevance in top-10/25 queries, per manual testing).  
+- Visualize results in a Gradio interface with FAISS-powered speed.  
 
-Inspired by this work — and a YouTube breakdown by physicist and science communicator **Sabine Hossenfelder**, titled *["AIs Predict Research Results Without Doing Research"](https://www.youtube.com/watch?v=Qgrl3JSWWDE)* — this project began as an attempt to explore similar methods of knowledge integration at the level of paper-pair relationships. Her clear explanation and commentary sparked the idea to apply this paradigm not just to forecasting outcomes, but to identifying latent connections between published studies.
+### Roadmap  
 
-Originally conceived as a perplexity-ranking experiment using LLMs directly (mirroring Luo et al.'s evaluation method), the project gradually evolved into what it is now — **Inkling**, a reasoning-aware embedding model fine-tuned on LLM-rated abstract pairings, and built to help researchers uncover links that would be obvious — *if only someone had the time to read everything*.
+- **v2**: Train on LLM-generated bridge triplets (e.g., "Paper A → Blood Viscosity → Paper B").  
+- **Gradio Enhancements**:  
+  - Interactive bridge visualization (D3.js or Plotly).  
+  - User feedback loop for improving the model.  
+- **Automated Updates**: Embed new arXiv papers nightly.  
+- **Domain-Specific Tools**:  
+  - Drug repurposing mode (e.g., "Find new uses for aspirin").  
+  - Interdisciplinary connection finder (e.g., "How does physics inform AI research?").  
 
 ---
 
-## Why Inkling?
+## Why This Matters  
+
+Inkling is **not** a polished product—it’s a chaotic, ADHD-fueled experiment in democratizing scientific discovery. It’s for:  
 
-> Because the right connection is often obvious — once someone points it out.
+- Researchers drowning in paper overload.  
+- Interdisciplinary thinkers who thrive on unexpected connections.  
+- Anyone who’s ever thought, *"I could’ve thought of that!"* after a breakthrough.  
 
-Researchers today are overwhelmed by volume. Inkling helps restore those missed-but-meaningful links between ideas, methods, and fields — links that could inspire new directions, clarify existing work, or enable cross-pollination across domains.
+As Sabine Hossenfelder put it: *"The future of research isn’t in doing more experiments—it’s in connecting the dots we already have."* - Citation needed.
 
 ---
 
-## Citation
+## Status  
 
-> Luo, X., Rechardt, A., Sun, G. et al. Large language models surpass human experts in predicting neuroscience results. *Nat Hum Behav* **9**, 305–315 (2025). [https://www.nature.com/articles/s41562-024-02046-9](https://www.nature.com/articles/s41562-024-02046-9)
+- **Model**: v1 (proof of concept, 50-50 if it does anything or my brain is just playing tricks).
+- **Dataset**: v1 (random pairs, too broad). v2 (in planning, focused on bridge detection).
+- **Interface**: Gradio-powered demo with FAISS backend.  
+- **Next Steps**: Refine training data, automate updates, and scale to all of arXiv.  
 
 ---
 
-## Status
+## Credits  
+
+- **Inspiration**: Sabine Hossenfelder’s ["AIs Predict Research Results" video](https://www.youtube.com/watch?v=Qgrl3JSWWDE).  
+- **Foundational Work**: Don R. Swanson, V. Tshitoyan, X. Luo.  
+- **Model Architecture**: Llama-7B + SentenceTransformer.  
+
+---
+
+## Try It  
+
+[**Live Demo**](https://nomadicsynth-research-compass.hf.space)  
+*Paste an abstract, find a bridge, and see if the connection feels obvious in hindsight.* 🚀  
+
+---
 
-Inkling is in **alpha** and under active development. The current model is hosted via Gradio, with a Hugging Face Space available for live interaction and feedback. Contributions, feedback, and collaboration are welcome.
+**This is a work in progress. Feedback, ideas, and nerd-sniped collaborators are welcome.**