Upload folder using huggingface_hub

.gradio/cached_examples/23/log.csv

@@ -0,0 +1,4 @@
+component 0,timestamp
+"[{""role"": ""user"", ""metadata"": null, ""content"": ""\u3053\u3093\u306b\u3061\u306f"", ""options"": null}, {""role"": ""assistant"", ""metadata"": null, ""content"": ""\u3053\u3093\u306b\u3061\u306f... \u308f\u304b\u3063\u305f\uff01\u3053\u3061\u3089\u306f\u3042\u306a\u305f\u306e\u8cea\u554f\u306b\u95a2\u9023\u3059\u308b\u8ad6\u6587\u3067\u3059\uff1a\n- Turkey\u2014Europe's Bridge to the Middle East: None\n- Hypertensive heart disease.: None\n- Revisiting the Dynamics of Forest Area Change: A Panel Data Assessment: None\n- Pompe disease: PAS-positive lymphocyte vacuoles as diagnostic screening test: None\n- Belly Dancers' Reunion: Some people called Elna a hermit, but she did still answer her door. She might only peek through a crack, but a knock could rouse her out of her darkly comfortable basement. Maybe she thought Harry would drop by. Or the Arabic drummer. Whatever it was that kept her answering, one day she heard a substantial commotion at her side door. She put a bookmark in her novel and climbed out of her basement, head down, aware of the lint on the stairs and the black scum on the bannister. Pausing at the top, she saw two faces on the other side of the door's glass pane. Two large women were peering in, looking curiously at Elna in her cable knit sweater and wool skirt spattered with turquoise and pink flecks. Because she was startled to see such girth on her narrow porch, such an abundance of cheeks, breasts, and upper arms, she opened her door wider than usual. \""We've been sent to you,\"" one of the women said through the screen door. \""We need help,\"" the other interrupted. The two were like water trying to rush through a slow drain pipe. \""There's a hundred dollar prize,\"" Elna managed to decipher from the chaos on her porch. \""For what?\"" she asked. \""Somebody at our office is a neighbor of yours. We told him we'd decided to be exotic dancers at the Silver Zephyr's Amateur Night. He said you used to belly dance and might have costumes.\""\n\n\u3069\u3046\u601d\u3044\u307e\u3059\u304b\uff1f"", ""options"": null}]",2025-05-08 00:59:28.230599
+"[{""role"": ""user"", ""metadata"": null, ""content"": ""LLM\u95a2\u9023\u306e\u8ad6\u6587\u3092\u63a2\u3057\u305f\u3044"", ""options"": null}, {""role"": ""assistant"", ""metadata"": null, ""content"": ""LLM\u95a2\u9023\u306e\u8ad6\u6587\u3092\u63a2\u3057\u305f\u3044... \u308f\u304b\u3063\u305f\uff01\u3053\u3061\u3089\u306f\u3042\u306a\u305f\u306e\u8cea\u554f\u306b\u95a2\u9023\u3059\u308b\u8ad6\u6587\u3067\u3059\uff1a\n- Managing Ethnicity in African Politics: None\n- Lethal drug interaction: Isoniazid and methylxanthines: Nonlethal doses of methylxanthines (caffeine or theophyline) produced dose-dependent lethality in rats pretreated with isoniazid. Isoniazid pretreatment did not alter theophylline concentration in blood or brain, suggesting that the drug interaction was not due to altered distribution or metabolism of theophylline. Death was associated with tonic-clonic seizures and pulmonary congestion. The toxicity of the drug combination was blocked by the anticonvulsants diazepam, barbital, and trimethadione, but not by chlorpromazine, a sedative drug which lacks anticonvulsant activity. Thus, there is a fatal drug interaction between isoniazid and theophylline which may be due to convulsions that trigger a shock lung syndrome.\n- An orbital sequence design for lunar missions combining elliptic orbital segments with constraints: None\n- Electrons for Neutrinos: Lepton Energy Reconstruction in the Resonance Excitation Region: None\n- SYMPOSIUM: ocular injuries.: None\n\n\u3069\u3046\u601d\u3044\u307e\u3059\u304b\uff1f"", ""options"": null}]",2025-05-08 00:59:30.122987
+"[{""role"": ""user"", ""metadata"": null, ""content"": ""I want to know which is the SOTA model for MedQA."", ""options"": null}, {""role"": ""assistant"", ""metadata"": null, ""content"": ""I want to know which is the SOTA model for MedQA.... \u308f\u304b\u3063\u305f\uff01\u3053\u3061\u3089\u306f\u3042\u306a\u305f\u306e\u8cea\u554f\u306b\u95a2\u9023\u3059\u308b\u8ad6\u6587\u3067\u3059\uff1a\n- Characterization of sources of 2\u03c0 phase discontinuity in speckle interferograms: One of the most successful phase-unwrapping algorithms uses branch cuts to join discontinuity sources that mark the beginning or the end of a 2\u03c0 phase discontinuity. Here, using phase-stepping speckle interferometry, we verify that these sources coincide with points of very low or zero modulus and that the displacement of sources as a result of speckle decorrelation between measurements of two phase maps leads to closely spaced dipole pairs of sources in the phase-difference map. By measuring the movement of sources at high magnification, we find that the length distribution of correct branch cuts needed to unwrap a phase-difference map is approximately Gaussian. This provides a theoretical justification for unwrapping with the set of branch cuts that minimizes the sum of squares of cut lengths.\n- Highly Enantioselective Conjugate Addition-Cyclization Cascade Reaction of Malonates with o-Hydroxycinnamaldehydes: Asymmetric Synthesis of 4-Substituted Chromanols: The asymmetric organocatalytic conjugate addition\u2013cyclization reaction of malonates with o-hydroxycinnamaldehydes, which affords 4-substituted chroman-2-ols, has been established using a diphenylprolinol trimethylsilyl (TMS) ether as organocatalyst. The desired products were obtained with good to excellent yields and high enantioselectivities (up to >99% ee). Synthetically useful chroman derivatives were formed after subsequent reactions.\n- Prospective validation of noninvasive prenatal testing on whole genome level (conference abstract - article in Slovak): None\n- Ilhan New, Soldier for the Modern Nation: Recovering a Protestant Martial Alternative to Korean Hegemonic Masculinity: Twentieth century Korean hegemonic masculinity has validated the right to employ violence for the benefit of the nation unchecked by any higher ethical concerns. This arose in the early twentieth century in reaction to a crisis of Korean masculinity, identified by the first Korean nationalists. The supposed pernicious effects of Confucianism created the crisis by making men effete. This in turn \u201cled\u201d Korea to lose its independence. While scholars have recognized alternative Korean masculinities arising since the 1990s, including Catholic masculinities, they have overlooked a Protestant martial masculinity personified by Ilhan New [1895\u20131971]. New was a much lauded business pioneer, but his military career has not been analyzed in terms of its place in the history of masculinity. Mentored by a leading Protestant nationalist, New personified in mid-century an alternative Protestant martial masculinity, which created soldiers fighting for the nation under the discipline of a conventional military, bound by Protestant norms.\n- Single-fraction versus hypofractionated stereotactic radiosurgery for medium-sized brain metastases of 2.5 to 3 cm: Purpose ::: Given recently suggested utility of hypofractionated stereotactic radiosurgery (SRS) in treating large brain metastases (BMs) > 3 cm, we sought to prospectively control tumor size variable to investigate the efficacy and safety of hypofractionated SRS for medium-sized BMs (2.5 to 3 cm) compared with single-fraction SRS.\n\n\u3069\u3046\u601d\u3044\u307e\u3059\u304b\uff1f"", ""options"": null}]",2025-05-08 00:59:35.776590

.gradio/certificate.pem

@@ -0,0 +1,31 @@
+-----BEGIN CERTIFICATE-----
+MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
+TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
+cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
+WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
+ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
+MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
+h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
+0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
+A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
+T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
+B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
+B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
+KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
+OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
+jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
+qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
+rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
+HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
+hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
+ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
+3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
+NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
+ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
+TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
+jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
+oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
+4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
+mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
+emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
+-----END CERTIFICATE-----

.gradio/flagged/log.csv

@@ -0,0 +1,6 @@
+conversation,index,value,flag,timestamp
+"[{""role"": ""user"", ""metadata"": null, ""content"": ""hi"", ""options"": null}, {""role"": ""assistant"", ""metadata"": null, ""content"": ""You typed: hi"", ""options"": null}]",1,Like,2025-05-07 16:07:57.747389
+"[{""role"": ""user"", ""metadata"": null, ""content"": ""hi"", ""options"": null}, {""role"": ""assistant"", ""metadata"": null, ""content"": ""You typed: hi"", ""options"": null}]",1,Spam,2025-05-07 16:08:02.536048
+"[{""role"": ""user"", ""metadata"": null, ""content"": ""hi"", ""options"": null}, {""role"": ""assistant"", ""metadata"": null, ""content"": ""You typed: hi"", ""options"": null}]",1,Inappropriate,2025-05-07 16:08:03.301466
+"[{""role"": ""user"", ""metadata"": null, ""content"": ""hi"", ""options"": null}, {""role"": ""assistant"", ""metadata"": null, ""content"": ""You typed: hi"", ""options"": null}]",1,Like,2025-05-07 16:08:05.084130
+"[{""role"": ""user"", ""metadata"": null, ""content"": ""LLM\u95a2\u9023\u306e\u8ad6\u6587\u3092\u63a2\u3057\u305f\u3044"", ""options"": null}, {""role"": ""assistant"", ""metadata"": null, ""content"": ""LLM\u95a2\u9023\u306e\u8ad6\u6587\u3092\u63a2\u3057\u305f\u3044... \u308f\u304b\u3063\u305f\uff01\u3053\u3061\u3089\u306f\u3042\u306a\u305f\u306e\u8cea\u554f\u306b\u95a2\u9023\u3059\u308b\u8ad6\u6587\u3067\u3059\uff1a\n- Glucocorticoid Receptor Activation Lowers the Threshold for NMDA-Receptor-Dependent Homosynaptic Long-Term Depression in the Hippocampus Through Activation of Voltage-Dependent Calcium Channels: Coussens, Christine M., D. Steven Kerr, and Wickliffe C. Abraham. Glucocorticoid receptor activation lowers the threshold for NMDA-receptor-dependent homosynaptic long-term depression in the hippoc...\n- Summary of the Discussions: None\n- VizieR Online Data Catalog: Sou323 ICRF reference sample (Liu+, 2017): None\n\n\u3069\u3046\u601d\u3044\u307e\u3059\u304b\uff1f"", ""options"": null}]",1,Like,2025-05-09 16:48:21.327291

.idea/.gitignore

@@ -0,0 +1,8 @@
+# Default ignored files
+/shelf/
+/workspace.xml
+# Editor-based HTTP Client requests
+/httpRequests/
+# Datasource local storage ignored files
+/dataSources/
+/dataSources.local.xml

.idea/JStage_RAG.iml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module type="PYTHON_MODULE" version="4">
+  <component name="NewModuleRootManager">
+    <content url="file://$MODULE_DIR$" />
+    <orderEntry type="inheritedJdk" />
+    <orderEntry type="sourceFolder" forTests="false" />
+  </component>
+</module>

.idea/inspectionProfiles/Project_Default.xml

@@ -0,0 +1,14 @@
+<component name="InspectionProjectProfileManager">
+  <profile version="1.0">
+    <option name="myName" value="Project Default" />
+    <inspection_tool class="PyPackageRequirementsInspection" enabled="true" level="WARNING" enabled_by_default="true">
+      <option name="ignoredPackages">
+        <value>
+          <list size="1">
+            <item index="0" class="java.lang.String" itemvalue="omegaconf" />
+          </list>
+        </value>
+      </option>
+    </inspection_tool>
+  </profile>
+</component>

.idea/inspectionProfiles/profiles_settings.xml

@@ -0,0 +1,6 @@
+<component name="InspectionProjectProfileManager">
+  <settings>
+    <option name="USE_PROJECT_PROFILE" value="false" />
+    <version value="1.0" />
+  </settings>
+</component>

.idea/misc.xml

@@ -0,0 +1,7 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="Black">
+    <option name="sdkName" value="rag" />
+  </component>
+  <component name="ProjectRootManager" version="2" project-jdk-name="rag" project-jdk-type="Python SDK" />
+</project>

.idea/modules.xml

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="ProjectModuleManager">
+    <modules>
+      <module fileurl="file://$PROJECT_DIR$/.idea/JStage_RAG.iml" filepath="$PROJECT_DIR$/.idea/JStage_RAG.iml" />
+    </modules>
+  </component>
+</project>

.idea/vcs.xml

@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="VcsDirectoryMappings">
+    <mapping directory="$PROJECT_DIR$/github/s2orc" vcs="Git" />
+  </component>
+</project>

.idea/workspace.xml

@@ -0,0 +1,111 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="AutoImportSettings">
+    <option name="autoReloadType" value="SELECTIVE" />
+  </component>
+  <component name="ChangeListManager">
+    <list default="true" id="c853a62f-55db-44a1-bce5-2e4d5d00c3a8" name="Changes" comment="" />
+    <option name="SHOW_DIALOG" value="false" />
+    <option name="HIGHLIGHT_CONFLICTS" value="true" />
+    <option name="HIGHLIGHT_NON_ACTIVE_CHANGELIST" value="false" />
+    <option name="LAST_RESOLUTION" value="IGNORE" />
+  </component>
+  <component name="FileTemplateManagerImpl">
+    <option name="RECENT_TEMPLATES">
+      <list>
+        <option value="Python Script" />
+      </list>
+    </option>
+  </component>
+  <component name="Git.Settings">
+    <option name="RECENT_GIT_ROOT_PATH" value="$PROJECT_DIR$/github/s2orc" />
+  </component>
+  <component name="MarkdownSettingsMigration">
+    <option name="stateVersion" value="1" />
+  </component>
+  <component name="ProjectColorInfo">{
+  &quot;associatedIndex&quot;: 2
+}</component>
+  <component name="ProjectId" id="2wkzRAQAXupWpP41JeH3GR1AJCi" />
+  <component name="ProjectViewState">
+    <option name="hideEmptyMiddlePackages" value="true" />
+    <option name="showLibraryContents" value="true" />
+  </component>
+  <component name="PropertiesComponent">{
+  &quot;keyToString&quot;: {
+    &quot;Python.app.executor&quot;: &quot;Run&quot;,
+    &quot;RunOnceActivity.OpenProjectViewOnStart&quot;: &quot;true&quot;,
+    &quot;RunOnceActivity.ShowReadmeOnStart&quot;: &quot;true&quot;,
+    &quot;git-widget-placeholder&quot;: &quot;master&quot;,
+    &quot;last_opened_file_path&quot;: &quot;/Users/jiangjunfeng/Desktop/coding/JStage_RAG&quot;,
+    &quot;node.js.detected.package.eslint&quot;: &quot;true&quot;,
+    &quot;node.js.detected.package.tslint&quot;: &quot;true&quot;,
+    &quot;node.js.selected.package.eslint&quot;: &quot;(autodetect)&quot;,
+    &quot;node.js.selected.package.tslint&quot;: &quot;(autodetect)&quot;,
+    &quot;nodejs_package_manager_path&quot;: &quot;npm&quot;,
+    &quot;vue.rearranger.settings.migration&quot;: &quot;true&quot;
+  }
+}</component>
+  <component name="RecentsManager">
+    <key name="CopyFile.RECENT_KEYS">
+      <recent name="$PROJECT_DIR$" />
+    </key>
+  </component>
+  <component name="RunManager">
+    <configuration name="app" type="PythonConfigurationType" factoryName="Python" temporary="true" nameIsGenerated="true">
+      <module name="JStage_RAG" />
+      <option name="ENV_FILES" value="" />
+      <option name="INTERPRETER_OPTIONS" value="" />
+      <option name="PARENT_ENVS" value="true" />
+      <envs>
+        <env name="PYTHONUNBUFFERED" value="1" />
+      </envs>
+      <option name="SDK_HOME" value="" />
+      <option name="WORKING_DIRECTORY" value="$PROJECT_DIR$" />
+      <option name="IS_MODULE_SDK" value="true" />
+      <option name="ADD_CONTENT_ROOTS" value="true" />
+      <option name="ADD_SOURCE_ROOTS" value="true" />
+      <EXTENSION ID="PythonCoverageRunConfigurationExtension" runner="coverage.py" />
+      <option name="SCRIPT_NAME" value="$PROJECT_DIR$/app.py" />
+      <option name="PARAMETERS" value="" />
+      <option name="SHOW_COMMAND_LINE" value="false" />
+      <option name="EMULATE_TERMINAL" value="false" />
+      <option name="MODULE_MODE" value="false" />
+      <option name="REDIRECT_INPUT" value="false" />
+      <option name="INPUT_FILE" value="" />
+      <method v="2" />
+    </configuration>
+    <recent_temporary>
+      <list>
+        <item itemvalue="Python.app" />
+      </list>
+    </recent_temporary>
+  </component>
+  <component name="SharedIndexes">
+    <attachedChunks>
+      <set>
+        <option value="bundled-python-sdk-d68999036c7f-b11f5e8da5ad-com.jetbrains.pycharm.pro.sharedIndexes.bundled-PY-233.14475.56" />
+      </set>
+    </attachedChunks>
+  </component>
+  <component name="SpellCheckerSettings" RuntimeDictionaries="0" Folders="0" CustomDictionaries="0" DefaultDictionary="application-level" UseSingleDictionary="true" transferred="true" />
+  <component name="TaskManager">
+    <task active="true" id="Default" summary="Default task">
+      <changelist id="c853a62f-55db-44a1-bce5-2e4d5d00c3a8" name="Changes" comment="" />
+      <created>1746600349656</created>
+      <option name="number" value="Default" />
+      <option name="presentableId" value="Default" />
+      <updated>1746600349656</updated>
+      <workItem from="1746600368067" duration="9690000" />
+      <workItem from="1746762687197" duration="1020000" />
+      <workItem from="1746776819364" duration="198000" />
+    </task>
+    <servers />
+  </component>
+  <component name="TypeScriptGeneratedFilesManager">
+    <option name="version" value="3" />
+  </component>
+  <component name="com.intellij.coverage.CoverageDataManagerImpl">
+    <SUITE FILE_PATH="coverage/JStage_RAG$app.coverage" NAME="app Coverage Results" MODIFIED="1746776823255" SOURCE_PROVIDER="com.intellij.coverage.DefaultCoverageFileProvider" RUNNER="coverage.py" COVERAGE_BY_TEST_ENABLED="true" COVERAGE_TRACING_ENABLED="false" WORKING_DIRECTORY="$PROJECT_DIR$" />
+  </component>
+</project>

README.md

@@ -1,12 +1,6 @@
 ---
-title: JStage RAG
-emoji: 😻
-colorFrom: yellow
-colorTo: blue
+title: JStage_RAG
+app_file: app.py
 sdk: gradio
 sdk_version: 5.29.0
-app_file: app.py
-pinned: false
 ---
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

app.py

@@ -0,0 +1,128 @@
+import time
+import random
+import ujson as json
+from typing import List
+from dataclasses import dataclass
+import gradio as gr
+
+
+@dataclass
+class Paper:
+    paper_id: str
+    title: str
+    abstract: str
+    authors: List[str] = None
+    year: int = None
+    doi: str = None
+
+
+def load_database(filename):
+    database = []
+    with open(filename, "r", encoding="utf-8") as f:
+        for line in f:
+            json_data = json.loads(line)
+
+            data_point = Paper(
+                paper_id=json_data["paper_id"],
+                title=json_data["title"],
+                abstract=json_data["abstract"],
+                authors=json_data.get("authors", []),
+                year=json_data.get("year", None),
+                doi=json_data.get("doi", None)
+            )
+
+            database.append(data_point)
+    return database
+
+
+class S2ORCRAGPipeline:
+    def __init__(
+            self,
+            s2orc_filename,
+            model=lambda x: x,
+    ):
+        self.s2orc_filename = s2orc_filename
+        self.database = load_database(s2orc_filename)
+        self.model = model
+
+    def retrieve_top_k(
+            self,
+            query: str,
+            topk=5
+    ):
+        # Fake
+        random.seed(len(query) + topk)
+        return random.sample(self.database, topk)
+
+        # Real
+        # TODO: DB-team
+
+
+    def generate_response(
+            self,
+            query,
+            retrieved_papers,
+    ):
+        # Fake
+        response = f"{query}... わかった！こちらはあなたの質問に関連する論文です：\n"
+        for paper in retrieved_papers:
+            response += f"- {paper.title}: {paper.abstract}\n"
+
+        response += "\nどう思いますか？\n"
+
+        response = self.model(response)
+
+        return response
+
+        # Real
+        # TODO: Generation-team
+
+    def __call__(
+            self,
+            query
+    ):
+        # Firstly, retrieve papers from database
+        retrieved_papers = self.retrieve_top_k(query, topk=3)
+
+        # Secondly, generate response based on query and the retrieved papers
+        response = self.generate_response(query, retrieved_papers)
+
+        return response
+
+    def slow_echo(self, message, history):
+        output = self.__call__(query=message)
+        for i in range(len(output)):
+            time.sleep(0.001)
+            yield output[: i + 1]
+
+
+if __name__ == "__main__":
+    # load from S2ORC
+    example_filename = "sample.jsonl"
+
+    pipeline = S2ORCRAGPipeline(
+        s2orc_filename=example_filename,
+        model=lambda x: x
+    )
+
+    initial_messages = [{"role": "assistant", "content": "こんにちは〜今日は何の論文を探したいですか？"}]
+
+    demo = gr.ChatInterface(
+        pipeline.slow_echo,
+        chatbot=gr.Chatbot(
+            value=initial_messages,
+            type="messages",
+            resizable=True, height=700,
+            placeholder="こんにちは〜今日は何の論文を探したいですか？"
+        ),
+        type="messages",
+        flagging_mode="manual",
+        flagging_options=["Like", "Spam", "Inappropriate", "Other"],
+        title="LLMC S2ORC 論文検索 (+RAG)",
+        description="",
+        save_history=True,
+        examples=["こんにちは", "LLM関連の論文を探したい", "Find Suzuki's papers on graphene from 2019 to 2021 in Surface Science Journal."],
+    )
+
+    demo.launch(debug=True, share=True)  # Share=True is failed when using NII Network
+

github/s2orc/.gitignore

@@ -0,0 +1,114 @@
+
+# vscode
+*.vscode
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+.idea/
+
+setup.sh
+
+output/*

github/s2orc/README.md

@@ -0,0 +1,152 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/allenai/s2orc/master/assets/logo.svg" alt="Logo of S2ORC, pronounced stork" width="20%">
+</p>
+
+
+# S2ORC: The Semantic Scholar Open Research Corpus
+
+S2ORC is a general-purpose corpus for NLP and text mining research over scientific papers.
+
+* **[Download instructions](#download-instructions)**.
+* S2ORC was developed by [Kyle Lo](https://kyleclo.github.io/) and [Lucy Lu Wang](https://llwang.net/) at the [Allen Institute for AI](https://allenai.org/). It is now being maintained as a product offering by the API team at [Semantic Scholar](https://www.semanticscholar.org/product/api).
+* S2ORC is released under the [ODC-By 1.0](https://opendatacommons.org/licenses/by/1-0/).  By using S2ORC, you agree to the terms in the license.
+* Please cite [our ACL 2020 paper](https://www.aclweb.org/anthology/2020.acl-main.447) if you use S2ORC for your project.  See the [BibTeX](#citation). You can also watch [our 12 min ACL 2020 talk](https://slideslive.com/38929131/s2orc-the-semantic-scholar-open-research-corpus).
+
+
+
+## News and Releases
+
+⭐ **S2ORC now available through S2 API**
+
+It's Jan 2023; happy new year! After years of managing S2ORC as a research project, it has now been adopted as a core dataset offering through the [Semantic Scholar Public API](https://www.semanticscholar.org/product/api). Please look for the instructions under "Bulk Dataset" for download! 
+
+S2ORC is now available through the [Semantic Scholar Public API](https://www.semanticscholar.org/product/api) as a "Bulk Dataset". It is continuously being rebuilt so if you access it through there, you'll get access to **new** papers as well!
+
+**Software Release: 2021-02-01**
+
+- Released [s2orc-doc2json](https://github.com/allenai/s2orc-doc2json) to support parsing of PDF and LaTeX to JSON format.
+
+
+**S2ORC Release: 2020-07-05**
+
+- Released a new version of S2ORC containing papers up until 2020-04-14, bringing full text coverage from 8M to 12M.
+- Lifted some paper filters to be more lenient toward papers that don't have sufficient amount of text.  This brought total paper count to 136M from 81M.   
+- Updated the schema to keep paper metadata and parsed paper text separate.
+- Fixed major bugs such as (i) missing section names, (ii) inline citation mention links that don't resolve to bibliographies, and (iii) unpredictable typing in certain metadata fields. 
+- Omitted LaTeX parses from this release.  They will be added in a subsequent release.  Part of the dataset schema change is to accommodate incremental releases (e.g. LaTeX-only release without having to re-run PDF parsing). 
+
+- Feb 2023 update: We are no longer supporting access to this version & recommend everyone use the latest way of accessing S2ORC through the Semantic Scholar Public API.  If you must use this version and need assistance, please contact Kyle and Lucy.
+
+
+**Project Status: 2020-04-07**
+
+- S2ORC has been accepted to ACL 2020!
+- We've changed the name of the project to S2ORC.  We will update the [preprint](https://arxiv.org/abs/1911.02782) shortly with the new name.
+- The [BibTeX citation](#citation) has also been changed to reflect this.
+- Feb 2023 update: We are no longer supporting access to this version & recommend everyone use the latest way of accessing S2ORC through the Semantic Scholar Public API.  If you must use this version and need assistance, please contact Kyle and Lucy.
+
+
+**S2ORC Release: 2019-09-28**
+
+- Statistics: 81M+ paper nodes; 73M+ gold abstracts; 8M+ full text papers
+- Due to release bugs (e.g. missing section names), we no longer recommend usage of this version.  If you must use this version and need assistance, please contact Kyle and Lucy.
+
+
+## Download instructions
+
+The original S2ORC dataset files were refactored into multiple datasets available through [the Semantic Scholar APIs](https://api.semanticscholar.org/) (See detailed documentation [here](https://api.semanticscholar.org/api-docs/datasets)).
+
+Once you obtain an API key from [Semantic Scholar Public API](https://www.semanticscholar.org/product/api), you should be able to access these bulk dumps like so:
+```
+import json
+import os
+import re
+import requests
+import wget
+from tqdm import tqdm
+
+# modify these
+API_KEY = "..."
+DATASET_NAME = "s2orc"
+LOCAL_PATH = "/my/local/path/for/s2orc/"
+os.makedirs(LOCAL_PATH, exist_ok=True)
+
+# get latest release's ID
+response = requests.get("https://api.semanticscholar.org/datasets/v1/release/latest").json()
+RELEASE_ID = response["release_id"]
+print(f"Latest release ID: {RELEASE_ID}")
+
+# get the download links for the s2orc dataset; needs to pass API key through `x-api-key` header
+# download via wget. this can take a while...
+response = requests.get(f"https://api.semanticscholar.org/datasets/v1/release/{RELEASE_ID}/dataset/{DATASET_NAME}/", headers={"x-api-key": API_KEY}).json()
+for url in tqdm(response["files"]):
+    match = re.match(r"https://ai2-s2ag.s3.amazonaws.com/staging/(.*)/s2orc/(.*).gz(.*)", url)
+    assert match.group(1) == RELEASE_ID
+    SHARD_ID = match.group(2)
+    wget.download(url, out=os.path.join(LOCAL_PATH, f"{SHARD_ID}.gz"))
+print("Downloaded all shards.")
+```
+
+For questions, feature requests, bug reports, please search existing issues on [the s2-folks Github repo](https://github.com/allenai/s2-folks/issues?q=is%3Aissue) before creating [a new issue](https://github.com/allenai/s2-folks/issues/new).
+
+
+## Contact us
+
+The best way to contact us is through email.  Don't hesitate to reach out about anything; we've helped a lot of people get started with the dataset, which can be a bit daunting given its size.
+
+**Email:** Please include `{kylel, lucyw, rodneyk` on all correspondence.
+
+**Twitter** [@kylelostat](https://twitter.com/kylelostat), [@lucyluwang](https://twitter.com/lucyluwang)
+
+**Give us Feedback:**  Totally optional, but we'd love to hear how you're using this dataset & any feedback for improving it.  Send us an email or leave a Github Issue. 
+
+**Report issues:** 
+
+S2ORC is now being maintained by the S2 API product team. For questions, feature requests, bug reports, please search existing issues on [the s2-folks Github repo](https://github.com/allenai/s2-folks/issues?q=is%3Aissue) before creating [a new issue](https://github.com/allenai/s2-folks/issues/new). 
+ 
+
+## FAQ
+
+#### What's the difference between [S2ORC](https://arxiv.org/abs/1911.02782) and [S2AG](https://dl.acm.org/doi/fullHtml/10.1145/3487553.3527147)?
+At a high level:
+
+- S2AG is everything that is covered in the literature graph, including Nodes (i.e. papers, authors) and Edges (i.e. citations, authorship). A `paper` in S2AG is represented by a bundle of Metadata, such as the Title, Authors, Year, Venue, Abstract, etc. You can download different releases of S2AG via the [the Semantic Scholar APIs](https://api.semanticscholar.org/) (See detailed documentation [here](https://api.semanticscholar.org/api-docs/datasets)).
+
+- S2ORC is everything that is machine-readable **full text** of the paper, which we derive using models run on the paper's PDF. The original S2ORC dataset files are no longer available for download. They were refactored into multiple datasets available through [the Semantic Scholar APIs](https://api.semanticscholar.org/) (See detailed documentation [here](https://api.semanticscholar.org/api-docs/datasets)).  
+
+If you're unsure what to use or cite, please email us and we'd be happy to discuss your project with you.
+
+#### I have an old version of S2ORC. How is it different from the version of S2ORC from the S2 API?
+
+- Original S2ORC was a research project w/ original code. The current S2ORC is a reimplementation of the ideas from the research project within the Semantic Scholar data pipeline. As such, there can be differences due to low level implementation details being different.
+
+- Current S2ORC is maintained by a different team than the original researchers. 
+
+
+- Original S2ORC was released under a non-commercial license. The current S2ORC is released under an ODC-By 1.0 license. We ask that users take care to double-check whether their intended usage of S2ORC and its underlying contents is permissible under this license.
+
+
+## License
+
+S2ORC is currently released through the [Semantic Scholar Public API](https://www.semanticscholar.org/product/api) under the [ODC-By 1.0](https://opendatacommons.org/licenses/by/1-0/). By using S2ORC, you are agreeing to its usage terms. 
+
+
+
+## Citation
+
+If using this dataset, please cite:
+
+```
+@inproceedings{lo-wang-2020-s2orc,
+    title = "{S}2{ORC}: The Semantic Scholar Open Research Corpus",
+    author = "Lo, Kyle  and Wang, Lucy Lu  and Neumann, Mark  and Kinney, Rodney  and Weld, Daniel",
+    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
+    month = jul,
+    year = "2020",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/2020.acl-main.447",
+    doi = "10.18653/v1/2020.acl-main.447",
+    pages = "4969--4983"
+}
+```

github/s2orc/requirements.txt

@@ -0,0 +1,2 @@
+boto3
+tqdm

github/s2orc/setup.py

@@ -0,0 +1,15 @@
+#!/usr/bin/python
+import setuptools
+
+setuptools.setup(
+    name='s2orc',
+    version='0.1',
+    packages=setuptools.find_packages(),
+    install_requires=[
+    ],
+    tests_require=[
+    ],
+    zip_safe=False,
+    test_suite='py.test',
+    entry_points='',
+)

requirements.txt

@@ -0,0 +1,2 @@
+ujson==5.10.0
+gradio==5.29.0