first commit for the demo

CODE_OF_CONDUCT.md

@@ -0,0 +1,80 @@
+# Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to make participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all project spaces, and it also applies when
+an individual is representing the project or its community in public spaces.
+Examples of representing a project or community include using an official
+project e-mail address, posting via an official social media account, or acting
+as an appointed representative at an online or offline event. Representation of
+a project may be further defined and clarified by project maintainers.
+
+This Code of Conduct also applies outside the project spaces when there is a
+reasonable belief that an individual's behavior may have a negative impact on
+the project or its community.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at <[email protected]>. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Contributing to CiT
+We want to make contributing to this project as easy and transparent as
+possible.
+
+## Pull Requests
+We actively welcome your pull requests.
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Facebook's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
+disclosure of security bugs. In those cases, please go through the process
+outlined on that page and do not file a public issue.
+
+## License
+By contributing to CiT, you agree that your contributions will be licensed
+under the LICENSE file in the root directory of this source tree.

HISTORY.md

@@ -0,0 +1,10 @@
+## 1.2.0
+
+* ViT-B/32 trained on Laion2B-en
+* add missing openai RN50x64 model
+
+## 1.1.1
+
+* ViT-B/16+
+* Add grad checkpointing support
+* more robust data loader

LICENSE

@@ -0,0 +1,399 @@
+Attribution-NonCommercial 4.0 International
+
+=======================================================================
+
+Creative Commons Corporation ("Creative Commons") is not a law firm and
+does not provide legal services or legal advice. Distribution of
+Creative Commons public licenses does not create a lawyer-client or
+other relationship. Creative Commons makes its licenses and related
+information available on an "as-is" basis. Creative Commons gives no
+warranties regarding its licenses, any material licensed under their
+terms and conditions, or any related information. Creative Commons
+disclaims all liability for damages resulting from their use to the
+fullest extent possible.
+
+Using Creative Commons Public Licenses
+
+Creative Commons public licenses provide a standard set of terms and
+conditions that creators and other rights holders may use to share
+original works of authorship and other material subject to copyright
+and certain other rights specified in the public license below. The
+following considerations are for informational purposes only, are not
+exhaustive, and do not form part of our licenses.
+
+     Considerations for licensors: Our public licenses are
+     intended for use by those authorized to give the public
+     permission to use material in ways otherwise restricted by
+     copyright and certain other rights. Our licenses are
+     irrevocable. Licensors should read and understand the terms
+     and conditions of the license they choose before applying it.
+     Licensors should also secure all rights necessary before
+     applying our licenses so that the public can reuse the
+     material as expected. Licensors should clearly mark any
+     material not subject to the license. This includes other CC-
+     licensed material, or material used under an exception or
+     limitation to copyright. More considerations for licensors:
+  wiki.creativecommons.org/Considerations_for_licensors
+
+     Considerations for the public: By using one of our public
+     licenses, a licensor grants the public permission to use the
+     licensed material under specified terms and conditions. If
+     the licensor's permission is not necessary for any reason--for
+     example, because of any applicable exception or limitation to
+     copyright--then that use is not regulated by the license. Our
+     licenses grant only permissions under copyright and certain
+     other rights that a licensor has authority to grant. Use of
+     the licensed material may still be restricted for other
+     reasons, including because others have copyright or other
+     rights in the material. A licensor may make special requests,
+     such as asking that all changes be marked or described.
+     Although not required by our licenses, you are encouraged to
+     respect those requests where reasonable. More_considerations
+     for the public: 
+  wiki.creativecommons.org/Considerations_for_licensees
+
+=======================================================================
+
+Creative Commons Attribution-NonCommercial 4.0 International Public
+License
+
+By exercising the Licensed Rights (defined below), You accept and agree
+to be bound by the terms and conditions of this Creative Commons
+Attribution-NonCommercial 4.0 International Public License ("Public
+License"). To the extent this Public License may be interpreted as a
+contract, You are granted the Licensed Rights in consideration of Your
+acceptance of these terms and conditions, and the Licensor grants You
+such rights in consideration of benefits the Licensor receives from
+making the Licensed Material available under these terms and
+conditions.
+
+Section 1 -- Definitions.
+
+  a. Adapted Material means material subject to Copyright and Similar
+     Rights that is derived from or based upon the Licensed Material
+     and in which the Licensed Material is translated, altered,
+     arranged, transformed, or otherwise modified in a manner requiring
+     permission under the Copyright and Similar Rights held by the
+     Licensor. For purposes of this Public License, where the Licensed
+     Material is a musical work, performance, or sound recording,
+     Adapted Material is always produced where the Licensed Material is
+     synched in timed relation with a moving image.
+
+  b. Adapter's License means the license You apply to Your Copyright
+     and Similar Rights in Your contributions to Adapted Material in
+     accordance with the terms and conditions of this Public License.
+
+  c. Copyright and Similar Rights means copyright and/or similar rights
+     closely related to copyright including, without limitation,
+     performance, broadcast, sound recording, and Sui Generis Database
+     Rights, without regard to how the rights are labeled or
+     categorized. For purposes of this Public License, the rights
+     specified in Section 2(b)(1)-(2) are not Copyright and Similar
+     Rights.
+  d. Effective Technological Measures means those measures that, in the
+     absence of proper authority, may not be circumvented under laws
+     fulfilling obligations under Article 11 of the WIPO Copyright
+     Treaty adopted on December 20, 1996, and/or similar international
+     agreements.
+
+  e. Exceptions and Limitations means fair use, fair dealing, and/or
+     any other exception or limitation to Copyright and Similar Rights
+     that applies to Your use of the Licensed Material.
+
+  f. Licensed Material means the artistic or literary work, database,
+     or other material to which the Licensor applied this Public
+     License.
+
+  g. Licensed Rights means the rights granted to You subject to the
+     terms and conditions of this Public License, which are limited to
+     all Copyright and Similar Rights that apply to Your use of the
+     Licensed Material and that the Licensor has authority to license.
+
+  h. Licensor means the individual(s) or entity(ies) granting rights
+     under this Public License.
+
+  i. NonCommercial means not primarily intended for or directed towards
+     commercial advantage or monetary compensation. For purposes of
+     this Public License, the exchange of the Licensed Material for
+     other material subject to Copyright and Similar Rights by digital
+     file-sharing or similar means is NonCommercial provided there is
+     no payment of monetary compensation in connection with the
+     exchange.
+
+  j. Share means to provide material to the public by any means or
+     process that requires permission under the Licensed Rights, such
+     as reproduction, public display, public performance, distribution,
+     dissemination, communication, or importation, and to make material
+     available to the public including in ways that members of the
+     public may access the material from a place and at a time
+     individually chosen by them.
+
+  k. Sui Generis Database Rights means rights other than copyright
+     resulting from Directive 96/9/EC of the European Parliament and of
+     the Council of 11 March 1996 on the legal protection of databases,
+     as amended and/or succeeded, as well as other essentially
+     equivalent rights anywhere in the world.
+
+  l. You means the individual or entity exercising the Licensed Rights
+     under this Public License. Your has a corresponding meaning.
+
+Section 2 -- Scope.
+
+  a. License grant.
+
+       1. Subject to the terms and conditions of this Public License,
+          the Licensor hereby grants You a worldwide, royalty-free,
+          non-sublicensable, non-exclusive, irrevocable license to
+          exercise the Licensed Rights in the Licensed Material to:
+
+            a. reproduce and Share the Licensed Material, in whole or
+               in part, for NonCommercial purposes only; and
+
+            b. produce, reproduce, and Share Adapted Material for
+               NonCommercial purposes only.
+
+       2. Exceptions and Limitations. For the avoidance of doubt, where
+          Exceptions and Limitations apply to Your use, this Public
+          License does not apply, and You do not need to comply with
+          its terms and conditions.
+
+       3. Term. The term of this Public License is specified in Section
+          6(a).
+
+       4. Media and formats; technical modifications allowed. The
+          Licensor authorizes You to exercise the Licensed Rights in
+          all media and formats whether now known or hereafter created,
+          and to make technical modifications necessary to do so. The
+          Licensor waives and/or agrees not to assert any right or
+          authority to forbid You from making technical modifications
+          necessary to exercise the Licensed Rights, including
+          technical modifications necessary to circumvent Effective
+          Technological Measures. For purposes of this Public License,
+          simply making modifications authorized by this Section 2(a)
+          (4) never produces Adapted Material.
+
+       5. Downstream recipients.
+
+            a. Offer from the Licensor -- Licensed Material. Every
+               recipient of the Licensed Material automatically
+               receives an offer from the Licensor to exercise the
+               Licensed Rights under the terms and conditions of this
+               Public License.
+
+            b. No downstream restrictions. You may not offer or impose
+               any additional or different terms or conditions on, or
+               apply any Effective Technological Measures to, the
+               Licensed Material if doing so restricts exercise of the
+               Licensed Rights by any recipient of the Licensed
+               Material.
+
+       6. No endorsement. Nothing in this Public License constitutes or
+          may be construed as permission to assert or imply that You
+          are, or that Your use of the Licensed Material is, connected
+          with, or sponsored, endorsed, or granted official status by,
+          the Licensor or others designated to receive attribution as
+          provided in Section 3(a)(1)(A)(i).
+
+  b. Other rights.
+
+       1. Moral rights, such as the right of integrity, are not
+          licensed under this Public License, nor are publicity,
+          privacy, and/or other similar personality rights; however, to
+          the extent possible, the Licensor waives and/or agrees not to
+          assert any such rights held by the Licensor to the limited
+          extent necessary to allow You to exercise the Licensed
+          Rights, but not otherwise.
+
+       2. Patent and trademark rights are not licensed under this
+          Public License.
+
+       3. To the extent possible, the Licensor waives any right to
+          collect royalties from You for the exercise of the Licensed
+          Rights, whether directly or through a collecting society
+          under any voluntary or waivable statutory or compulsory
+          licensing scheme. In all other cases the Licensor expressly
+          reserves any right to collect such royalties, including when
+          the Licensed Material is used other than for NonCommercial
+          purposes.
+
+Section 3 -- License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the
+following conditions.
+
+  a. Attribution.
+
+       1. If You Share the Licensed Material (including in modified
+          form), You must:
+
+            a. retain the following if it is supplied by the Licensor
+               with the Licensed Material:
+
+                 i. identification of the creator(s) of the Licensed
+                    Material and any others designated to receive
+                    attribution, in any reasonable manner requested by
+                    the Licensor (including by pseudonym if
+                    designated);
+
+                ii. a copyright notice;
+
+               iii. a notice that refers to this Public License;
+
+                iv. a notice that refers to the disclaimer of
+                    warranties;
+
+                 v. a URI or hyperlink to the Licensed Material to the
+                    extent reasonably practicable;
+
+            b. indicate if You modified the Licensed Material and
+               retain an indication of any previous modifications; and
+
+            c. indicate the Licensed Material is licensed under this
+               Public License, and include the text of, or the URI or
+               hyperlink to, this Public License.
+
+       2. You may satisfy the conditions in Section 3(a)(1) in any
+          reasonable manner based on the medium, means, and context in
+          which You Share the Licensed Material. For example, it may be
+          reasonable to satisfy the conditions by providing a URI or
+          hyperlink to a resource that includes the required
+          information.
+
+       3. If requested by the Licensor, You must remove any of the
+          information required by Section 3(a)(1)(A) to the extent
+          reasonably practicable.
+
+       4. If You Share Adapted Material You produce, the Adapter's
+          License You apply must not prevent recipients of the Adapted
+          Material from complying with this Public License.
+
+Section 4 -- Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that
+apply to Your use of the Licensed Material:
+
+  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
+     to extract, reuse, reproduce, and Share all or a substantial
+     portion of the contents of the database for NonCommercial purposes
+     only;
+
+  b. if You include all or a substantial portion of the database
+     contents in a database in which You have Sui Generis Database
+     Rights, then the database in which You have Sui Generis Database
+     Rights (but not its individual contents) is Adapted Material; and
+
+  c. You must comply with the conditions in Section 3(a) if You Share
+     all or a substantial portion of the contents of the database.
+
+For the avoidance of doubt, this Section 4 supplements and does not
+replace Your obligations under this Public License where the Licensed
+Rights include other Copyright and Similar Rights.
+
+Section 5 -- Disclaimer of Warranties and Limitation of Liability.
+
+  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
+     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
+     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
+     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
+     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
+     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
+     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
+     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
+     KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
+     ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
+
+  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
+     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
+     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
+     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
+     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
+     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
+     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
+     DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
+     IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
+
+  c. The disclaimer of warranties and limitation of liability provided
+     above shall be interpreted in a manner that, to the extent
+     possible, most closely approximates an absolute disclaimer and
+     waiver of all liability.
+
+Section 6 -- Term and Termination.
+
+  a. This Public License applies for the term of the Copyright and
+     Similar Rights licensed here. However, if You fail to comply with
+     this Public License, then Your rights under this Public License
+     terminate automatically.
+
+  b. Where Your right to use the Licensed Material has terminated under
+     Section 6(a), it reinstates:
+
+       1. automatically as of the date the violation is cured, provided
+          it is cured within 30 days of Your discovery of the
+          violation; or
+
+       2. upon express reinstatement by the Licensor.
+
+     For the avoidance of doubt, this Section 6(b) does not affect any
+     right the Licensor may have to seek remedies for Your violations
+     of this Public License.
+
+  c. For the avoidance of doubt, the Licensor may also offer the
+     Licensed Material under separate terms or conditions or stop
+     distributing the Licensed Material at any time; however, doing so
+     will not terminate this Public License.
+
+  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
+     License.
+
+Section 7 -- Other Terms and Conditions.
+
+  a. The Licensor shall not be bound by any additional or different
+     terms or conditions communicated by You unless expressly agreed.
+
+  b. Any arrangements, understandings, or agreements regarding the
+     Licensed Material not stated herein are separate from and
+     independent of the terms and conditions of this Public License.
+
+Section 8 -- Interpretation.
+
+  a. For the avoidance of doubt, this Public License does not, and
+     shall not be interpreted to, reduce, limit, restrict, or impose
+     conditions on any use of the Licensed Material that could lawfully
+     be made without permission under this Public License.
+
+  b. To the extent possible, if any provision of this Public License is
+     deemed unenforceable, it shall be automatically reformed to the
+     minimum extent necessary to make it enforceable. If the provision
+     cannot be reformed, it shall be severed from this Public License
+     without affecting the enforceability of the remaining terms and
+     conditions.
+
+  c. No term or condition of this Public License will be waived and no
+     failure to comply consented to unless expressly agreed to by the
+     Licensor.
+
+  d. Nothing in this Public License constitutes or may be interpreted
+     as a limitation upon, or waiver of, any privileges and immunities
+     that apply to the Licensor or You, including from the legal
+     processes of any jurisdiction or authority.
+
+=======================================================================
+
+Creative Commons is not a party to its public
+licenses. Notwithstanding, Creative Commons may elect to apply one of
+its public licenses to material it publishes and in those instances
+will be considered the “Licensor.” The text of the Creative Commons
+public licenses is dedicated to the public domain under the CC0 Public
+Domain Dedication. Except for the limited purpose of indicating that
+material is shared under a Creative Commons public license or as
+otherwise permitted by the Creative Commons policies published at
+creativecommons.org/policies, Creative Commons does not authorize the
+use of the trademark "Creative Commons" or any other trademark or logo
+of Creative Commons without its prior written consent including,
+without limitation, in connection with any unauthorized modifications
+to any of its public licenses or any other arrangements,
+understandings, or agreements concerning use of licensed material. For
+the avoidance of doubt, this paragraph does not form part of the
+public licenses.
+
+Creative Commons may be contacted at creativecommons.org.

MANIFEST.in

@@ -0,0 +1,3 @@
+include src/open_clip/bpe_simple_vocab_16e6.txt.gz
+include src/open_clip/model_configs/*.json
+

Makefile

@@ -0,0 +1,9 @@
+install: ## [Local development] Upgrade pip, install requirements, install package.
+	python -m pip install -U pip
+	python -m pip install -e .
+
+install-dev: ## [Local development] Install test requirements
+	python -m pip install -r requirements-test.txt
+
+test: ## [Local development] Run unit tests
+	python -m pytest -x -s -v tests

app.py.py

@@ -0,0 +1,135 @@
+
+import gradio as gr
+# Switch path to root of project
+import os
+import sys
+# Get the current working directory
+current_dir = os.getcwd()
+src_path = os.path.join(current_dir, 'src')
+os.chdir(src_path)
+# Add src directory to sys.path
+sys.path.append(src_path)
+from open_clip import create_model_and_transforms
+from huggingface_hub import hf_hub_download
+from open_clip import HFTokenizer
+import torch
+
+class create_unimed_clip_model:
+    def __init__(self, model_name):
+        # self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        self.device = 'cpu'
+        mean = (0.48145466, 0.4578275, 0.40821073)  # OpenAI dataset mean
+        std = (0.26862954, 0.26130258, 0.27577711)  # OpenAI dataset std
+        if model_name == "ViT/B-16":
+            # Download the weights
+            weights_path = hf_hub_download(
+                repo_id="UzairK/unimed-clip-vit-b16",
+                filename="unimed-clip-vit-b16.pt"
+            )
+            self.pretrained = weights_path  # Path to pretrained weights
+            self.text_encoder_name = "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract"
+            self.model_name = "ViT-B-16-quickgelu"
+        elif model_name == 'ViT/L-14@336px-base-text':
+            # Download the weights
+            self.model_name = "ViT-L-14-336-quickgelu"
+            weights_path = hf_hub_download(
+                repo_id="UzairK/unimed_clip_vit_l14_base_text_encoder",
+                filename="unimed_clip_vit_l14_base_text_encoder.pt"
+            )
+            self.pretrained = weights_path  # Path to pretrained weights
+            self.text_encoder_name = "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract"
+        self.tokenizer = HFTokenizer(
+            self.text_encoder_name,
+            context_length=256,
+            **{},
+        )
+        self.model, _, self.processor = create_model_and_transforms(
+            self.model_name,
+            self.pretrained,
+            precision='amp',
+            device=self.device,
+            force_quick_gelu=True,
+            pretrained_image=False,
+            mean=mean, std=std,
+            inmem=True,
+            text_encoder_name=self.text_encoder_name,
+        )
+
+    def __call__(self, input_image, candidate_labels, hypothesis_template):
+        # Preprocess input
+        input_image = self.processor(input_image).unsqueeze(0).to(self.device)
+        if hypothesis_template == "":
+            texts = [
+                self.tokenizer(cls_text).to(self.device)
+                for cls_text in candidate_labels
+            ]
+        else:
+            texts = [
+                self.tokenizer(hypothesis_template + " " + cls_text).to(self.device)
+                for cls_text in candidate_labels
+            ]
+        texts = torch.cat(texts, dim=0)
+        # Perform inference
+        with torch.no_grad():
+            text_features = self.model.encode_text(texts)
+            text_features = text_features / text_features.norm(dim=-1, keepdim=True)
+            image_features = self.model.encode_image(input_image)
+            logits = (image_features @ text_features.t()).softmax(dim=-1).cpu().numpy()
+            return {cls_text: float(score) for cls_text, score in zip(candidate_labels, logits[0])}
+
+pipes = {
+    "ViT/B-16": create_unimed_clip_model(model_name="ViT/B-16"),
+    "ViT/L-14@336px-base-text": create_unimed_clip_model(model_name='ViT/L-14@336px-base-text'),
+}
+# Define Gradio inputs and outputs
+inputs = [
+    gr.Image(type="pil", label="Image"),
+    gr.Textbox(label="Candidate Labels (comma-separated)"),
+    gr.Radio(
+        choices=["ViT/B-16", "ViT/L-14@336px-base-text"],
+        label="Model",
+        value="ViT/B-16",
+    ),
+    gr.Textbox(label="Prompt Template", placeholder="Optional prompt template as prefix",
+               value=""),
+]
+outputs = gr.Label(label="Predicted Scores")
+
+def shot(image, labels_text, model_name, hypothesis_template):
+    labels = [label.strip(" ") for label in labels_text.strip(" ").split(",")]
+    res = pipes[model_name](input_image=image,
+           candidate_labels=labels,
+           hypothesis_template=hypothesis_template)
+    return {single_key: res[single_key] for single_key in res.keys()}
+# Define examples
+
+examples = [
+    ["../docs/sample_images/brain_MRI.jpg", "CT scan image displaying the anatomical structure of the right kidney., pneumonia is indicated in this chest X-ray image., this is a MRI photo of a brain., this fundus image shows optic nerve damage due to glaucoma., a histopathology slide showing Tumor, Cardiomegaly is evident in the X-ray image of the chest.", "ViT/B-16", ""],
+    ["../docs/sample_images/ct_scan_right_kidney.jpg",
+     "CT scan image displaying the anatomical structure of the right kidney., pneumonia is indicated in this chest X-ray image., this is a MRI photo of a brain., this fundus image shows optic nerve damage due to glaucoma., a histopathology slide showing Tumor, Cardiomegaly is evident in the X-ray image of the chest.",
+     "ViT/B-16", ""],
+    ["../docs/sample_images/retina_glaucoma.jpg",
+     "CT scan image displaying the anatomical structure of the right kidney., pneumonia is indicated in this chest X-ray image., this is a MRI photo of a brain., this fundus image shows optic nerve damage due to glaucoma., a histopathology slide showing Tumor, Cardiomegaly is evident in the X-ray image of the chest.",
+     "ViT/B-16", ""],
+    ["../docs/sample_images/tumor_histo_pathology.jpg",
+     "CT scan image displaying the anatomical structure of the right kidney., pneumonia is indicated in this chest X-ray image., this is a MRI photo of a brain., this fundus image shows optic nerve damage due to glaucoma., a histopathology slide showing Tumor, Cardiomegaly is evident in the X-ray image of the chest.",
+     "ViT/B-16", ""],
+    ["../docs/sample_images/xray_cardiomegaly.jpg",
+     "CT scan image displaying the anatomical structure of the right kidney., pneumonia is indicated in this chest X-ray image., this is a MRI photo of a brain., this fundus image shows optic nerve damage due to glaucoma., a histopathology slide showing Tumor, Cardiomegaly is evident in the X-ray image of the chest.",
+     "ViT/B-16", ""],
+    ["../docs/sample_images//xray_pneumonia.png",
+     "CT scan image displaying the anatomical structure of the right kidney., pneumonia is indicated in this chest X-ray image., this is a MRI photo of a brain., this fundus image shows optic nerve damage due to glaucoma., a histopathology slide showing Tumor, Cardiomegaly is evident in the X-ray image of the chest.",
+     "ViT/B-16", ""],
+]
+
+iface = gr.Interface(shot,
+            inputs,
+            outputs,
+            examples=examples,
+            description="""<p>Demo for UniMed CLIP, a family of strong Medical Contrastive VLMs trained on UniMed-dataset. For more information about our project, refer to our paper and github repository. <br>
+            Paper: <a href='https://arxiv.org/abs/2412.10372'>https://arxiv.org/abs/2412.10372</a> <br>
+            Github: <a href='https://github.com/mbzuai-oryx/UniMed-CLIP'>https://github.com/mbzuai-oryx/UniMed-CLIP</a> <br><br>
+            <b>[DEMO USAGE]</b> To begin with the demo, provide a picture (either upload manually, or select from the given examples) and class labels. Optionally you can also add template as an prefix to the class labels. <br> </p>""",
+            title="Zero-shot Medical Image Classification with UniMed-CLIP")
+
+iface.launch()

clipeval/dataset_catalog.json

@@ -0,0 +1,157 @@
+{      "acl_mri": {
+        "path": "/scratch/uzair/med/evaluation_datasets/acl_mri/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+      "meniscal_mri": {
+        "path": "/scratch/uzair/med/evaluation_datasets/meniscal_mri/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+
+    "breast_us": {
+        "path": "/scratch/uzair/med/evaluation_datasets/breast_us/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+  "thyroid_us": {
+        "path": "/scratch/uzair/med/evaluation_datasets/thyroid_us/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+  "chexpert-5x200": {
+        "path": "/scratch/uzair/med/evaluation_datasets/chexpert/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+      "rsna_pneumonia": {
+        "path": "/scratch/uzair/med/evaluation_datasets/rsna_custom_test_set/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+  "radimagenet": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+    "imagenet": {
+        "path": "/scratch/uzair/med/evaluation_datasets/talal_datasets/imagenet/images/",
+        "type": "imagefolder",
+        "train": "train",
+        "test": "val"
+    },
+      "CT_axial": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+        "CT_coronal": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+        "CT_sagittal": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+          "dr_regular": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+          "dr_uwf": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+          "LC25000_lung": {
+        "path": "/scratch/uzair/med/evaluation_datasets/LC25000_test/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+          "LC25000_colon": {
+        "path": "/scratch/uzair/med/evaluation_datasets/LC25000_test/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+          "PCAM": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+          "NCK_CRC": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+  "BACH": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+          "SICAPv2": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+          "Osteo": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+
+          "skin_cancer": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+
+          "skin_tumor": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+
+  "refuge_retina": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+
+  "five_retina": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    },
+
+  "odir_retina": {
+        "path": "/scratch/uzair/med/evaluation_datasets/",
+        "type": "special",
+        "train": "train",
+        "test": "val"
+    }
+}

clipeval/datasets.py

@@ -0,0 +1,718 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+from collections import defaultdict
+import json
+import os
+import pickle
+import zipfile
+import pandas as pd
+import numpy as np
+from PIL import Image, ImageFile
+import ast
+import torch
+import random
+from constants import CHEXPERT_COMPETITION_TASKS
+from torchvision import transforms
+from torchvision import datasets as t_datasets
+import ast
+
+ImageFile.LOAD_TRUNCATED_IMAGES = True
+
+
+def pil_loader(path):
+    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
+    with open(path, 'rb') as f:
+        img = Image.open(f)
+        return img.convert('RGB')
+
+
+class FileListDataset(torch.utils.data.Dataset):
+    def __init__(self, images, labels, transform=None, target_transform=None):
+        self.transform = transform
+        self.target_transform = target_transform
+        self.images = np.load(images)
+        self.labels = np.load(labels)
+
+    def __getitem__(self, index):
+        img = pil_loader(self.images[index])
+        target = self.labels[index]
+
+        if self.transform is not None:
+            img = self.transform(img)
+
+        if self.target_transform is not None:
+            target = self.target_transform(target)
+
+        return img, target
+
+    def __len__(self):
+        return len(self.images)
+
+
+def get_downstream_dataset(catalog, name, is_train, transform):
+    entry = catalog[name]
+    root = entry['path']
+    if entry['type'] == 'imagefolder':
+        dataset = t_datasets.ImageFolder(os.path.join(root, entry['train'] if is_train else entry['test']),
+                                         transform=transform)
+    elif entry['type'] == 'special':
+        if name == 'CIFAR10':
+            dataset = t_datasets.CIFAR10(root, train=is_train,
+                                         transform=transform, download=True)
+        elif name == 'CIFAR100':
+            dataset = t_datasets.CIFAR100(root, train=is_train,
+                                          transform=transform, download=True)
+        elif name == 'STL10':
+            dataset = t_datasets.STL10(root, split='train' if is_train else 'test',
+                                       transform=transform, download=True)
+        elif name == 'MNIST':
+            dataset = t_datasets.MNIST(root, train=is_train,
+                                       transform=transform, download=True)
+        elif name == 'chexpert-5x200':
+            dataset = ZeroShotImageDataset(['chexpert-5x200'], CHEXPERT_COMPETITION_TASKS, transform
+                                           , parent_data_path=root)
+        elif name == "radimagenet":
+            dataset = RadImageNet(root, transform)
+        elif name == "rsna_pneumonia":
+            dataset = RSNA_Pneumonia(root, transform)
+        elif name == "thyroid_us":
+            dataset = thyroid_us_and_breast(root, transform, "thyroid_test_fold1.csv")
+        elif name == "breast_us":
+            dataset = thyroid_us_and_breast(root, transform, "breast_test_fold1.csv")
+        elif name == "meniscal_mri":
+            dataset = meniscal_mri(root, transform, "meniscus_test_fold1.csv")
+        elif name == 'acl_mri':
+            dataset = acl_mri(root, transform, "test_fold1.csv")
+        elif name == 'CT_axial':
+            dataset = CT_dataset(root, transform, "organs_axial")
+        elif name == 'CT_coronal':
+            dataset = CT_dataset(root, transform, "organs_coronal")
+        elif name == 'CT_sagittal':
+            dataset = CT_dataset(root, transform, "organs_sagittal")
+        elif name == 'dr_regular':
+            dataset = CT_dataset(root, transform, "dr_regular")
+        elif name == 'dr_uwf':
+            dataset = CT_dataset(root, transform, "dr_uwf")
+        elif name == 'LC25000_lung':
+            dataset = LC25000(root, transform, "lung")
+        elif name == 'LC25000_colon':
+            dataset = LC25000(root, transform, "colon")
+        elif name == 'PCAM':
+            dataset = PCAM(root, transform, "PCam_Test_preprocessed")
+        elif name == 'NCK_CRC':
+            dataset = NCK_CRC(root, transform, "CRC-VAL-HE-7K")
+        elif name == 'BACH':
+            dataset = BACH(root, transform, "BACH")
+        elif name == 'Osteo':
+            dataset = Osteo(root, transform, "Osteosarcoma")
+        elif name == 'skin_cancer':
+            dataset = Skin_datasets(root, transform, "skin_tumor", 'cancer')
+        elif name == 'skin_tumor':
+            dataset = Skin_datasets(root, transform, "skin_tumor", 'tumor')
+        elif name == 'refuge_retina':
+            dataset = Retina_datasets(root, transform, '25_REFUGE.csv')
+        elif name == 'five_retina':
+            dataset = Retina_datasets(root, transform, '13_FIVES.csv')
+        elif name == 'odir_retina':
+            dataset = Retina_datasets(root, transform, '08_ODIR200x3.csv')
+
+    elif entry['type'] == 'filelist':
+        path = entry['train'] if is_train else entry['test']
+        val_images = os.path.join(root, path + '_images.npy')
+        val_labels = os.path.join(root, path + '_labels.npy')
+        if name == 'CLEVRCounts':
+            target_transform = lambda x: ['count_10', 'count_3', 'count_4', 'count_5', 'count_6', 'count_7', 'count_8',
+                                          'count_9'].index(x)
+        else:
+            target_transform = None
+        dataset = FileListDataset(val_images, val_labels, transform, target_transform)
+    else:
+        raise Exception('Unknown dataset')
+
+    return dataset
+
+
+class ZeroShotImageDataset(torch.utils.data.Dataset):
+    def __init__(self,
+                 datalist=['chexpert-5x200'],
+                 class_names=None,
+                 imgtransform=None,
+                 parent_data_path="",
+                 ) -> None:
+        '''support data list in mimic-5x200, chexpert-5x200, rsna-balanced-test, covid-test
+        args:
+            imgtransform: a torchvision transform
+            cls_prompts: a dict of prompt sentences, cls:[sent1, sent2, ..],
+        '''
+        super().__init__()
+
+        self.transform = imgtransform
+
+        self.class_names = class_names
+        self.parent_data_path = parent_data_path
+        # imgpath, subject_id, report, labels...(14 labels)
+        df_list = []
+        for data in datalist:
+            filename = f'./local_data/{data}.csv'
+            print('load data from', filename)
+            df = pd.read_csv(filename, index_col=0)
+            df_list.append(df)
+        self.df = pd.concat(df_list, axis=0).reset_index(drop=True)
+
+    def __getitem__(self, index):
+        row = self.df.iloc[index]
+        img = Image.open(os.path.join(self.parent_data_path, row.imgpath))
+        # img = self._pad_img(img)
+        img = self.transform(img)
+        label = torch.from_numpy(row[self.class_names].values.astype(np.float_))
+        return img, label
+
+    def _pad_img(self, img, min_size=224, fill_color=0):
+        '''pad img to square.
+        '''
+        x, y = img.size
+        size = max(min_size, x, y)
+        new_im = Image.new('L', (size, size), fill_color)
+        new_im.paste(img, (int((size - x) / 2), int((size - y) / 2)))
+        return new_im
+
+    def __len__(self):
+        return len(self.df)
+
+
+class RadImageNet(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        self.data = pd.read_csv(os.path.join(parent_path, "radimagenet_test_set_formatted.csv"))
+        self.transform = transform
+        self.parent_data_path = parent_path
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 0]
+        img_name = ast.literal_eval(img_name)[0]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+        label = self.data.iloc[idx, 1]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class CT_dataset(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None, foldername=None):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        all_data = pd.read_csv(os.path.join(os.path.join(parent_path, foldername),
+                                            "annotations.csv"))
+        # Filter the data to only retain the test samples
+        self.data = all_data[all_data['split'] == 'test']
+        self.transform = transform
+        self.parent_data_path = os.path.join(parent_path, foldername)
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 0]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+        label = self.data.iloc[idx, 2]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class LC25000(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None, split=None):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        if split == "lung":
+            classes = ['lung_aca', 'lung_n', 'lung_scc']
+        else:
+            classes = ['colon_aca', 'colon_n']
+        self.split = split
+        self.class_name_folder = []
+        self.images = []
+        self.labels = []
+        for idx, single_class_folder in enumerate(classes):
+            images_per_classes = list(os.listdir(os.path.join(parent_path, single_class_folder)))
+            self.images = self.images + images_per_classes
+            self.labels = self.labels + ([idx] * len(images_per_classes))
+            self.class_name_folder = self.class_name_folder + ([single_class_folder] * len(images_per_classes))
+        self.transform = transform
+        self.parent_data_path = parent_path
+        assert len(self.images) == len(self.labels) == len(self.class_name_folder)
+
+    def __len__(self):
+        return len(self.images)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.images[idx]
+        class_folder_name = self.class_name_folder[idx]
+        img = Image.open(os.path.join(os.path.join(self.parent_data_path, class_folder_name), img_name))
+        label = self.labels[idx]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class PCAM(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None, foldername=None):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        all_files = os.listdir(os.path.join(parent_path, foldername))
+        # Filter the data to only retain the test samples
+        self.data = all_files
+        # Create labels
+        labels = []
+        for single_file in all_files:
+            splitted_label = int(single_file.split("_")[1].split(".")[0])
+            labels.append(splitted_label)
+        self.labels = labels
+        self.transform = transform
+        self.parent_data_path = os.path.join(parent_path, foldername)
+        assert len(self.labels) == len(self.data)
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data[idx]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+        label = self.labels[idx]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class NCK_CRC(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None, foldername=None):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        NCK_CRC_converter = {"ADI": 0,
+                             "DEB": 1,
+                             "LYM": 2,
+                             "MUC": 3,
+                             "MUS": 4,
+                             "NORM": 5,
+                             "STR": 6,
+                             "TUM": 7,
+                             }
+        all_data = []
+        all_class_names = []
+        all_labels = []
+        folder_names = os.listdir(os.path.join(parent_path, foldername))
+        for single_folder in folder_names:
+            class_path = os.path.join(os.path.join(parent_path, foldername), single_folder)
+            images_inside_folder = os.listdir(class_path)
+            class_label = [NCK_CRC_converter[single_folder]] * len(images_inside_folder)
+            all_data.extend(images_inside_folder)
+            all_labels.extend(class_label)
+            all_class_names.extend([single_folder] * len(images_inside_folder))
+        # Filter the data to only retain the test samples
+        self.data = all_data
+        self.labels = all_labels
+        self.prefix_name = all_class_names
+        assert len(self.data) == len(self.labels) == len(self.prefix_name)
+        self.transform = transform
+        self.parent_data_path = os.path.join(parent_path, foldername)
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data[idx]
+        class_name = self.prefix_name[idx]
+        label = self.labels[idx]
+        img = Image.open(os.path.join(self.parent_data_path, os.path.join(class_name, img_name)))
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class BACH(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None, foldername=None):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        all_data = pd.read_csv(os.path.join(os.path.join(parent_path, foldername),
+                                            "microscopy_ground_truth.csv"))
+
+        self.data = all_data
+        self.transform = transform
+        self.parent_data_path = os.path.join(parent_path, foldername)
+        self.label_to_text_mapping = {'Normal': 3, 'Invasive': 2, 'InSitu': 1, "Benign": 0}
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 0]
+        label_text = self.data.iloc[idx, 1]
+        img = Image.open(os.path.join(self.parent_data_path, label_text + "/" + img_name))
+
+        label = self.label_to_text_mapping[label_text]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class Retina_datasets(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None, data=None):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        filename = f'./local_data/{data}'
+        all_data = pd.read_csv(filename)
+
+        self.data = all_data
+        self.transform = transform
+        self.parent_data_path = parent_path
+
+        if data == '25_REFUGE.csv':
+            self.label_to_text_mapping = {'no glaucoma': 0, 'glaucoma': 1}
+        elif data == '13_FIVES.csv':
+            self.label_to_text_mapping = {"age related macular degeneration": 0,
+                                          "diabetic retinopathy": 1,
+                                          "glaucoma": 2,
+                                          "normal": 3}
+        elif data == '08_ODIR200x3.csv':
+            self.label_to_text_mapping = {"normal": 0,
+                                          "pathologic myopia": 1,
+                                          "cataract": 2}
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 1]
+        label_text = self.data.iloc[idx, 3]  # it is a list
+        label_text = ast.literal_eval(label_text)[0]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+
+        label = self.label_to_text_mapping[label_text]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+class Skin_datasets(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None, foldername=None, split_type='cancer'):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        if split_type == 'cancer':
+            all_data = pd.read_csv(os.path.join(os.path.join(parent_path, foldername),
+                                                "data/tiles-v2.csv"))
+            # Filter the dataset and take only test samples...
+            all_data = all_data[all_data['set'] == 'Test']
+            self.label_to_text_mapping = {
+                "nontumor_skin_necrosis_necrosis": 0,
+
+                "nontumor_skin_muscle_skeletal":
+                    1,
+
+                "nontumor_skin_sweatglands_sweatglands":
+                    2,
+
+                "nontumor_skin_vessel_vessel":
+                    3,
+
+                "nontumor_skin_elastosis_elastosis":
+                    4,
+
+                "nontumor_skin_chondraltissue_chondraltissue":
+                    5,
+
+                "nontumor_skin_hairfollicle_hairfollicle":
+                    6,
+                "nontumor_skin_epidermis_epidermis": 7,
+                "nontumor_skin_nerves_nerves":
+                    8,
+
+                "nontumor_skin_subcutis_subcutis":
+                    9,
+
+                "nontumor_skin_dermis_dermis":
+                    10,
+
+                "nontumor_skin_sebaceousglands_sebaceousglands":
+                    11,
+
+                "tumor_skin_epithelial_sqcc":
+                    12,
+
+                "tumor_skin_melanoma_melanoma":
+                    13,
+
+                "tumor_skin_epithelial_bcc":
+                    14,
+
+                "tumor_skin_naevus_naevus":
+                    15
+            }
+        else:
+            all_data = pd.read_csv(os.path.join(os.path.join(parent_path, foldername),
+                                                "data/SkinTumorSubset.csv"))
+            # Filter the dataset and take only test samples...
+            all_data = all_data[all_data['set'] == 'Test']
+            self.label_to_text_mapping = {"tumor_skin_epithelial_sqcc":
+                                              0,
+
+                                          "tumor_skin_melanoma_melanoma":
+                                              1,
+
+                                          "tumor_skin_epithelial_bcc":
+                                              2,
+
+                                          "tumor_skin_naevus_naevus":
+                                              3
+                                          }
+
+        self.data = all_data
+        self.transform = transform
+        self.parent_data_path = os.path.join(parent_path, foldername)
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 1]
+        label_text = self.data.iloc[idx, 2]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+
+        label = self.label_to_text_mapping[label_text]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class Osteo(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None, foldername=None):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+
+        all_data = pd.read_csv(os.path.join(os.path.join(parent_path, foldername),
+                                            "annotations_final.csv"))
+
+        self.data = all_data
+        self.transform = transform
+        self.parent_data_path = os.path.join(parent_path, foldername)
+        self.label_to_text_mapping = {'Viable': 2, 'Non-Tumor': 0, "Non-Viable-Tumor": 1}
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 0]
+        label_text = self.data.iloc[idx, 1]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+
+        label = self.label_to_text_mapping[label_text]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class RSNA_Pneumonia(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform=None):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        self.data = pd.read_csv(os.path.join(parent_path, "RSNA_pneumonia_balanced_testfile.csv"))
+        self.transform = transform
+        self.parent_data_path = parent_path
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 1]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+        label = self.data.iloc[idx, 2]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class thyroid_us_and_breast(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform, csv_file_name):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        self.data = pd.read_csv(os.path.join(parent_path, csv_file_name))
+        self.transform = transform
+        self.parent_data_path = parent_path
+        # self.mapping = {'malignant': 0, 'benign': 1}
+        self.mapping = {'malignant': 1, 'benign': 0}
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 0]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+        label_name = self.data.iloc[idx, 1]
+        label = self.mapping[label_name]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class meniscal_mri(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform, csv_file_name):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        self.data = pd.read_csv(os.path.join(parent_path, csv_file_name))
+        self.transform = transform
+        self.parent_data_path = parent_path
+        self.mapping = {'p': 1, 'n': 0}
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 0]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+        label_name = self.data.iloc[idx, 1]
+        label = self.mapping[label_name]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label
+
+
+class acl_mri(torch.utils.data.Dataset):
+    def __init__(self, parent_path, transform, csv_file_name):
+        """
+        Args:
+            csv_file (string): Path to the CSV file containing image paths and labels.
+            transform (callable, optional): Optional transform to be applied
+                on a sample.
+        """
+        self.data = pd.read_csv(os.path.join(parent_path, csv_file_name))
+        self.transform = transform
+        self.parent_data_path = parent_path
+        self.mapping = {'yes': 1, 'no': 0}
+
+    def __len__(self):
+        return len(self.data)
+
+    def __getitem__(self, idx):
+        if torch.is_tensor(idx):
+            idx = idx.tolist()
+
+        img_name = self.data.iloc[idx, 0]
+        img = Image.open(os.path.join(self.parent_data_path, img_name))
+        label_name = self.data.iloc[idx, 1]
+        label = self.mapping[label_name]
+
+        if self.transform:
+            image = self.transform(img)
+
+        return image, label

clipeval/eval_zeroshot.py

@@ -0,0 +1,359 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+
+# This source code is licensed under the license found in the
+# LICENSE file in the root directory of this source tree.
+
+"""refactored from `main` in `eval_zeroshot.py` (SLIP) for clarity.
+"""
+import random
+import torch
+import json
+import os
+from tqdm import tqdm
+from sklearn import metrics
+from constants import RSNA_CLASS_PROMPTS_webdataset, modality_indices_radimagenet_test_set
+from collections import defaultdict
+
+
+def load_metadata(metadir="clipeval"):
+    with open(os.path.join(metadir, 'dataset_catalog.json')) as f:
+        catalog = json.load(f)
+
+    with open(os.path.join(metadir, 'templates.json')) as f:
+        all_templates = json.load(f)
+
+    with open(os.path.join(metadir, 'labels.json')) as f:
+        all_labels = json.load(f)
+    return catalog, all_templates, all_labels
+
+
+def evaluate(d, val_loader, templates, labels, model, tokenizer, classnorm=False):
+    print('Evaluating {}'.format(d))
+
+    is_acc = d not in ['FGVCAircraft', 'OxfordPets', 'Caltech101', 'Flowers102', 'Kinetics700', 'HatefulMemes']
+    if d == 'radimagenet':
+        acc, us_acc, mri_acc, ct_acc = validate_zeroshot(val_loader, templates, labels, model, tokenizer,
+                                                         is_acc, d, classnorm)
+    else:
+        acc_or_outputs = validate_zeroshot(val_loader, templates, labels, model, tokenizer, is_acc, d, classnorm)
+    if d in ['FGVCAircraft', 'OxfordPets', 'Caltech101', 'Flowers102']:
+        metric = mean_per_class(*acc_or_outputs)
+    elif d == 'Kinetics700':
+        top1, top5 = accuracy(*acc_or_outputs, topk=(1, 5))
+        metric = (top1 + top5) / 2
+        metric = metric.item()
+    elif d == 'HatefulMemes':
+        metric = roc_auc(*acc_or_outputs)
+    elif d == 'radimagenet':
+        metric = {"acc": acc, "US acc": us_acc, "MRI acc": mri_acc, "CT acc": ct_acc}
+    else:
+        metric = acc_or_outputs
+
+    return metric
+
+
+@torch.no_grad()
+def build_text_features(templates, labels, model, tokenizer, skip_text_projection=False, classnorm=False):
+    # TODO: add device
+    text_features = []
+    if type(templates) == dict:
+        class_similarities = []
+        class_names = []
+        for cls_name, cls_text in templates.items():
+            texts = tokenizer(cls_text).to(next(model.parameters()).device, non_blocking=True)
+            class_embeddings = model.encode_text(texts)
+            class_embeddings = class_embeddings / class_embeddings.norm(dim=-1, keepdim=True)
+
+            if True:
+                cls_sim = class_embeddings.mean(dim=0)  # equivalent to prompt ensembling
+            else:
+                cls_sim = class_embeddings[0]
+            class_similarities.append(cls_sim)
+            class_names.append(cls_name)
+
+        text_features = torch.stack(class_similarities, dim=0)
+    elif type(templates) == list and templates[0] == "Meniscal abnormality detected in MRI imaging of the knee.":
+        print("Encoding captions for RadImageNet dataset")
+        for single_template in templates:
+            texts = tokenizer(single_template).to(next(model.parameters()).device, non_blocking=True)
+            class_embeddings = model.encode_text(texts)
+            class_embeddings = class_embeddings / class_embeddings.norm(dim=-1, keepdim=True)
+            text_features.append(class_embeddings)
+        text_features = torch.stack(text_features, dim=0).squeeze(1)
+    else:
+        for label in labels:
+            if isinstance(label, list):
+                texts = [t.format(l) for t in templates for l in label]
+            else:
+                texts = [t.format(label) for t in templates]
+
+            texts = tokenizer(texts).to(next(model.parameters()).device, non_blocking=True)
+            class_embeddings = model.encode_text(texts)
+            class_embeddings = class_embeddings / class_embeddings.norm(dim=-1, keepdim=True)
+            class_embeddings = class_embeddings.mean(dim=0)
+            text_features.append(class_embeddings)
+        text_features = torch.stack(text_features, dim=0)
+    mean, std = None, None
+    if classnorm:
+        mean, std = text_features.mean(dim=0)[None, :], text_features.std(dim=0)[None, :]
+        text_features = (text_features - mean) / std
+    text_features = text_features / text_features.norm(dim=-1, keepdim=True)
+    return text_features, mean, std
+
+
+def generate_chexpert_class_prompts(class_prompts, n=None):
+    """Generate text prompts for each CheXpert classification task
+    Parameters
+    ----------
+    n:  int
+        number of prompts per class
+    Returns
+    -------
+    class prompts : dict
+        dictionary of class to prompts
+    """
+
+    prompts = {}
+    for k, v in class_prompts.items():
+        cls_prompts = []
+        keys = list(v.keys())
+
+        # severity
+        for k0 in v[keys[0]]:
+            # subtype
+            for k1 in v[keys[1]]:
+                # location
+                for k2 in v[keys[2]]:
+                    cls_prompts.append(f"{k0} {k1} {k2}")
+
+        # randomly sample n prompts for zero-shot classification
+        # TODO: we shall make use all the candidate prompts for autoprompt tuning
+        if n is not None and n < len(cls_prompts):
+            prompts[k] = random.sample(cls_prompts, n)
+        else:
+            prompts[k] = cls_prompts
+        print(f'sample {len(prompts[k])} num of prompts for {k} from total {len(cls_prompts)}')
+    return prompts
+
+
+def generate_rsna_class_prompts(class_prompts, n=None):
+    prompts = {}
+    for k, v in class_prompts.items():
+        cls_prompts = []
+        keys = list(v.keys())
+
+        for k0 in v[keys[0]]:
+            for k1 in v[keys[1]]:
+                for k2 in v[keys[2]]:
+                    cls_prompts.append(f"{k0} {k1} {k2}")
+
+        # randomly sample n prompts for zero-shot classification
+        if n is not None and n < len(cls_prompts):
+            prompts[k] = random.sample(cls_prompts, n)
+        else:
+            prompts[k] = cls_prompts
+        print(f'sample {len(prompts[k])} num of prompts for {k} from total {len(cls_prompts)}')
+    return prompts
+
+
+@torch.no_grad()
+def validate_zeroshot(val_loader, templates, labels, model, tokenizer, is_acc, name, classnorm=False):
+    # switch to evaluate mode
+    model.cuda()
+    model.eval()
+
+    total_top1 = 0
+    total_images = 0
+
+    all_outputs = []
+    all_targets = []
+
+    text_features = None
+    # Initialize per-class accuracy variables
+    class_correct = defaultdict(int)
+    class_total = defaultdict(int)
+
+    for samples in tqdm(val_loader):
+        # Below if will run only for one iteration
+        if text_features is None:
+            print('=> encoding captions')
+            if name == "chexpert-5x200":
+                if not type(templates[list(templates.keys())[0]]) == list:
+                    prompted_templates = generate_chexpert_class_prompts(templates, 10)  # 10 prompts per class
+                else:
+                    k = 11  # This means all 10 templates to be used...
+                    print(f"Using {k - 1} templates for the ensembling at test time")
+                    for single_key in templates.keys():
+                        templates[single_key] = templates[single_key][0:k]
+                    prompted_templates = templates
+                text_features, mean, std = build_text_features(prompted_templates, None, model, tokenizer,
+                                                               classnorm=classnorm)
+            elif name == "rsna_pneumonia":
+                if not type(templates[list(templates.keys())[0]]) == list:
+                    temp = generate_rsna_class_prompts(templates, 10)  # 10 prompts per class
+                    # For the case of Pneumonia, we also need second template for normal as well
+                    prompted_templates = {'normal': RSNA_CLASS_PROMPTS_webdataset['Normal'],
+                                          'pneumonia': temp['Pneumonia']}
+                else:
+                    k = 1  # This means all 10 templates to be used...
+                    print(f"Using {k - 1} templates for the ensembling at test time")
+                    for single_key in templates.keys():
+                        templates[single_key] = templates[single_key][0:k]
+                    prompted_templates = templates
+                text_features, mean, std = build_text_features(prompted_templates, None, model, tokenizer,
+                                                               classnorm=classnorm)
+            else:
+                if type(templates) == dict:
+                    k = 11  # This means all 10 templates to be used...
+                    print(f"Using {k - 1} templates for the ensembling at test time")
+                    for single_key in templates.keys():
+                        length = len(templates[single_key])
+                        templates[single_key] = templates[single_key][0:length]
+                    prompted_templates = templates
+                else:
+                    prompted_templates = templates
+                text_features, mean, std = build_text_features(prompted_templates, labels, model, tokenizer,
+                                                               classnorm=classnorm)
+        if isinstance(samples, tuple) or isinstance(samples, list):
+            images, target = samples[0], samples[1]
+        elif isinstance(samples, dict):
+            images, target = samples["pixel_values"], samples["targets"]
+        else:
+            raise ValueError("unknown sample type", type(samples))
+
+        images = images.cuda(non_blocking=True)
+        target = target.cuda(non_blocking=True)
+
+        # encode images
+        image_features = model.encode_image(images)
+
+        if classnorm:
+            image_features = (image_features - mean) / std
+        print("no normalizing this time)")
+        # image_features = image_features / image_features.norm(dim=-1, keepdim=True)
+        # cosine similarity as logits
+        logits_per_image = image_features @ text_features.t()
+        logits_per_image = logits_per_image.cpu()
+        target = target.cpu()
+        if name == "chexpert-5x200":
+            # convert to label encoding
+            target = torch.argmax(target, axis=1)
+        if is_acc:
+            # measure accuracy and record loss
+            pred = logits_per_image.argmax(dim=1)
+            correct = pred.eq(target).sum()
+            total_top1 += correct.item()
+            total_images += images.size(0)
+            if name == "radimagenet":
+                # Update per-class accuracy counts
+                for t, p in zip(target, pred):
+                    class_correct[t.item()] += p.eq(t).item()
+                    class_total[t.item()] += 1
+            # Also save those to have results for the other metrics
+            all_outputs.append(logits_per_image)
+            all_targets.append(target)
+        else:
+            all_outputs.append(logits_per_image)
+            all_targets.append(target)
+
+    if is_acc:
+        if name == "radimagenet":
+            # Now calculate accuracies for each modality
+            US_all_class_correct = 0
+            MRI_all_class_correct = 0
+            CT_all_class_correct = 0
+            US_all_class_total = 0
+            MRI_all_class_total = 0
+            CT_all_class_total = 0
+            for single_us_index in modality_indices_radimagenet_test_set['US']:
+                US_all_class_correct += class_correct[single_us_index]
+                US_all_class_total += class_total[single_us_index]
+            for single_mri_index in modality_indices_radimagenet_test_set['MRI']:
+                MRI_all_class_correct += class_correct[single_mri_index]
+                MRI_all_class_total += class_total[single_mri_index]
+            for single_ct_index in modality_indices_radimagenet_test_set['CT']:
+                CT_all_class_correct += class_correct[single_ct_index]
+                CT_all_class_total += class_total[single_ct_index]
+
+            return 100 * total_top1 / total_images, \
+                   100 * US_all_class_correct / US_all_class_total, \
+                   100 * MRI_all_class_correct / MRI_all_class_total, \
+                   100 * CT_all_class_correct / CT_all_class_total
+        if name == 'radimagenet' or name == 'chexpert-5x200' or name == 'CT_sagittal' or name == 'CT_axial' \
+                or name == 'CT_coronal' or name == 'dr_uwf' or name == 'dr_regular' \
+                or name == 'PCAM' or name == 'LC25000_lung' or name == 'LC25000_colon' \
+                or name == "NCK_CRC" or name == 'BACH' or name == 'Osteo' \
+                or name == 'skin_cancer' or name == 'skin_tumor' or name == 'SICAPv2' \
+                or name == 'five_retina' or name == 'odir_retina':
+            return 100 * total_top1 / total_images
+        else:
+            # Now also return the other metric results
+            all_outputs = torch.cat(all_outputs)
+            all_targets = torch.cat(all_targets)
+            accuracy = 100 * total_top1 / total_images
+            auc_roc = roc_auc(all_outputs, all_targets)
+            f1_score = F1_score(all_outputs, all_targets)
+            precision_score = Precision_score(all_outputs, all_targets)
+            recall_score = Recall_score(all_outputs, all_targets)
+            return {"acc": accuracy, "auc_roc": auc_roc, "f1_score": f1_score,
+                    "precision_score": precision_score, "recall_score": recall_score}
+    else:
+        return torch.cat(all_outputs), torch.cat(all_targets)
+
+
+def accuracy(output, target, topk=(1,)):
+    """Computes the accuracy over the k top predictions for the specified values of k"""
+    with torch.no_grad():
+        maxk = max(topk)
+        batch_size = target.size(0)
+
+        _, pred = output.topk(maxk, 1, True, True)
+        pred = pred.t()
+        correct = pred.eq(target.reshape(1, -1).expand_as(pred))
+
+        res = []
+        for k in topk:
+            correct_k = correct[:k].reshape(-1).float().sum(0, keepdim=True)
+            res.append(correct_k.mul_(100.0 / batch_size))
+        return res
+
+
+def Recall_score(outputs, targets):
+    pred = outputs.argmax(1)
+    Recall_score = metrics.recall_score(targets, pred)
+    return 100 * Recall_score
+
+
+def F1_score(outputs, targets):
+    pred = outputs.argmax(1)
+    F1_score = metrics.f1_score(targets, pred)
+    return 100 * F1_score
+
+
+def Precision_score(outputs, targets):
+    pred = outputs.argmax(1)
+    Precision_score = metrics.precision_score(targets, pred)
+    return 100 * Precision_score
+
+
+def mean_per_class(outputs, targets):
+    pred = outputs.argmax(1)
+    confusion_matrix = metrics.confusion_matrix(targets, pred)
+    per_classes = confusion_matrix.diagonal() / confusion_matrix.sum(axis=1)
+
+    return 100 * per_classes.mean()
+
+
+def roc_auc(outputs, targets):
+    pos_score = outputs[:, 1] - outputs[:, 0]
+    metric = metrics.roc_auc_score(targets, pos_score)
+
+    return 100 * metric
+
+
+if __name__ == '__main__':
+    logits = torch.randn(128, 10)
+    targets = torch.randint(size=(128,), low=0, high=10)
+
+    evaluate("imagenet", logits, targets)

clipeval/radimagenet_labels.txt

@@ -0,0 +1,165 @@
+Meniscal abnormality detected in MRI imaging of the knee.
+Chondral abnormality evident in the MRI image of the knee.
+MRI scan showing pathology of the medial collateral ligament (MCL) in the knee.
+Soft tissue fluid collection evident in the MRI of the knee.
+Fractured knee depicted in MRI imaging.
+MRI imaging depicts inflammation of the bone in the knee region.
+ACL pathology is evident in the MRI imaging of the knee..
+Pathological condition of the patella evident in the MRI image of the knee.
+MRI depicting intra-articular pathology of the knee.
+Quad pathology evident in the MRI of the knee.
+Post-operative ACL MRI scan of the knee.
+MRI imaging revealing edema within the soft tissues of the knee.
+MRI image demonstrating pathology of the posterior cruciate ligament (PCL) in the knee.
+MRI image displaying pathology of the fibular collateral ligament in the knee.
+MRI image depicts a hematoma in the knee region..
+Muscle strain is evident in the MRI image of the knee..
+Soft tissue mass identified in the knee on MRI imaging.
+Labral pathology identified in an MRI image of the shoulder..
+Supraspinatus pathology evident in MRI of the shoulder.
+Soft tissue fluid evident in an MRI image of the shoulder.
+Osteoarthritis visibly characterized in the shoulder joint on MRI imaging.
+Acromioclavicular joint osteoarthritis evident in MRI of the shoulder.
+Osseous lesion identified in the shoulder from MRI imaging.
+Subscapularis pathology depicted in MRI imaging of the shoulder.
+Postoperative MRI scan of the shoulder.
+Inflammatory changes noted within the marrow of the shoulder on MRI imaging.
+MRI image showing edema within the soft tissues of the shoulder.
+Infraspinatus pathology is evident in the MRI of the shoulder.
+MRI image depicting calcific tendinosis in the shoulder region.
+MRI scan demonstrating pathology of the biceps in the shoulder region.
+Chondral pathology evident in the MRI image of the hip.
+MRI image reveals labral pathology in the hip region.
+MRI image showing inflammation in the bone marrow of the hip.
+Fluid accumulation in soft tissues observed in the hip MRI image.
+Post-operative MRI scan of the hip demonstrating surgical changes.
+MRI image depicting a soft tissue mass in the hip region..
+Osseous disruption evident in the MRI image of the hip.
+Osseous lesion identified in the hip region on MRI imaging.
+Abductor pathology is evident in the MRI image of the hip.
+Soft tissue edema evident in MRI of the hip.
+Hematoma evident in the hip region on MRI imaging.
+Hamstring pathology is evident in the MRI image of the hip.
+Capsular pathology is evident in the MRI image of the hip..
+Cord pathology evident in MRI imaging of the spine.
+MRI reveals the presence of a dural epidural abnormality in the spine.
+Foraminal pathology evident in the MRI of the spine.
+Disc pathology evident in MRI imaging of the spine.
+Facet arthropathy is displayed in this MRI image of the spine..
+Cystic lesions identified on MRI imaging of the spine.
+Osseous abnormalities detected in the MRI of the spine.
+Image depicts scoliosis on an MRI of the spine.
+Chronic infarct evidenced in brain MRI imaging..
+White matter alterations observable on brain MRI imaging.
+Focal FLAIR hyperintensity observed in the brain MRI image.
+Arteriovenous anomaly depicted in the MRI of the brain.
+MRI imaging reveals cerebral edema.
+Acute infarct displayed in the MRI image of the brain.
+MRI image reveals extraneous tissue within the brain.
+MRI image showing intracranial abnormality in the brain.
+Pituitary lesion depicted in MRI imaging of the brain.
+MRI image showing signs of hepatic pathology.
+Adrenal pathology is evident in the MRI photo of the abdomen..
+Arterial pathology depicted in the MRI image of the abdomen.
+MRI image showing ascites in the abdominal cavity.
+MRI image showing ovarian pathology in the abdomen.
+Intraperitoneal mass evident in MRI abdominal imaging.
+MRI scan showing biliary dilatation in the abdomen.
+MRI imaging reveals a lesion on the liver.
+Bladder pathology depicted in MRI abdomen imaging.
+Abnormality in bowel structure as depicted in an MRI of the abdomen.
+MRI image depicting inflammation of the bowel.
+MRI image displaying a mass in the abdominal bowel region..
+Degenerative changes are evident in the MRI of the abdomen.
+MRI image showing dilation of the urinary tract.
+MRI image reveals organomegaly in the abdominal region.
+MRI image depicting gallbladder pathology in the abdominal region.
+Renal lesion is evident in the MRI of the abdomen.
+MRI photo reveals a uterine pathology in the abdominal scan.
+MRI depicts abnormal marrow in the abdomen..
+Osseous neoplasm identified in MRI of the abdomen.
+MRI image depicting a lesion in the pancreas.
+Prostate lesion depicted in MRI abdominal imaging..
+Soft tissue collection identified in MRI of the abdomen.
+Soft tissue mass observed in abdominal MRI imaging.
+Splenic lesion evident on MRI abdominal image.
+Gallstone identified on abdominal CT scan.
+Prostate lesion identified in CT imaging of the abdomen.
+The entire organ shows abnormalities on the CT scan of the abdomen..
+Pancreatic lesion identified using CT imaging of the abdomen.
+CT imaging reveals biliary dilation involving the abdomen.
+CT imaging shows dilation of the urinary tract..
+Urolithiasis identified in the abdominal CT scan.
+A computed tomography scan revealing an abnormality in the abdominal bowel.
+Bowel inflammation depicted on abdominal CT imaging.
+Osseous neoplasm identified in abdominal CT imaging.
+Liver lesion identified in CT scan of the abdomen.
+Renal lesion identified on abdominal CT scan..
+Post-operative changes observed in abdominal CT imaging.
+CT imaging reveals pathology of the uterus..
+Adrenal pathology identified on computed tomography of the abdomen.
+Ovarian pathology is evident in the CT scan of the abdomen..
+Arterial pathology identified in the abdominal CT scan.
+Ascites evidenced in CT imaging of the abdomen.
+Splenic lesion identified on the CT scan of the abdomen.
+Bladder pathology identified in CT imaging of the abdomen.
+A fat-containing tumor is depicted in the abdominal CT imaging..
+Degenerative changes identified in the abdomen on CT imaging.
+Soft tissue collection identified in the abdominal CT imaging..
+Gallbladder pathology identified on abdominal CT scan.
+Soft tissue mass identified on abdominal CT scan.
+An intraperitoneal mass is identified on the CT scan of the abdomen.
+A bowel mass is identified in the abdominal CT imaging..
+MRI image depicting an osseous disruption in the atrial fibrillation (AF) region..
+Bone inflammation depicted in MRI imaging of the forearm.
+Fluid accumulation in soft tissues evidenced in the MRI image.
+Osseous neoplasm depicted in MRI imaging of the foot.
+Chondral abnormality depicted in MRI imaging of the atrial fibrillation affected area.
+Plantar fascia pathology evident in MRI imaging.
+ATFL pathology depicted in MRI imaging of the ankle.
+MRI image depicting soft tissue edomain the ankle region..
+MRI image demonstrating a soft tissue mass in the atrial fibrillation area.
+Intra-atrial fibrillation observed in an MRI imaging.
+MRI image reveals a pathology in the extensor compartment of the forearm..
+Plantar plate tear evident in MRI imaging of the foot.
+Achilles pathology evident in MRI imaging of the ankle.
+Hematoma presented in an MRI image of the atrial fibrillation affected area.
+CFL pathology evident in the MRI image of the ankle.
+Peroneal pathology evidenced in MRI imaging.
+MRI image depicting pathology of the deltoid muscle.
+Post-operative MRI showing atrial fibrillation characteristics.
+Flexor pathology evident in MRI imaging of the forearm.
+MRI image showing a tumor characterized by fat content in the atrial fibrillation region.
+Pathology of syndesmosis depicted in an MRI image of the ankle..
+MRI image demonstrating Lisfranc pathology in a patient with atrial fibrillation.
+MRI reveals a coalition in the tarsal bones.
+MRI image depicting a spring ligament injury in a patient with atrial fibrillation.
+A CT scan of the abdomen shows no abnormalities..
+Normal abdominal MRI image showing no abnormalities.
+MRI image showing a normal atrial fibrillation pattern..
+MRI image depicts a normal brain structure..
+MRI image displays a normal hip structure.
+Normal appearance of the knee in MRI imaging.
+Normal anatomical structures are evident in this MRI image of the shoulder..
+Normal spinal anatomy depicted in MRI imaging..
+Fibroid identified in ultrasound imaging of the uterus..
+Ultrasound depiction of a kidney showing the renal structure..
+Ultrasound image depicting a healthy uterus..
+Ultrasound depicting ovarian morphology..
+The ultrasound image depicts pancreatitis involving the pancreas..
+Ultrasound scan showing liver morphology..
+Ultrasound depicting gallbladder pathology..
+Ultrasound imaging reveals common bile duct dilation.
+Ultrasound reveals an enlarged splean indicative of splenomegaly..
+The ultrasound reveals an image of the aorta..
+Ultrasound imaging depicting the Inferior Vena Cava (IVC).
+Bladder inflammation depicted in ultrasound imaging.
+Portal vein thrombosis evident in the ultrasound image of the portal vein..
+Ultrasound imaging reveals the structure of the thyroid gland..
+The CT scan displays normal lung anatomy without pathological findings..
+Airspace opacity is evident in the lung on the CT scan.
+A CT scan image showing a nodule in the lung tissue..
+Interstitial lung disease as depicted in a CT scan of the lung..
+Parenchyma destruction is evident in the lung from the CT scan.
+Bronchiectasis is evident in the CT scan of the lung.
+Ultrasound imaging reveals the structure of the thyroid gland..

clipeval/templates.json

@@ -0,0 +1,353 @@
+{
+    "CUB200": [
+        "a photo of a {}, a type of bird."
+    ],
+    "Birdsnap": [
+        "a photo of a {}, a type of bird."
+    ],
+    "CIFAR10": [
+        "a photo of a {}.",
+        "a blurry photo of a {}.",
+        "a black and white photo of a {}.",
+        "a low contrast photo of a {}.",
+        "a high contrast photo of a {}.",
+        "a bad photo of a {}.",
+        "a good photo of a {}.",
+        "a photo of a small {}.",
+        "a photo of a big {}.",
+        "a photo of the {}.",
+        "a blurry photo of the {}.",
+        "a black and white photo of the {}.",
+        "a low contrast photo of the {}.",
+        "a high contrast photo of the {}.",
+        "a bad photo of the {}.",
+        "a good photo of the {}.",
+        "a photo of the small {}.",
+        "a photo of the big {}."
+    ],
+    "CIFAR100": [
+        "a photo of a {}.",
+        "a blurry photo of a {}.",
+        "a black and white photo of a {}.",
+        "a low contrast photo of a {}.",
+        "a high contrast photo of a {}.",
+        "a bad photo of a {}.",
+        "a good photo of a {}.",
+        "a photo of a small {}.",
+        "a photo of a big {}.",
+        "a photo of the {}.",
+        "a blurry photo of the {}.",
+        "a black and white photo of the {}.",
+        "a low contrast photo of the {}.",
+        "a high contrast photo of the {}.",
+        "a bad photo of the {}.",
+        "a good photo of the {}.",
+        "a photo of the small {}.",
+        "a photo of the big {}."
+    ],
+    "CLEVRCounts": [
+        "a photo of {} objects."
+    ],
+    "Caltech101": [
+        "a photo of a {}.",
+        "a painting of a {}.",
+        "a plastic {}.",
+        "a sculpture of a {}.",
+        "a sketch of a {}.",
+        "a tattoo of a {}.",
+        "a toy {}.",
+        "a rendition of a {}.",
+        "a embroidered {}.",
+        "a cartoon {}.",
+        "a {} in a video game.",
+        "a plushie {}.",
+        "a origami {}.",
+        "art of a {}.",
+        "graffiti of a {}.",
+        "a drawing of a {}.",
+        "a doodle of a {}.",
+        "a photo of the {}.",
+        "a painting of the {}.",
+        "the plastic {}.",
+        "a sculpture of the {}.",
+        "a sketch of the {}.",
+        "a tattoo of the {}.",
+        "the toy {}.",
+        "a rendition of the {}.",
+        "the embroidered {}.",
+        "the cartoon {}.",
+        "the {} in a video game.",
+        "the plushie {}.",
+        "the origami {}.",
+        "art of the {}.",
+        "graffiti of the {}.",
+        "a drawing of the {}.",
+        "a doodle of the {}."
+    ],
+    "Country211": [
+        "a photo i took in {}.",
+        "a photo i took while visiting {}.",
+        "a photo from my home country of {}.",
+        "a photo from my visit to {}.",
+        "a photo showing the country of {}."
+    ],
+    "DescribableTextures": [
+        "a photo of a {} texture.",
+        "a photo of a {} pattern.",
+        "a photo of a {} thing.",
+        "a photo of a {} object.",
+        "a photo of the {} texture.",
+        "a photo of the {} pattern.",
+        "a photo of the {} thing.",
+        "a photo of the {} object."
+    ],
+    "EuroSAT": [
+        "a centered satellite photo of {}.",
+        "a centered satellite photo of a {}.",
+        "a centered satellite photo of the {}."
+    ],
+    "FGVCAircraft": [
+        "a photo of a {}, a type of aircraft.",
+        "a photo of the {}, a type of aircraft."
+    ],
+    "FacialEmotionRecognition2013": [
+        "a photo of a {} looking face.",
+        "a photo of a face showing the emotion: {}.",
+        "a photo of a face looking {}.",
+        "a face that looks {}.",
+        "they look {}.",
+        "look at how {} they are."
+    ],
+    "Flowers102": [
+        "a photo of a {}, a type of flower."
+    ],
+    "Food101": [
+        "a photo of {}, a type of food."
+    ],
+    "GTSRB": [
+        "a zoomed in photo of a \"{}\" traffic sign.",
+        "a centered photo of a \"{}\" traffic sign.",
+        "a close up photo of a \"{}\" traffic sign."
+    ],
+    "HatefulMemes": [
+        "a {}."
+    ],
+    "KITTI": [
+        "{}"
+    ],
+    "Kinetics700": [
+        "a photo of {}.",
+        "a photo of a person {}.",
+        "a photo of a person using {}.",
+        "a photo of a person doing {}.",
+        "a photo of a person during {}.",
+        "a photo of a person performing {}.",
+        "a photo of a person practicing {}.",
+        "a video of {}.",
+        "a video of a person {}.",
+        "a video of a person using {}.",
+        "a video of a person doing {}.",
+        "a video of a person during {}.",
+        "a video of a person performing {}.",
+        "a video of a person practicing {}.",
+        "a example of {}.",
+        "a example of a person {}.",
+        "a example of a person using {}.",
+        "a example of a person doing {}.",
+        "a example of a person during {}.",
+        "a example of a person performing {}.",
+        "a example of a person practicing {}.",
+        "a demonstration of {}.",
+        "a demonstration of a person {}.",
+        "a demonstration of a person using {}.",
+        "a demonstration of a person doing {}.",
+        "a demonstration of a person during {}.",
+        "a demonstration of a person performing {}.",
+        "a demonstration of a person practicing {}."
+    ],
+    "MNIST": [
+        "a photo of the number: \"{}\"."
+    ],
+    "OxfordPets": [
+        "a photo of a {}, a type of pet."
+    ],
+    "PascalVOC2007": [
+        "a photo of a {}."
+    ],
+    "PatchCamelyon": [
+        "this is a photo of {}"
+    ],
+    "RESISC45": [
+        "satellite imagery of {}.",
+        "aerial imagery of {}.",
+        "satellite photo of {}.",
+        "aerial photo of {}.",
+        "satellite view of {}.",
+        "aerial view of {}.",
+        "satellite imagery of a {}.",
+        "aerial imagery of a {}.",
+        "satellite photo of a {}.",
+        "aerial photo of a {}.",
+        "satellite view of a {}.",
+        "aerial view of a {}.",
+        "satellite imagery of the {}.",
+        "aerial imagery of the {}.",
+        "satellite photo of the {}.",
+        "aerial photo of the {}.",
+        "satellite view of the {}.",
+        "aerial view of the {}."
+    ],
+    "SST2": [
+        "a {} review of a movie."
+    ],
+    "STL10": [
+        "a photo of a {}.",
+        "a photo of the {}."
+    ],
+    "SUN397": [
+        "a photo of a {}.",
+        "a photo of the {}."
+    ],
+    "StanfordCars": [
+        "a photo of a {}.",
+        "a photo of the {}.",
+        "a photo of my {}.",
+        "i love my {}!",
+        "a photo of my dirty {}.",
+        "a photo of my clean {}.",
+        "a photo of my new {}.",
+        "a photo of my old {}."
+    ],
+    "UCF101": [
+        "a photo of a person {}.",
+        "a video of a person {}.",
+        "a example of a person {}.",
+        "a demonstration of a person {}.",
+        "a photo of the person {}.",
+        "a video of the person {}.",
+        "a example of the person {}.",
+        "a demonstration of the person {}.",
+        "a photo of a person using {}.",
+        "a video of a person using {}.",
+        "a example of a person using {}.",
+        "a demonstration of a person using {}.",
+        "a photo of the person using {}.",
+        "a video of the person using {}.",
+        "a example of the person using {}.",
+        "a demonstration of the person using {}.",
+        "a photo of a person doing {}.",
+        "a video of a person doing {}.",
+        "a example of a person doing {}.",
+        "a demonstration of a person doing {}.",
+        "a photo of the person doing {}.",
+        "a video of the person doing {}.",
+        "a example of the person doing {}.",
+        "a demonstration of the person doing {}.",
+        "a photo of a person during {}.",
+        "a video of a person during {}.",
+        "a example of a person during {}.",
+        "a demonstration of a person during {}.",
+        "a photo of the person during {}.",
+        "a video of the person during {}.",
+        "a example of the person during {}.",
+        "a demonstration of the person during {}.",
+        "a photo of a person performing {}.",
+        "a video of a person performing {}.",
+        "a example of a person performing {}.",
+        "a demonstration of a person performing {}.",
+        "a photo of the person performing {}.",
+        "a video of the person performing {}.",
+        "a example of the person performing {}.",
+        "a demonstration of the person performing {}.",
+        "a photo of a person practicing {}.",
+        "a video of a person practicing {}.",
+        "a example of a person practicing {}.",
+        "a demonstration of a person practicing {}.",
+        "a photo of the person practicing {}.",
+        "a video of the person practicing {}.",
+        "a example of the person practicing {}.",
+        "a demonstration of the person practicing {}."
+    ],
+    "imagenet": [
+        "a bad photo of a {}.",
+        "a photo of many {}.",
+        "a sculpture of a {}.",
+        "a photo of the hard to see {}.",
+        "a low resolution photo of the {}.",
+        "a rendering of a {}.",
+        "graffiti of a {}.",
+        "a bad photo of the {}.",
+        "a cropped photo of the {}.",
+        "a tattoo of a {}.",
+        "the embroidered {}.",
+        "a photo of a hard to see {}.",
+        "a bright photo of a {}.",
+        "a photo of a clean {}.",
+        "a photo of a dirty {}.",
+        "a dark photo of the {}.",
+        "a drawing of a {}.",
+        "a photo of my {}.",
+        "the plastic {}.",
+        "a photo of the cool {}.",
+        "a close-up photo of a {}.",
+        "a black and white photo of the {}.",
+        "a painting of the {}.",
+        "a painting of a {}.",
+        "a pixelated photo of the {}.",
+        "a sculpture of the {}.",
+        "a bright photo of the {}.",
+        "a cropped photo of a {}.",
+        "a plastic {}.",
+        "a photo of the dirty {}.",
+        "a jpeg corrupted photo of a {}.",
+        "a blurry photo of the {}.",
+        "a photo of the {}.",
+        "a good photo of the {}.",
+        "a rendering of the {}.",
+        "a {} in a video game.",
+        "a photo of one {}.",
+        "a doodle of a {}.",
+        "a close-up photo of the {}.",
+        "a photo of a {}.",
+        "the origami {}.",
+        "the {} in a video game.",
+        "a sketch of a {}.",
+        "a doodle of the {}.",
+        "a origami {}.",
+        "a low resolution photo of a {}.",
+        "the toy {}.",
+        "a rendition of the {}.",
+        "a photo of the clean {}.",
+        "a photo of a large {}.",
+        "a rendition of a {}.",
+        "a photo of a nice {}.",
+        "a photo of a weird {}.",
+        "a blurry photo of a {}.",
+        "a cartoon {}.",
+        "art of a {}.",
+        "a sketch of the {}.",
+        "a embroidered {}.",
+        "a pixelated photo of a {}.",
+        "itap of the {}.",
+        "a jpeg corrupted photo of the {}.",
+        "a good photo of a {}.",
+        "a plushie {}.",
+        "a photo of the nice {}.",
+        "a photo of the small {}.",
+        "a photo of the weird {}.",
+        "the cartoon {}.",
+        "art of the {}.",
+        "a drawing of the {}.",
+        "a photo of the large {}.",
+        "a black and white photo of a {}.",
+        "the plushie {}.",
+        "a dark photo of a {}.",
+        "itap of a {}.",
+        "graffiti of the {}.",
+        "a toy {}.",
+        "itap of my {}.",
+        "a photo of a cool {}.",
+        "a photo of a small {}.",
+        "a tattoo of the {}."
+    ]
+}

configs.py

@@ -0,0 +1,105 @@
+import os
+import inspect
+
+from collections import OrderedDict
+from dataclasses import dataclass
+
+import sys
+sys.path.append("src")
+
+from training.params import get_default_params
+
+
+@dataclass
+class Config:
+    tokenizer_context_length = 256
+    eval_freq = 1
+    text_encoder_model_name = 'microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract'
+    distributed_engine = "ddp"
+    train_data = None
+    val_data = None
+    train_num_samples = None
+    val_num_samples = None
+    dataset_type = "auto"
+    dataset_resampled = False
+    csv_separator = "\t"
+    csv_img_key = "filepath"
+    csv_caption_key = "title"
+    imagenet_val = "/datasets01/imagenet_full_size/061417/val"
+    imagenet_v2 = None
+    logs = "./logs/"
+    log_local = False
+    name = None
+    workers = 8
+    batch_size = 64
+    epochs = 32
+    lr = None
+    beta1 = None
+    beta2 = None
+    eps = None
+    wd = 0.2
+    warmup = 2000  # 10000
+    use_bn_sync = False
+    skip_scheduler = False
+    save_frequency = 10
+    save_most_recent = True  # False
+    zeroshot_frequency = 1
+    val_frequency = 1
+    resume = None
+    precision = "amp"
+    clip_model = "CLIP"
+    model = "RN50"
+    pretrained = ''
+    pretrained_image = False
+    lock_image = False
+    lock_image_unlocked_groups = 0
+    lock_image_freeze_bn_stats = False
+    grad_checkpointing = False
+    local_loss = False
+    gather_with_grad = False
+    force_quick_gelu = False
+    torchscript = False
+    trace = False
+    dist_url = "env://"
+    dist_backend = "nccl"
+    report_to = ""
+    wandb_notes = ''
+    debug = False
+    copy_codebase = False
+    horovod = False
+    ddp_static_graph = False
+    no_set_device_rank = False
+    seed = 0
+    norm_gradient_clip = None
+    train_data_upsampling_factors = None
+
+    def __post_init__(self):
+        args = self
+        args.name = self.__class__.__name__
+        args.output_dir = os.path.join(args.logs, args.name)
+
+        for name, val in get_default_params(args.model).items():
+            if getattr(args, name) is None:
+                setattr(args, name, val)
+
+
+def parse_start_end(shards):
+    start, end = os.path.basename(shards).split("{")[1].split("}")[0].split("..")
+    return int(start), int(end)
+
+
+def search_config(config_name):
+    import importlib
+    project_dir = os.path.dirname(__file__)
+    all_configs = {}
+    for code in os.listdir(project_dir):
+        if code.endswith(".py") and code.startswith("run_configs"):
+            module = importlib.import_module(code[:-3])
+            for _config_name in dir(module):
+                if _config_name in ["Config"] or _config_name.startswith("__") or _config_name.startswith("run_config"):
+                    continue
+                if _config_name not in all_configs:
+                    all_configs[_config_name] = module
+    print(f"launching {config_name} from {all_configs[config_name].__file__}")
+    config = getattr(all_configs[config_name], config_name)()
+    return config

data_prepration_scripts/CheXpert/webdataset_chexpert.py

@@ -0,0 +1,74 @@
+import os
+import tarfile
+import io
+import pandas as pd
+import ast
+from tqdm import tqdm
+import argparse
+
+
+def create_webdataset(csv_file, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(csv_file, newline='') as f:
+        reader = pd.read_csv(csv_file, delimiter=',')
+
+        tar_index = 0
+        file_count = 0
+        tar = None
+
+        for row in tqdm(reader.values):
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+
+            filename = ast.literal_eval(row[0])[0]
+            label = ast.literal_eval(row[1])
+            all_caption = ast.literal_eval(row[2])
+            caption = ''
+            for single_caption in all_caption: caption += single_caption.strip('.') + "._chexpert_"
+            # Read the image file
+            image_path = os.path.join(parent_dataset_path, filename)
+            with open(image_path, 'rb') as img_file:
+                img_data = img_file.read()
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+            # Add label.txt to the tarfile
+            label_data = label[0].encode('utf-8')
+            label_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.cls")
+            label_tarinfo.size = len(label_data)
+            # tar.addfile(label_tarinfo, io.BytesIO(label_data))
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+        if tar:
+            tar.close()
+
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_file', type=str, required=True, help="Path to the CSV file")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_dataset_path', type=str, required=True,
+                        help="Path to the parent dataset containing images")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)

data_prepration_scripts/ChestX-ray8/chest-xray_8_webdataset.py

@@ -0,0 +1,72 @@
+import os
+import tarfile
+import io
+import pandas as pd
+import ast
+from tqdm import tqdm
+import argparse
+
+def create_webdataset(csv_file, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(csv_file, newline='') as f:
+        reader = pd.read_csv(csv_file, delimiter=',')
+
+        tar_index = 0
+        file_count = 0
+        tar = None
+
+        for row in tqdm(reader.values):
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+
+            filename = ast.literal_eval(row[0])[0]
+            label = ast.literal_eval(row[1])
+            all_caption = ast.literal_eval(row[2])
+            caption = ''
+            for single_caption in all_caption: caption += single_caption.strip('.') + "._chestxray8_"
+            # Read the image file
+            image_path = os.path.join(parent_dataset_path, filename)
+            with open(image_path, 'rb') as img_file:
+                img_data = img_file.read()
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+            # Add label.txt to the tarfile
+            label_data = label[0].encode('utf-8')
+            label_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.cls")
+            label_tarinfo.size = len(label_data)
+            # tar.addfile(label_tarinfo, io.BytesIO(label_data))
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+        if tar:
+            tar.close()
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_file', type=str, required=True, help="Path to the CSV file")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_dataset_path', type=str, required=True,
+                        help="Path to the parent dataset containing images")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)

data_prepration_scripts/LLaVA-Med/llava_med_alignment_webdataset.py

@@ -0,0 +1,76 @@
+import json
+import os
+import tarfile
+import io
+from tqdm import tqdm
+import argparse
+
+def create_webdataset(json_file_path, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(json_file_path, "r") as f:
+        json_dict = json.load(f)
+        tar_index = 0
+        file_count = 0
+        tar = None
+
+        for single_dict in tqdm(json_dict):
+            row = single_dict
+            # Read the image file
+            filename = row["image"]
+            image_path = os.path.join(parent_dataset_path, filename)
+            try:
+                with open(image_path, 'rb') as img_file:
+                    img_data = img_file.read()
+            except:
+                print(f"image not found: {image_path}, skipping... ")
+                continue
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+
+
+            # label = ast.literal_eval(row[1])
+            all_caption = row["conversations"][1]["value"]  # GPT response...
+            caption = all_caption.strip().strip("\n\n").strip("\n")
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+        if tar:
+            tar.close()
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_file', type=str, required=True, help="Path to the CSV file")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_dataset_path', type=str, required=True,
+                        help="Path to the parent dataset containing images")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)
+
+# Usage example
+json_file = '/home/muzammal/uzair_experiments/datasets/llava_med/llava_med_alignment_500k_filtered.json'
+output_dir = '/home/muzammal/uzair_experiments/datasets/llava_med/llava_med_alignment_set_webdataset/'
+parent_dataset_path = '/home/muzammal/uzair_experiments/datasets/llava_med/images/'
+create_webdataset(json_file, output_dir, parent_dataset_path)

data_prepration_scripts/LLaVA-Med/llava_med_instruct_webdataset.py

@@ -0,0 +1,104 @@
+import json
+import os
+import tarfile
+import io
+import argparse
+from tqdm import tqdm
+
+def create_webdataset(json_file_path, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(json_file_path, "r") as f:
+        json_dict = json.load(f)
+        tar_index = 0
+        file_count = 0
+        tar = None
+        # One for loop for main caption
+        for single_key in json_dict.keys():
+
+            my_list = json_dict[single_key]  # this is a list
+            for single_entry in tqdm(my_list):
+                # Read the image file
+                filename = single_entry["pair_id"] + ".jpg"
+                image_path = os.path.join(parent_dataset_path, filename)
+                try:
+                    with open(image_path, 'rb') as img_file:
+                        img_data = img_file.read()
+                except:
+                    print(f"image not found: {image_path}, skipping... ")
+                    continue
+
+                # label = ast.literal_eval(row[1])
+                all_caption = single_entry["fig_caption"] # GPT response...
+                if str(all_caption) == 'nan':
+                    print(f"original caption not found: {image_path}, skipping... ")
+                    continue
+                caption = all_caption.strip().strip("\n\n").strip("\n")
+                if file_count % tar_size == 0:
+                    if tar:
+                        tar.close()
+                    tar_index += 1
+                    tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                    tar = tarfile.open(tar_path, 'w')
+
+                # Create an in-memory tarfile
+                img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+                img_tarinfo.size = len(img_data)
+                tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+                # Add caption.txt to the tarfile
+                caption_data = caption.encode('utf-8')
+                caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+                caption_tarinfo.size = len(caption_data)
+                tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+                file_count += 1
+
+        # One for loop for inline mention as the captions...
+        for single_key in json_dict.keys():
+
+            my_list = json_dict[single_key]  # this is a list
+            for single_entry in tqdm(my_list):
+                # Read the image file
+                filename = single_entry["pair_id"] + ".jpg"
+                image_path = os.path.join(parent_dataset_path, filename)
+                try:
+                    with open(image_path, 'rb') as img_file:
+                        img_data = img_file.read()
+                except:
+                    print(f"image not found: {image_path}, skipping... ")
+                    continue
+
+                if single_entry["in_text_mention"] is None:
+                    print(f"Inline caption not found: {image_path}, skipping... ")
+                    continue
+                all_caption = single_entry["in_text_mention"][0]['tokens'] # GPT response...
+                caption = all_caption.strip().strip("\n\n").strip("\n")
+                if file_count % tar_size == 0:
+                    if tar:
+                        tar.close()
+                    tar_index += 1
+                    tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                    tar = tarfile.open(tar_path, 'w')
+
+                # Create an in-memory tarfile
+                img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+                img_tarinfo.size = len(img_data)
+                tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+                # Add caption.txt to the tarfile
+                caption_data = caption.encode('utf-8')
+                caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+                caption_tarinfo.size = len(caption_data)
+                tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+                file_count += 1
+        if tar:
+            tar.close()
+
+
+# Usage example
+json_file = '/home/muzammal/uzair_experiments/datasets/llava_med/llava_med_instruct_fig_captions.json'
+output_dir = '/home/muzammal/uzair_experiments/datasets/llava_med/llava_med_hq_60k_set_webdataset/'
+parent_dataset_path = '/home/muzammal/uzair_experiments/datasets/llava_med/images/'
+create_webdataset(json_file, output_dir, parent_dataset_path)

data_prepration_scripts/MIMIC-CXR/webdataset_mimic_cxr.py

@@ -0,0 +1,83 @@
+import os
+import tarfile
+import io
+import pandas as pd
+import ast
+from tqdm import tqdm
+import argparse
+
+def create_webdataset(csv_file, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(csv_file, newline='') as f:
+        reader = pd.read_csv(csv_file, delimiter=',')
+
+        tar_index = 0
+        file_count = 0
+        tar = None
+        total_skipped = 0
+        for row in tqdm(reader.values):
+            filename = ast.literal_eval(row[0])[0]
+            # Read the image file
+            image_path = os.path.join(parent_dataset_path, filename)
+            if not os.path.exists(image_path):
+                print(f"Image path {image_path} does not exist")
+                print(f"Skipping this iteration")
+                total_skipped += 1
+                continue
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+
+            label = ast.literal_eval(row[1])
+            all_caption = ast.literal_eval(row[2])
+            report = row[3]
+            if type(report) != str:
+                report = "noreportpresent"
+            caption = ''
+            # print(f"REPORT IS {report} \n")
+            for single_caption in all_caption: caption += single_caption.strip('.') + "._mimiccxr_"
+            caption = caption + "_mimiccxr_" + report
+            with open(image_path, 'rb') as img_file:
+                img_data = img_file.read()
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+            # Add label.txt to the tarfile
+            label_data = label[0].encode('utf-8')
+            label_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.cls")
+            label_tarinfo.size = len(label_data)
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+        if tar:
+            tar.close()
+
+        print(f"Total {total_skipped} files have been skipped because no image was found for that in our folder.")
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_file', type=str, required=True, help="Path to the CSV file")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_dataset_path', type=str, required=True,
+                        help="Path to the parent dataset containing images")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)

data_prepration_scripts/Openi/openi_webdataset.py

@@ -0,0 +1,70 @@
+import os
+import tarfile
+import io
+import json
+from tqdm.contrib import tzip
+import argparse
+
+def create_webdataset(orig_file_path, refined_file_path, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(orig_file_path, "r") as f:
+        reader = json.load(f)['annotations']
+        reader_refined = json.load(open(refined_file_path, "r"))['refined_captions']
+        tar_index = 0
+        file_count = 0
+        tar = None
+
+        for full_caption, refined_caption_list in tzip(reader, reader_refined):
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+
+            filename = full_caption['image_id']
+            caption_orig = [full_caption['caption']]
+            caption_orig += refined_caption_list
+
+            all_caption = caption_orig
+            caption = ''
+            for single_caption in all_caption: caption += single_caption.strip('.') + "._openi_"
+            # Read the image file
+            image_path = os.path.join(parent_dataset_path, filename + ".png")
+            with open(image_path, 'rb') as img_file:
+                img_data = img_file.read()
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+        if tar:
+            tar.close()
+
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--original_json_file_summarizations_path', type=str, required=True)
+    parser.add_argument('--gpt_text_descriptions_path', type=str, required=True, help="Path to the CSV file")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_dataset_path', type=str, required=True,
+                        help="Path to the parent dataset containing images")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.original_json_file_summarizations_path, args.gpt_text_descriptions_path, args.output_dir,
+                      args.parent_dataset_path)

data_prepration_scripts/PMC-OA/pmc_oa_webdataset.py

@@ -0,0 +1,64 @@
+import json
+import os
+import tarfile
+import io
+from tqdm import tqdm
+import argparse
+
+def create_webdataset(csv_file, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(csv_file, "r") as f:
+        json_list = list(f)
+        tar_index = 0
+        file_count = 0
+        tar = None
+
+        for row_raw in tqdm(json_list):
+            row = json.loads(row_raw)
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+
+            filename = row["image"]
+            # label = ast.literal_eval(row[1])
+            all_caption = row["caption"]
+            caption = all_caption.strip().strip("\n\n").strip("\n")
+            # Read the image file
+            image_path = os.path.join(parent_dataset_path, filename)
+            with open(image_path, 'rb') as img_file:
+                img_data = img_file.read()
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+        if tar:
+            tar.close()
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_file', type=str, required=True, help="Path to the CSV file")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_dataset_path', type=str, required=True,
+                        help="Path to the parent dataset containing images")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)

data_prepration_scripts/Quilt-1M/quilt_1m_webdataset.py

@@ -0,0 +1,70 @@
+import os
+import tarfile
+import io
+import pandas as pd
+from tqdm import tqdm
+import argparse
+
+def create_webdataset(csv_file, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(csv_file, newline='') as f:
+        reader = pd.read_csv(csv_file, delimiter=',')
+
+        tar_index = 0
+        file_count = 0
+        tar = None
+
+        for row in tqdm(reader.values):
+            filename = row[2]
+            # label = ast.literal_eval(row[1])
+            all_caption = row[1]
+            if str(all_caption) == 'nan':
+                continue
+            caption = all_caption.strip().strip("\n\n").strip("\n")
+            # Read the image file
+            image_path = os.path.join(parent_dataset_path, filename)
+            try:
+                with open(image_path, 'rb') as img_file:
+                    img_data = img_file.read()
+            except:
+                print(f"image not found: {image_path} \n subset is {row[3]} ")
+                continue
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+        if tar:
+            tar.close()
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_file', type=str, required=True, help="Path to the CSV file")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_dataset_path', type=str, required=True,
+                        help="Path to the parent dataset containing images")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)

data_prepration_scripts/ROCOV2/roco_webdataset.py

@@ -0,0 +1,65 @@
+import os
+import tarfile
+import io
+import pandas as pd
+from tqdm import tqdm
+import argparse
+
+def create_webdataset(csv_file, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(csv_file, newline='') as f:
+        reader = pd.read_csv(csv_file, delimiter=',')
+
+        tar_index = 0
+        file_count = 0
+        tar = None
+
+        for row in tqdm(reader.values):
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+
+            filename = row[0] + '.jpg'
+            # label = ast.literal_eval(row[1])
+            all_caption = row[1]
+            caption = all_caption.strip().strip("\n\n").strip("\n")
+            # Read the image file
+            image_path = os.path.join(parent_dataset_path, filename)
+            with open(image_path, 'rb') as img_file:
+                img_data = img_file.read()
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+        if tar:
+            tar.close()
+
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_file', type=str, required=True, help="Path to the CSV file")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_dataset_path', type=str, required=True,
+                        help="Path to the parent dataset containing images")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)

data_prepration_scripts/RadImageNet/radimagenet_webdataset.py

@@ -0,0 +1,72 @@
+import os
+import tarfile
+import io
+import pandas as pd
+import ast
+from tqdm import tqdm
+import argparse
+
+def create_webdataset(csv_file, output_dir, parent_dataset_path, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+
+    with open(csv_file, newline='') as f:
+        reader = pd.read_csv(csv_file, delimiter=',')
+
+        tar_index = 0
+        file_count = 0
+        tar = None
+
+        for row in tqdm(reader.values):
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+
+            filename = ast.literal_eval(row[0])[0]
+            label = ast.literal_eval(row[1])
+            all_caption = ast.literal_eval(row[2])
+            caption = ''
+            for single_caption in all_caption: caption += single_caption + "._radimagenet_"
+            # Read the image file
+            image_path = os.path.join(parent_dataset_path, filename)
+            with open(image_path, 'rb') as img_file:
+                img_data = img_file.read()
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+            # Add label.txt to the tarfile
+            label_data = label[0].encode('utf-8')
+            label_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.cls")
+            label_tarinfo.size = len(label_data)
+            # tar.addfile(label_tarinfo, io.BytesIO(label_data))
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+        if tar:
+            tar.close()
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_file', type=str, required=True, help="Path to the CSV file")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_dataset_path', type=str, required=True,
+                        help="Path to the parent dataset containing images")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)

data_prepration_scripts/Retinal-Datasets/retina_webdataset_part1.py

@@ -0,0 +1,86 @@
+import os
+import tarfile
+import io
+import pandas as pd
+import ast
+from tqdm import tqdm
+import argparse
+
+banned_categories = ['myopia', 'cataract', 'macular hole', 'retinitis pigmentosa', "myopic", "myope", "myop", "retinitis"]
+
+def create_webdataset(main_csv_directory, image_dir_path, output_dir, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+    # Load both csv files
+    all_datasets = os.listdir(main_csv_directory)
+    tar_index = 0
+    file_count = 0
+    tar = None
+    for iDataset in tqdm(all_datasets):
+        print("Processing data: " + iDataset)
+        if iDataset == "06_DEN.csv" or iDataset == "39_MM_Retinal_dataset.csv" or \
+                iDataset == "28_OIA-DDR_revised.csv" or iDataset == '07_LAG_revised.csv' \
+                or iDataset == '01_EYEPACS_revised.csv':
+            continue
+        dataframe = pd.read_csv(main_csv_directory + iDataset)
+        selected_id_list = range(len(dataframe))  # 100%数据   100% data
+
+        for i in selected_id_list:
+            if file_count % tar_size == 0:
+                if tar:
+                    tar.close()
+                tar_index += 1
+                tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+                tar = tarfile.open(tar_path, 'w')
+            data_i = dataframe.loc[i, :].to_dict()  # image,attributes,categories   Turn each line into a dictionary
+
+            image_file_name = data_i['filename']
+            all_caption = ast.literal_eval(data_i['captions'])
+            sentence_level_caption = [data_i['sentence_level_captions']]
+            all_caption += sentence_level_caption
+            # Now need to process the captions
+            if str(all_caption) == 'nan':
+                continue
+
+            caption = ''
+            for single_caption in all_caption: caption += single_caption.strip('.') + "._all_retina_merged_"
+            # Read the image file
+            image_path = os.path.join(image_dir_path, image_file_name)
+            try:
+                with open(image_path, 'rb') as img_file:
+                    img_data = img_file.read()
+            except:
+                print(f"image not found: {image_path} \n subset is {image_file_name} ")
+                continue
+
+
+            # Create an in-memory tarfile
+            img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+            img_tarinfo.size = len(img_data)
+            tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+            # Add caption.txt to the tarfile
+            caption_data = caption.encode('utf-8')
+            caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+            caption_tarinfo.size = len(caption_data)
+            tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+            file_count += 1
+
+    if tar:
+        tar.close()
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_files_directory', type=str, required=True, help="Path to the CSV files for all datasets")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_datasets_path', type=str, required=True,
+                        help="Path to the parent folder containing Retina Datasets folders")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)
+

data_prepration_scripts/Retinal-Datasets/retina_webdataset_part2.py

@@ -0,0 +1,105 @@
+import os
+import random
+import tarfile
+import io
+import pandas as pd
+from tqdm import tqdm
+import argparse
+
+banned_categories = ['myopia', 'cataract', 'macular hole', 'retinitis pigmentosa', "myopic", "myope", "myop", "retinitis"]
+
+def create_webdataset(main_csv_directory, image_dir_path, output_dir, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+    # Load both csv files
+    tar_index = 0
+    file_count = 0
+    tar = None
+    # Now lets do it for that vision language model
+    dataframe = pd.read_csv(main_csv_directory + "06_DEN.csv")
+    selected_id_list = range(len(dataframe))  # 100%数据   100% data
+
+    for i in tqdm(selected_id_list):
+        if file_count % tar_size == 0:
+            if tar:
+                tar.close()
+            tar_index += 1
+            tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+            tar = tarfile.open(tar_path, 'w')
+        data_i = dataframe.loc[i, :].to_dict()  # image,attributes,categories   Turn each line into a dictionary
+        data_i["categories"] = eval(data_i["categories"])
+        data_i["atributes"] = eval(data_i["atributes"])
+        all_categories = data_i["categories"]
+        final_caption = None
+        for single_category in all_categories:
+            # Filtering noisy captions...
+            if ("year" not in single_category.strip('/')) and ("//" not in single_category.strip('/')):
+                final_caption = "The fundus image of " +  single_category
+                if file_count < 50:
+                    print(final_caption)
+                
+        if final_caption == None:
+            final_caption = random.sample(all_categories, 1)[0]
+            # print(final_caption)
+
+        image_file_name = data_i['image']
+        # Now need to process the captions
+        if str(final_caption) == 'nan':
+            continue
+
+        caption = final_caption
+        # Read the image file
+        image_path = os.path.join(image_dir_path, image_file_name)
+        try:
+            with open(image_path, 'rb') as img_file:
+                img_data = img_file.read()
+        except:
+            print(f"image not found: {image_path} \n subset is {image_file_name} ")
+            continue
+
+        # Create an in-memory tarfile
+        img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+        img_tarinfo.size = len(img_data)
+        tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+        # Add caption.txt to the tarfile
+        caption_data = caption.encode('utf-8')
+        caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+        caption_tarinfo.size = len(caption_data)
+        tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+        file_count += 1
+
+
+    if tar:
+        tar.close()
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_files_directory', type=str, required=True, help="Path to the CSV files for all datasets")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_datasets_path', type=str, required=True,
+                        help="Path to the parent folder containing Retina Datasets folders")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_files_directory', type=str, required=True, help="Path to the CSV files for all datasets")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_datasets_path', type=str, required=True,
+                        help="Path to the parent folder containing Retina Datasets folders")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)
+

data_prepration_scripts/Retinal-Datasets/retina_webdataset_part3.py

@@ -0,0 +1,77 @@
+import os
+import tarfile
+import io
+import pandas as pd
+from tqdm import tqdm
+import argparse
+
+banned_categories = ['myopia', 'cataract', 'macular hole', 'retinitis pigmentosa', "myopic", "myope", "myop", "retinitis"]
+
+def create_webdataset(main_csv_directory, image_dir_path, output_dir, tar_size=1000):
+    os.makedirs(output_dir, exist_ok=True)
+    # Load both csv files
+    tar_index = 0
+    file_count = 0
+    tar = None
+    # Now lets do it for that vision language model
+    dataframe = pd.read_csv(main_csv_directory + "39_MM_Retinal_dataset.csv")
+    selected_id_list = range(len(dataframe))  # 100%数据   100% data
+
+    for i in tqdm(selected_id_list):
+        if file_count % tar_size == 0:
+            if tar:
+                tar.close()
+            tar_index += 1
+            tar_path = os.path.join(output_dir, f"dataset-{tar_index:06d}.tar")
+            tar = tarfile.open(tar_path, 'w')
+        sample_df = dataframe.loc[i, :].to_dict()  # image,attributes,categories   Turn each line into a dictionary
+        image_path = sample_df["image"]
+        caption = sample_df["caption"]
+        final_caption = caption
+        image_file_name = image_path
+        # Now need to process the captions
+        if str(final_caption) == 'nan':
+            continue
+
+        caption = final_caption
+        # Read the image file
+        image_path = os.path.join(image_dir_path, image_file_name)
+        try:
+            with open(image_path, 'rb') as img_file:
+                img_data = img_file.read()
+        except:
+            print(f"image not found: {image_path} \n subset is {image_file_name} ")
+            continue
+
+        # Create an in-memory tarfile
+        img_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.jpg")
+        img_tarinfo.size = len(img_data)
+        tar.addfile(img_tarinfo, io.BytesIO(img_data))
+
+        # Add caption.txt to the tarfile
+        caption_data = caption.encode('utf-8')
+        caption_tarinfo = tarfile.TarInfo(name=f"{file_count:06d}.txt")
+        caption_tarinfo.size = len(caption_data)
+        tar.addfile(caption_tarinfo, io.BytesIO(caption_data))
+
+        file_count += 1
+
+
+    if tar:
+        tar.close()
+
+
+if __name__ == "__main__":
+    # Argument parser setup
+    parser = argparse.ArgumentParser(description="Create a WebDataset from CSV")
+    parser.add_argument('--csv_files_directory', type=str, required=True, help="Path to the CSV files for all datasets")
+    parser.add_argument('--output_dir', type=str, required=True, help="Directory to store the output tar files")
+    parser.add_argument('--parent_datasets_path', type=str, required=True,
+                        help="Path to the parent folder containing Retina Datasets folders")
+    parser.add_argument('--tar_size', type=int, default=1000, help="Number of files per tar file")
+
+    # Parse the arguments
+    args = parser.parse_args()
+
+    # Call the function with the parsed arguments
+    create_webdataset(args.csv_file, args.output_dir, args.parent_dataset_path, args.tar_size)

getting_started_unimed_clip.ipynb

@@ -0,0 +1,227 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "# Switch path to root of project\n",
+    "import os\n",
+    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"\n",
+    "# Get the current working directory\n",
+    "current_dir = os.getcwd()\n",
+    "src_path = os.path.join(current_dir, 'src')\n",
+    "os.chdir(src_path)"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "from open_clip import create_model_and_transforms, get_mean_std\n",
+    "from open_clip import HFTokenizer\n",
+    "from PIL import Image\n",
+    "import torch\n",
+    "from urllib.request import urlopen"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "# Define main parameters\n",
+    "model = 'ViT-L-14-336-quickgelu' # available pretrained weights ['ViT-L-14-336-quickgelu', 'ViT-B-16-quickgelu']\n",
+    "pretrained = \"./unimed_clip_vit_l14_base_text_encoder.pt\" # Path to pretrained weights\n",
+    "text_encoder_name = \"microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract\" # available pretrained weights [\"microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract\", \"microsoft/BiomedNLP-BiomedBERT-large-uncased-abstract\"]\n",
+    "mean, std = get_mean_std()\n",
+    "device='cuda'"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "model, _, preprocess = create_model_and_transforms(\n",
+    "    model,\n",
+    "    pretrained,\n",
+    "    precision='amp',\n",
+    "    device=device,\n",
+    "    force_quick_gelu=True,\n",
+    "    pretrained_image=False,\n",
+    "    mean=mean, std=std,\n",
+    "    inmem=True,\n",
+    "    text_encoder_name=text_encoder_name,\n",
+    ")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "tokenizer = HFTokenizer(\n",
+    "    text_encoder_name,\n",
+    "    context_length=256,\n",
+    "    **{},\n",
+    ")"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "# Zeroshot Inference"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "# Zero-shot image classification\n",
+    "template = 'this is a photo of '\n",
+    "labels = [\n",
+    "    'adenocarcinoma histopathology',\n",
+    "    'brain MRI',\n",
+    "    'covid line chart',\n",
+    "    'squamous cell carcinoma histopathology',\n",
+    "    'immunohistochemistry histopathology',\n",
+    "    'bone X-ray',\n",
+    "    'chest X-ray',\n",
+    "    'pie chart',\n",
+    "    'hematoxylin and eosin histopathology'\n",
+    "]"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "dataset_url = 'https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/resolve/main/example_data/biomed_image_classification_example_data/'\n",
+    "test_imgs = [\n",
+    "    'squamous_cell_carcinoma_histopathology.jpeg',\n",
+    "    'H_and_E_histopathology.jpg',\n",
+    "    'bone_X-ray.jpg',\n",
+    "    'adenocarcinoma_histopathology.jpg',\n",
+    "    'covid_line_chart.png',\n",
+    "    'IHC_histopathology.jpg',\n",
+    "    'chest_X-ray.jpg',\n",
+    "    'brain_MRI.jpg',\n",
+    "    'pie_chart.png'\n",
+    "]"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "images = torch.stack([preprocess(Image.open(urlopen(dataset_url + img))) for img in test_imgs]).to(device)\n",
+    "texts = [tokenizer(template + cls_text).to(next(model.parameters()).device, non_blocking=True) for cls_text in labels]\n",
+    "texts = torch.cat(texts, dim=0)\n",
+    "with torch.no_grad():\n",
+    "    text_features = model.encode_text(texts)\n",
+    "    text_features = text_features / text_features.norm(dim=-1, keepdim=True)\n",
+    "    image_features = model.encode_image(images)\n",
+    "    image_features = image_features / image_features.norm(dim=-1, keepdim=True)\n",
+    "    logits = (image_features @ text_features.t()).detach().softmax(dim=-1)\n",
+    "    sorted_indices = torch.argsort(logits, dim=-1, descending=True)\n",
+    "\n",
+    "    logits = logits.cpu().numpy()\n",
+    "    sorted_indices = sorted_indices.cpu().numpy()\n",
+    "\n",
+    "top_k = -1\n",
+    "\n",
+    "for i, img in enumerate(test_imgs):\n",
+    "    pred = labels[sorted_indices[i][0]]\n",
+    "\n",
+    "    top_k = len(labels) if top_k == -1 else top_k\n",
+    "    print(img.split('/')[-1] + ':')\n",
+    "    for j in range(top_k):\n",
+    "        jth_index = sorted_indices[i][j]\n",
+    "        print(f'{labels[jth_index]}: {logits[i][jth_index]}')\n",
+    "    print('\\n')"
+   ],
+   "metadata": {
+    "collapsed": false,
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   }
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

local_data/08_ODIR200x3.csv

@@ -0,0 +1,601 @@
+,image,atributes,categories
+0,08_ODIR-5K/preprocessed_images/0_right.jpg,[],['normal']
+1,08_ODIR-5K/preprocessed_images/0_left.jpg,[],['cataract']
+2,08_ODIR-5K/preprocessed_images/1_right.jpg,[],['normal']
+3,08_ODIR-5K/preprocessed_images/1_left.jpg,[],['normal']
+4,08_ODIR-5K/preprocessed_images/8_right.jpg,[],['normal']
+5,08_ODIR-5K/preprocessed_images/8_left.jpg,[],['normal']
+6,08_ODIR-5K/preprocessed_images/13_right.jpg,[],['pathologic myopia']
+7,08_ODIR-5K/preprocessed_images/13_left.jpg,[],['pathologic myopia']
+8,08_ODIR-5K/preprocessed_images/16_right.jpg,[],['pathologic myopia']
+9,08_ODIR-5K/preprocessed_images/16_left.jpg,[],['normal']
+10,08_ODIR-5K/preprocessed_images/18_right.jpg,[],['pathologic myopia']
+11,08_ODIR-5K/preprocessed_images/18_left.jpg,[],['pathologic myopia']
+12,08_ODIR-5K/preprocessed_images/24_right.jpg,[],['cataract']
+13,08_ODIR-5K/preprocessed_images/24_left.jpg,[],['normal']
+14,08_ODIR-5K/preprocessed_images/35_right.jpg,[],['normal']
+15,08_ODIR-5K/preprocessed_images/35_left.jpg,[],['pathologic myopia']
+16,08_ODIR-5K/preprocessed_images/84_right.jpg,[],['normal']
+17,08_ODIR-5K/preprocessed_images/84_left.jpg,[],['normal']
+18,08_ODIR-5K/preprocessed_images/103_right.jpg,[],['normal']
+19,08_ODIR-5K/preprocessed_images/103_left.jpg,[],['cataract']
+20,08_ODIR-5K/preprocessed_images/106_right.jpg,[],['pathologic myopia']
+21,08_ODIR-5K/preprocessed_images/106_left.jpg,[],['pathologic myopia']
+22,08_ODIR-5K/preprocessed_images/112_right.jpg,[],['cataract']
+23,08_ODIR-5K/preprocessed_images/112_left.jpg,[],['normal']
+24,08_ODIR-5K/preprocessed_images/144_right.jpg,[],['normal']
+25,08_ODIR-5K/preprocessed_images/144_left.jpg,[],['pathologic myopia']
+26,08_ODIR-5K/preprocessed_images/145_right.jpg,[],['pathologic myopia']
+27,08_ODIR-5K/preprocessed_images/145_left.jpg,[],['pathologic myopia']
+28,08_ODIR-5K/preprocessed_images/174_right.jpg,[],['normal']
+29,08_ODIR-5K/preprocessed_images/174_left.jpg,[],['pathologic myopia']
+30,08_ODIR-5K/preprocessed_images/191_right.jpg,[],['normal']
+31,08_ODIR-5K/preprocessed_images/191_left.jpg,[],['normal']
+32,08_ODIR-5K/preprocessed_images/214_right.jpg,[],['pathologic myopia']
+33,08_ODIR-5K/preprocessed_images/214_left.jpg,[],['normal']
+34,08_ODIR-5K/preprocessed_images/233_right.jpg,[],['pathologic myopia']
+35,08_ODIR-5K/preprocessed_images/233_left.jpg,[],['normal']
+36,08_ODIR-5K/preprocessed_images/294_right.jpg,[],['normal']
+37,08_ODIR-5K/preprocessed_images/294_left.jpg,[],['cataract']
+38,08_ODIR-5K/preprocessed_images/330_right.jpg,[],['normal']
+39,08_ODIR-5K/preprocessed_images/330_left.jpg,[],['cataract']
+40,08_ODIR-5K/preprocessed_images/354_right.jpg,[],['cataract']
+41,08_ODIR-5K/preprocessed_images/354_left.jpg,[],['normal']
+42,08_ODIR-5K/preprocessed_images/379_right.jpg,[],['normal']
+43,08_ODIR-5K/preprocessed_images/379_left.jpg,[],['pathologic myopia']
+44,08_ODIR-5K/preprocessed_images/382_right.jpg,[],['pathologic myopia']
+45,08_ODIR-5K/preprocessed_images/382_left.jpg,[],['pathologic myopia']
+46,08_ODIR-5K/preprocessed_images/390_right.jpg,[],['pathologic myopia']
+47,08_ODIR-5K/preprocessed_images/390_left.jpg,[],['normal']
+48,08_ODIR-5K/preprocessed_images/394_right.jpg,[],['normal']
+49,08_ODIR-5K/preprocessed_images/394_left.jpg,[],['normal']
+50,08_ODIR-5K/preprocessed_images/401_right.jpg,[],['pathologic myopia']
+51,08_ODIR-5K/preprocessed_images/401_left.jpg,[],['pathologic myopia']
+52,08_ODIR-5K/preprocessed_images/477_right.jpg,[],['cataract']
+53,08_ODIR-5K/preprocessed_images/477_left.jpg,[],['cataract']
+54,08_ODIR-5K/preprocessed_images/481_right.jpg,[],['normal']
+55,08_ODIR-5K/preprocessed_images/481_left.jpg,[],['pathologic myopia']
+56,08_ODIR-5K/preprocessed_images/504_right.jpg,[],['pathologic myopia']
+57,08_ODIR-5K/preprocessed_images/504_left.jpg,[],['pathologic myopia']
+58,08_ODIR-5K/preprocessed_images/525_right.jpg,[],['pathologic myopia']
+59,08_ODIR-5K/preprocessed_images/525_left.jpg,[],['pathologic myopia']
+60,08_ODIR-5K/preprocessed_images/530_right.jpg,[],['normal']
+61,08_ODIR-5K/preprocessed_images/530_left.jpg,[],['normal']
+62,08_ODIR-5K/preprocessed_images/553_right.jpg,[],['cataract']
+63,08_ODIR-5K/preprocessed_images/553_left.jpg,[],['cataract']
+64,08_ODIR-5K/preprocessed_images/560_right.jpg,[],['cataract']
+65,08_ODIR-5K/preprocessed_images/560_left.jpg,[],['cataract']
+66,08_ODIR-5K/preprocessed_images/695_right.jpg,[],['normal']
+67,08_ODIR-5K/preprocessed_images/695_left.jpg,[],['normal']
+68,08_ODIR-5K/preprocessed_images/726_right.jpg,[],['cataract']
+69,08_ODIR-5K/preprocessed_images/726_left.jpg,[],['cataract']
+70,08_ODIR-5K/preprocessed_images/737_right.jpg,[],['pathologic myopia']
+71,08_ODIR-5K/preprocessed_images/737_left.jpg,[],['normal']
+72,08_ODIR-5K/preprocessed_images/741_right.jpg,[],['normal']
+73,08_ODIR-5K/preprocessed_images/741_left.jpg,[],['pathologic myopia']
+74,08_ODIR-5K/preprocessed_images/771_right.jpg,[],['normal']
+75,08_ODIR-5K/preprocessed_images/771_left.jpg,[],['pathologic myopia']
+76,08_ODIR-5K/preprocessed_images/809_right.jpg,[],['normal']
+77,08_ODIR-5K/preprocessed_images/809_left.jpg,[],['cataract']
+78,08_ODIR-5K/preprocessed_images/820_right.jpg,[],['normal']
+79,08_ODIR-5K/preprocessed_images/820_left.jpg,[],['cataract']
+80,08_ODIR-5K/preprocessed_images/824_right.jpg,[],['normal']
+81,08_ODIR-5K/preprocessed_images/824_left.jpg,[],['cataract']
+82,08_ODIR-5K/preprocessed_images/832_right.jpg,[],['normal']
+83,08_ODIR-5K/preprocessed_images/832_left.jpg,[],['normal']
+84,08_ODIR-5K/preprocessed_images/908_right.jpg,[],['pathologic myopia']
+85,08_ODIR-5K/preprocessed_images/939_right.jpg,[],['normal']
+86,08_ODIR-5K/preprocessed_images/939_left.jpg,[],['normal']
+87,08_ODIR-5K/preprocessed_images/951_right.jpg,[],['normal']
+88,08_ODIR-5K/preprocessed_images/951_left.jpg,[],['normal']
+89,08_ODIR-5K/preprocessed_images/955_right.jpg,[],['cataract']
+90,08_ODIR-5K/preprocessed_images/999_right.jpg,[],['cataract']
+91,08_ODIR-5K/preprocessed_images/999_left.jpg,[],['cataract']
+92,08_ODIR-5K/preprocessed_images/1034_right.jpg,[],['normal']
+93,08_ODIR-5K/preprocessed_images/1034_left.jpg,[],['normal']
+94,08_ODIR-5K/preprocessed_images/1054_right.jpg,[],['normal']
+95,08_ODIR-5K/preprocessed_images/1054_left.jpg,[],['normal']
+96,08_ODIR-5K/preprocessed_images/1060_right.jpg,[],['normal']
+97,08_ODIR-5K/preprocessed_images/1060_left.jpg,[],['normal']
+98,08_ODIR-5K/preprocessed_images/1062_right.jpg,[],['cataract']
+99,08_ODIR-5K/preprocessed_images/1072_right.jpg,[],['normal']
+100,08_ODIR-5K/preprocessed_images/1072_left.jpg,[],['normal']
+101,08_ODIR-5K/preprocessed_images/1102_right.jpg,[],['cataract']
+102,08_ODIR-5K/preprocessed_images/1102_left.jpg,[],['cataract']
+103,08_ODIR-5K/preprocessed_images/1103_right.jpg,[],['pathologic myopia']
+104,08_ODIR-5K/preprocessed_images/1103_left.jpg,[],['pathologic myopia']
+105,08_ODIR-5K/preprocessed_images/1115_right.jpg,[],['normal']
+106,08_ODIR-5K/preprocessed_images/1115_left.jpg,[],['cataract']
+107,08_ODIR-5K/preprocessed_images/1126_right.jpg,[],['cataract']
+108,08_ODIR-5K/preprocessed_images/1126_left.jpg,[],['normal']
+109,08_ODIR-5K/preprocessed_images/1150_right.jpg,[],['pathologic myopia']
+110,08_ODIR-5K/preprocessed_images/1150_left.jpg,[],['pathologic myopia']
+111,08_ODIR-5K/preprocessed_images/1164_right.jpg,[],['normal']
+112,08_ODIR-5K/preprocessed_images/1164_left.jpg,[],['cataract']
+113,08_ODIR-5K/preprocessed_images/1253_right.jpg,[],['normal']
+114,08_ODIR-5K/preprocessed_images/1253_left.jpg,[],['normal']
+115,08_ODIR-5K/preprocessed_images/1349_right.jpg,[],['pathologic myopia']
+116,08_ODIR-5K/preprocessed_images/1349_left.jpg,[],['pathologic myopia']
+117,08_ODIR-5K/preprocessed_images/1435_right.jpg,[],['normal']
+118,08_ODIR-5K/preprocessed_images/1435_left.jpg,[],['cataract']
+119,08_ODIR-5K/preprocessed_images/1444_right.jpg,[],['normal']
+120,08_ODIR-5K/preprocessed_images/1444_left.jpg,[],['cataract']
+121,08_ODIR-5K/preprocessed_images/1452_right.jpg,[],['normal']
+122,08_ODIR-5K/preprocessed_images/1452_left.jpg,[],['cataract']
+123,08_ODIR-5K/preprocessed_images/1460_right.jpg,[],['cataract']
+124,08_ODIR-5K/preprocessed_images/1460_left.jpg,[],['normal']
+125,08_ODIR-5K/preprocessed_images/1468_right.jpg,[],['normal']
+126,08_ODIR-5K/preprocessed_images/1468_left.jpg,[],['cataract']
+127,08_ODIR-5K/preprocessed_images/1483_right.jpg,[],['cataract']
+128,08_ODIR-5K/preprocessed_images/1483_left.jpg,[],['cataract']
+129,08_ODIR-5K/preprocessed_images/1484_right.jpg,[],['cataract']
+130,08_ODIR-5K/preprocessed_images/1484_left.jpg,[],['cataract']
+131,08_ODIR-5K/preprocessed_images/1490_right.jpg,[],['cataract']
+132,08_ODIR-5K/preprocessed_images/1490_left.jpg,[],['normal']
+133,08_ODIR-5K/preprocessed_images/1513_right.jpg,[],['pathologic myopia']
+134,08_ODIR-5K/preprocessed_images/1513_left.jpg,[],['normal']
+135,08_ODIR-5K/preprocessed_images/1519_right.jpg,[],['pathologic myopia']
+136,08_ODIR-5K/preprocessed_images/1519_left.jpg,[],['normal']
+137,08_ODIR-5K/preprocessed_images/1523_right.jpg,[],['pathologic myopia']
+138,08_ODIR-5K/preprocessed_images/1523_left.jpg,[],['pathologic myopia']
+139,08_ODIR-5K/preprocessed_images/1526_right.jpg,[],['pathologic myopia']
+140,08_ODIR-5K/preprocessed_images/1526_left.jpg,[],['pathologic myopia']
+141,08_ODIR-5K/preprocessed_images/1527_right.jpg,[],['pathologic myopia']
+142,08_ODIR-5K/preprocessed_images/1527_left.jpg,[],['pathologic myopia']
+143,08_ODIR-5K/preprocessed_images/1529_right.jpg,[],['pathologic myopia']
+144,08_ODIR-5K/preprocessed_images/1529_left.jpg,[],['pathologic myopia']
+145,08_ODIR-5K/preprocessed_images/1530_right.jpg,[],['pathologic myopia']
+146,08_ODIR-5K/preprocessed_images/1530_left.jpg,[],['pathologic myopia']
+147,08_ODIR-5K/preprocessed_images/1531_right.jpg,[],['pathologic myopia']
+148,08_ODIR-5K/preprocessed_images/1531_left.jpg,[],['pathologic myopia']
+149,08_ODIR-5K/preprocessed_images/1532_right.jpg,[],['pathologic myopia']
+150,08_ODIR-5K/preprocessed_images/1532_left.jpg,[],['pathologic myopia']
+151,08_ODIR-5K/preprocessed_images/1533_right.jpg,[],['pathologic myopia']
+152,08_ODIR-5K/preprocessed_images/1533_left.jpg,[],['pathologic myopia']
+153,08_ODIR-5K/preprocessed_images/1534_right.jpg,[],['pathologic myopia']
+154,08_ODIR-5K/preprocessed_images/1534_left.jpg,[],['pathologic myopia']
+155,08_ODIR-5K/preprocessed_images/1535_right.jpg,[],['pathologic myopia']
+156,08_ODIR-5K/preprocessed_images/1535_left.jpg,[],['pathologic myopia']
+157,08_ODIR-5K/preprocessed_images/1537_right.jpg,[],['pathologic myopia']
+158,08_ODIR-5K/preprocessed_images/1537_left.jpg,[],['pathologic myopia']
+159,08_ODIR-5K/preprocessed_images/1538_right.jpg,[],['pathologic myopia']
+160,08_ODIR-5K/preprocessed_images/1538_left.jpg,[],['pathologic myopia']
+161,08_ODIR-5K/preprocessed_images/1539_right.jpg,[],['pathologic myopia']
+162,08_ODIR-5K/preprocessed_images/1539_left.jpg,[],['pathologic myopia']
+163,08_ODIR-5K/preprocessed_images/1540_right.jpg,[],['pathologic myopia']
+164,08_ODIR-5K/preprocessed_images/1541_right.jpg,[],['normal']
+165,08_ODIR-5K/preprocessed_images/1541_left.jpg,[],['pathologic myopia']
+166,08_ODIR-5K/preprocessed_images/1544_right.jpg,[],['pathologic myopia']
+167,08_ODIR-5K/preprocessed_images/1544_left.jpg,[],['pathologic myopia']
+168,08_ODIR-5K/preprocessed_images/1545_right.jpg,[],['pathologic myopia']
+169,08_ODIR-5K/preprocessed_images/1545_left.jpg,[],['pathologic myopia']
+170,08_ODIR-5K/preprocessed_images/1546_right.jpg,[],['pathologic myopia']
+171,08_ODIR-5K/preprocessed_images/1546_left.jpg,[],['pathologic myopia']
+172,08_ODIR-5K/preprocessed_images/1547_right.jpg,[],['pathologic myopia']
+173,08_ODIR-5K/preprocessed_images/1547_left.jpg,[],['pathologic myopia']
+174,08_ODIR-5K/preprocessed_images/1548_right.jpg,[],['pathologic myopia']
+175,08_ODIR-5K/preprocessed_images/1548_left.jpg,[],['pathologic myopia']
+176,08_ODIR-5K/preprocessed_images/1549_right.jpg,[],['pathologic myopia']
+177,08_ODIR-5K/preprocessed_images/1549_left.jpg,[],['pathologic myopia']
+178,08_ODIR-5K/preprocessed_images/1550_right.jpg,[],['pathologic myopia']
+179,08_ODIR-5K/preprocessed_images/1550_left.jpg,[],['pathologic myopia']
+180,08_ODIR-5K/preprocessed_images/1551_right.jpg,[],['pathologic myopia']
+181,08_ODIR-5K/preprocessed_images/1551_left.jpg,[],['pathologic myopia']
+182,08_ODIR-5K/preprocessed_images/1552_right.jpg,[],['pathologic myopia']
+183,08_ODIR-5K/preprocessed_images/1552_left.jpg,[],['pathologic myopia']
+184,08_ODIR-5K/preprocessed_images/1553_right.jpg,[],['pathologic myopia']
+185,08_ODIR-5K/preprocessed_images/1553_left.jpg,[],['pathologic myopia']
+186,08_ODIR-5K/preprocessed_images/1554_right.jpg,[],['pathologic myopia']
+187,08_ODIR-5K/preprocessed_images/1554_left.jpg,[],['pathologic myopia']
+188,08_ODIR-5K/preprocessed_images/1555_right.jpg,[],['pathologic myopia']
+189,08_ODIR-5K/preprocessed_images/1555_left.jpg,[],['pathologic myopia']
+190,08_ODIR-5K/preprocessed_images/1556_right.jpg,[],['normal']
+191,08_ODIR-5K/preprocessed_images/1556_left.jpg,[],['pathologic myopia']
+192,08_ODIR-5K/preprocessed_images/1558_right.jpg,[],['pathologic myopia']
+193,08_ODIR-5K/preprocessed_images/1558_left.jpg,[],['pathologic myopia']
+194,08_ODIR-5K/preprocessed_images/1561_right.jpg,[],['pathologic myopia']
+195,08_ODIR-5K/preprocessed_images/1561_left.jpg,[],['pathologic myopia']
+196,08_ODIR-5K/preprocessed_images/1562_right.jpg,[],['pathologic myopia']
+197,08_ODIR-5K/preprocessed_images/1562_left.jpg,[],['pathologic myopia']
+198,08_ODIR-5K/preprocessed_images/1563_right.jpg,[],['pathologic myopia']
+199,08_ODIR-5K/preprocessed_images/1563_left.jpg,[],['pathologic myopia']
+200,08_ODIR-5K/preprocessed_images/1564_right.jpg,[],['pathologic myopia']
+201,08_ODIR-5K/preprocessed_images/1564_left.jpg,[],['pathologic myopia']
+202,08_ODIR-5K/preprocessed_images/1565_right.jpg,[],['pathologic myopia']
+203,08_ODIR-5K/preprocessed_images/1565_left.jpg,[],['pathologic myopia']
+204,08_ODIR-5K/preprocessed_images/1568_right.jpg,[],['pathologic myopia']
+205,08_ODIR-5K/preprocessed_images/1568_left.jpg,[],['pathologic myopia']
+206,08_ODIR-5K/preprocessed_images/1569_right.jpg,[],['pathologic myopia']
+207,08_ODIR-5K/preprocessed_images/1569_left.jpg,[],['normal']
+208,08_ODIR-5K/preprocessed_images/1570_right.jpg,[],['pathologic myopia']
+209,08_ODIR-5K/preprocessed_images/1570_left.jpg,[],['pathologic myopia']
+210,08_ODIR-5K/preprocessed_images/1572_right.jpg,[],['pathologic myopia']
+211,08_ODIR-5K/preprocessed_images/1572_left.jpg,[],['pathologic myopia']
+212,08_ODIR-5K/preprocessed_images/1576_right.jpg,[],['pathologic myopia']
+213,08_ODIR-5K/preprocessed_images/1576_left.jpg,[],['pathologic myopia']
+214,08_ODIR-5K/preprocessed_images/1578_right.jpg,[],['pathologic myopia']
+215,08_ODIR-5K/preprocessed_images/1578_left.jpg,[],['pathologic myopia']
+216,08_ODIR-5K/preprocessed_images/1579_right.jpg,[],['pathologic myopia']
+217,08_ODIR-5K/preprocessed_images/1579_left.jpg,[],['pathologic myopia']
+218,08_ODIR-5K/preprocessed_images/1580_right.jpg,[],['pathologic myopia']
+219,08_ODIR-5K/preprocessed_images/1580_left.jpg,[],['pathologic myopia']
+220,08_ODIR-5K/preprocessed_images/1583_right.jpg,[],['pathologic myopia']
+221,08_ODIR-5K/preprocessed_images/1583_left.jpg,[],['pathologic myopia']
+222,08_ODIR-5K/preprocessed_images/1584_right.jpg,[],['pathologic myopia']
+223,08_ODIR-5K/preprocessed_images/1584_left.jpg,[],['pathologic myopia']
+224,08_ODIR-5K/preprocessed_images/1586_right.jpg,[],['pathologic myopia']
+225,08_ODIR-5K/preprocessed_images/1586_left.jpg,[],['pathologic myopia']
+226,08_ODIR-5K/preprocessed_images/1588_right.jpg,[],['pathologic myopia']
+227,08_ODIR-5K/preprocessed_images/1588_left.jpg,[],['pathologic myopia']
+228,08_ODIR-5K/preprocessed_images/1589_right.jpg,[],['pathologic myopia']
+229,08_ODIR-5K/preprocessed_images/1589_left.jpg,[],['pathologic myopia']
+230,08_ODIR-5K/preprocessed_images/1590_right.jpg,[],['pathologic myopia']
+231,08_ODIR-5K/preprocessed_images/1590_left.jpg,[],['normal']
+232,08_ODIR-5K/preprocessed_images/1593_right.jpg,[],['pathologic myopia']
+233,08_ODIR-5K/preprocessed_images/1593_left.jpg,[],['pathologic myopia']
+234,08_ODIR-5K/preprocessed_images/1594_right.jpg,[],['pathologic myopia']
+235,08_ODIR-5K/preprocessed_images/1594_left.jpg,[],['pathologic myopia']
+236,08_ODIR-5K/preprocessed_images/1599_right.jpg,[],['pathologic myopia']
+237,08_ODIR-5K/preprocessed_images/1599_left.jpg,[],['pathologic myopia']
+238,08_ODIR-5K/preprocessed_images/1600_right.jpg,[],['pathologic myopia']
+239,08_ODIR-5K/preprocessed_images/1600_left.jpg,[],['pathologic myopia']
+240,08_ODIR-5K/preprocessed_images/1601_right.jpg,[],['pathologic myopia']
+241,08_ODIR-5K/preprocessed_images/1601_left.jpg,[],['pathologic myopia']
+242,08_ODIR-5K/preprocessed_images/1602_right.jpg,[],['pathologic myopia']
+243,08_ODIR-5K/preprocessed_images/1602_left.jpg,[],['pathologic myopia']
+244,08_ODIR-5K/preprocessed_images/1603_right.jpg,[],['pathologic myopia']
+245,08_ODIR-5K/preprocessed_images/1603_left.jpg,[],['pathologic myopia']
+246,08_ODIR-5K/preprocessed_images/1605_right.jpg,[],['pathologic myopia']
+247,08_ODIR-5K/preprocessed_images/1605_left.jpg,[],['pathologic myopia']
+248,08_ODIR-5K/preprocessed_images/1606_right.jpg,[],['pathologic myopia']
+249,08_ODIR-5K/preprocessed_images/1606_left.jpg,[],['pathologic myopia']
+250,08_ODIR-5K/preprocessed_images/1607_right.jpg,[],['pathologic myopia']
+251,08_ODIR-5K/preprocessed_images/1607_left.jpg,[],['pathologic myopia']
+252,08_ODIR-5K/preprocessed_images/1610_right.jpg,[],['normal']
+253,08_ODIR-5K/preprocessed_images/1610_left.jpg,[],['pathologic myopia']
+254,08_ODIR-5K/preprocessed_images/1611_right.jpg,[],['normal']
+255,08_ODIR-5K/preprocessed_images/1611_left.jpg,[],['pathologic myopia']
+256,08_ODIR-5K/preprocessed_images/1612_right.jpg,[],['pathologic myopia']
+257,08_ODIR-5K/preprocessed_images/1612_left.jpg,[],['pathologic myopia']
+258,08_ODIR-5K/preprocessed_images/1613_right.jpg,[],['pathologic myopia']
+259,08_ODIR-5K/preprocessed_images/1613_left.jpg,[],['pathologic myopia']
+260,08_ODIR-5K/preprocessed_images/1615_right.jpg,[],['pathologic myopia']
+261,08_ODIR-5K/preprocessed_images/1615_left.jpg,[],['pathologic myopia']
+262,08_ODIR-5K/preprocessed_images/1616_right.jpg,[],['pathologic myopia']
+263,08_ODIR-5K/preprocessed_images/1616_left.jpg,[],['pathologic myopia']
+264,08_ODIR-5K/preprocessed_images/1617_right.jpg,[],['pathologic myopia']
+265,08_ODIR-5K/preprocessed_images/1617_left.jpg,[],['pathologic myopia']
+266,08_ODIR-5K/preprocessed_images/1618_right.jpg,[],['pathologic myopia']
+267,08_ODIR-5K/preprocessed_images/1618_left.jpg,[],['pathologic myopia']
+268,08_ODIR-5K/preprocessed_images/1619_right.jpg,[],['pathologic myopia']
+269,08_ODIR-5K/preprocessed_images/1619_left.jpg,[],['normal']
+270,08_ODIR-5K/preprocessed_images/1620_right.jpg,[],['pathologic myopia']
+271,08_ODIR-5K/preprocessed_images/1620_left.jpg,[],['normal']
+272,08_ODIR-5K/preprocessed_images/1621_right.jpg,[],['pathologic myopia']
+273,08_ODIR-5K/preprocessed_images/1621_left.jpg,[],['pathologic myopia']
+274,08_ODIR-5K/preprocessed_images/1623_right.jpg,[],['pathologic myopia']
+275,08_ODIR-5K/preprocessed_images/1623_left.jpg,[],['pathologic myopia']
+276,08_ODIR-5K/preprocessed_images/1624_right.jpg,[],['pathologic myopia']
+277,08_ODIR-5K/preprocessed_images/1624_left.jpg,[],['pathologic myopia']
+278,08_ODIR-5K/preprocessed_images/1629_right.jpg,[],['pathologic myopia']
+279,08_ODIR-5K/preprocessed_images/1629_left.jpg,[],['pathologic myopia']
+280,08_ODIR-5K/preprocessed_images/1630_right.jpg,[],['pathologic myopia']
+281,08_ODIR-5K/preprocessed_images/1630_left.jpg,[],['pathologic myopia']
+282,08_ODIR-5K/preprocessed_images/1647_right.jpg,[],['pathologic myopia']
+283,08_ODIR-5K/preprocessed_images/1647_left.jpg,[],['pathologic myopia']
+284,08_ODIR-5K/preprocessed_images/1663_right.jpg,[],['normal']
+285,08_ODIR-5K/preprocessed_images/1663_left.jpg,[],['pathologic myopia']
+286,08_ODIR-5K/preprocessed_images/1666_right.jpg,[],['pathologic myopia']
+287,08_ODIR-5K/preprocessed_images/1666_left.jpg,[],['normal']
+288,08_ODIR-5K/preprocessed_images/1686_right.jpg,[],['pathologic myopia']
+289,08_ODIR-5K/preprocessed_images/1686_left.jpg,[],['pathologic myopia']
+290,08_ODIR-5K/preprocessed_images/2095_right.jpg,[],['cataract']
+291,08_ODIR-5K/preprocessed_images/2095_left.jpg,[],['normal']
+292,08_ODIR-5K/preprocessed_images/2097_right.jpg,[],['cataract']
+293,08_ODIR-5K/preprocessed_images/2097_left.jpg,[],['normal']
+294,08_ODIR-5K/preprocessed_images/2098_right.jpg,[],['cataract']
+295,08_ODIR-5K/preprocessed_images/2098_left.jpg,[],['cataract']
+296,08_ODIR-5K/preprocessed_images/2101_right.jpg,[],['cataract']
+297,08_ODIR-5K/preprocessed_images/2101_left.jpg,[],['cataract']
+298,08_ODIR-5K/preprocessed_images/2104_right.jpg,[],['cataract']
+299,08_ODIR-5K/preprocessed_images/2104_left.jpg,[],['cataract']
+300,08_ODIR-5K/preprocessed_images/2105_right.jpg,[],['cataract']
+301,08_ODIR-5K/preprocessed_images/2105_left.jpg,[],['cataract']
+302,08_ODIR-5K/preprocessed_images/2106_right.jpg,[],['cataract']
+303,08_ODIR-5K/preprocessed_images/2106_left.jpg,[],['normal']
+304,08_ODIR-5K/preprocessed_images/2108_right.jpg,[],['cataract']
+305,08_ODIR-5K/preprocessed_images/2108_left.jpg,[],['cataract']
+306,08_ODIR-5K/preprocessed_images/2109_right.jpg,[],['cataract']
+307,08_ODIR-5K/preprocessed_images/2109_left.jpg,[],['cataract']
+308,08_ODIR-5K/preprocessed_images/2110_right.jpg,[],['cataract']
+309,08_ODIR-5K/preprocessed_images/2110_left.jpg,[],['cataract']
+310,08_ODIR-5K/preprocessed_images/2112_right.jpg,[],['cataract']
+311,08_ODIR-5K/preprocessed_images/2112_left.jpg,[],['normal']
+312,08_ODIR-5K/preprocessed_images/2113_right.jpg,[],['cataract']
+313,08_ODIR-5K/preprocessed_images/2113_left.jpg,[],['cataract']
+314,08_ODIR-5K/preprocessed_images/2114_right.jpg,[],['cataract']
+315,08_ODIR-5K/preprocessed_images/2114_left.jpg,[],['cataract']
+316,08_ODIR-5K/preprocessed_images/2116_right.jpg,[],['cataract']
+317,08_ODIR-5K/preprocessed_images/2116_left.jpg,[],['cataract']
+318,08_ODIR-5K/preprocessed_images/2119_right.jpg,[],['cataract']
+319,08_ODIR-5K/preprocessed_images/2119_left.jpg,[],['normal']
+320,08_ODIR-5K/preprocessed_images/2120_right.jpg,[],['cataract']
+321,08_ODIR-5K/preprocessed_images/2120_left.jpg,[],['cataract']
+322,08_ODIR-5K/preprocessed_images/2121_right.jpg,[],['normal']
+323,08_ODIR-5K/preprocessed_images/2121_left.jpg,[],['cataract']
+324,08_ODIR-5K/preprocessed_images/2122_right.jpg,[],['cataract']
+325,08_ODIR-5K/preprocessed_images/2122_left.jpg,[],['cataract']
+326,08_ODIR-5K/preprocessed_images/2125_right.jpg,[],['normal']
+327,08_ODIR-5K/preprocessed_images/2125_left.jpg,[],['cataract']
+328,08_ODIR-5K/preprocessed_images/2126_right.jpg,[],['cataract']
+329,08_ODIR-5K/preprocessed_images/2126_left.jpg,[],['cataract']
+330,08_ODIR-5K/preprocessed_images/2127_right.jpg,[],['cataract']
+331,08_ODIR-5K/preprocessed_images/2127_left.jpg,[],['cataract']
+332,08_ODIR-5K/preprocessed_images/2128_right.jpg,[],['cataract']
+333,08_ODIR-5K/preprocessed_images/2128_left.jpg,[],['cataract']
+334,08_ODIR-5K/preprocessed_images/2129_right.jpg,[],['cataract']
+335,08_ODIR-5K/preprocessed_images/2129_left.jpg,[],['cataract']
+336,08_ODIR-5K/preprocessed_images/2130_right.jpg,[],['cataract']
+337,08_ODIR-5K/preprocessed_images/2130_left.jpg,[],['cataract']
+338,08_ODIR-5K/preprocessed_images/2132_right.jpg,[],['normal']
+339,08_ODIR-5K/preprocessed_images/2132_left.jpg,[],['cataract']
+340,08_ODIR-5K/preprocessed_images/2133_right.jpg,[],['cataract']
+341,08_ODIR-5K/preprocessed_images/2133_left.jpg,[],['cataract']
+342,08_ODIR-5K/preprocessed_images/2134_right.jpg,[],['cataract']
+343,08_ODIR-5K/preprocessed_images/2134_left.jpg,[],['cataract']
+344,08_ODIR-5K/preprocessed_images/2135_right.jpg,[],['cataract']
+345,08_ODIR-5K/preprocessed_images/2135_left.jpg,[],['cataract']
+346,08_ODIR-5K/preprocessed_images/2136_right.jpg,[],['normal']
+347,08_ODIR-5K/preprocessed_images/2136_left.jpg,[],['cataract']
+348,08_ODIR-5K/preprocessed_images/2137_right.jpg,[],['cataract']
+349,08_ODIR-5K/preprocessed_images/2137_left.jpg,[],['cataract']
+350,08_ODIR-5K/preprocessed_images/2138_right.jpg,[],['cataract']
+351,08_ODIR-5K/preprocessed_images/2138_left.jpg,[],['cataract']
+352,08_ODIR-5K/preprocessed_images/2139_right.jpg,[],['cataract']
+353,08_ODIR-5K/preprocessed_images/2140_right.jpg,[],['cataract']
+354,08_ODIR-5K/preprocessed_images/2140_left.jpg,[],['cataract']
+355,08_ODIR-5K/preprocessed_images/2142_right.jpg,[],['cataract']
+356,08_ODIR-5K/preprocessed_images/2142_left.jpg,[],['cataract']
+357,08_ODIR-5K/preprocessed_images/2143_right.jpg,[],['cataract']
+358,08_ODIR-5K/preprocessed_images/2143_left.jpg,[],['normal']
+359,08_ODIR-5K/preprocessed_images/2144_right.jpg,[],['cataract']
+360,08_ODIR-5K/preprocessed_images/2144_left.jpg,[],['cataract']
+361,08_ODIR-5K/preprocessed_images/2146_right.jpg,[],['cataract']
+362,08_ODIR-5K/preprocessed_images/2146_left.jpg,[],['cataract']
+363,08_ODIR-5K/preprocessed_images/2149_right.jpg,[],['cataract']
+364,08_ODIR-5K/preprocessed_images/2149_left.jpg,[],['normal']
+365,08_ODIR-5K/preprocessed_images/2150_right.jpg,[],['cataract']
+366,08_ODIR-5K/preprocessed_images/2150_left.jpg,[],['cataract']
+367,08_ODIR-5K/preprocessed_images/2151_right.jpg,[],['cataract']
+368,08_ODIR-5K/preprocessed_images/2151_left.jpg,[],['cataract']
+369,08_ODIR-5K/preprocessed_images/2152_right.jpg,[],['normal']
+370,08_ODIR-5K/preprocessed_images/2152_left.jpg,[],['cataract']
+371,08_ODIR-5K/preprocessed_images/2153_right.jpg,[],['cataract']
+372,08_ODIR-5K/preprocessed_images/2153_left.jpg,[],['cataract']
+373,08_ODIR-5K/preprocessed_images/2154_right.jpg,[],['normal']
+374,08_ODIR-5K/preprocessed_images/2154_left.jpg,[],['cataract']
+375,08_ODIR-5K/preprocessed_images/2155_right.jpg,[],['cataract']
+376,08_ODIR-5K/preprocessed_images/2155_left.jpg,[],['cataract']
+377,08_ODIR-5K/preprocessed_images/2157_right.jpg,[],['cataract']
+378,08_ODIR-5K/preprocessed_images/2157_left.jpg,[],['cataract']
+379,08_ODIR-5K/preprocessed_images/2158_right.jpg,[],['cataract']
+380,08_ODIR-5K/preprocessed_images/2158_left.jpg,[],['cataract']
+381,08_ODIR-5K/preprocessed_images/2160_right.jpg,[],['cataract']
+382,08_ODIR-5K/preprocessed_images/2160_left.jpg,[],['normal']
+383,08_ODIR-5K/preprocessed_images/2162_right.jpg,[],['cataract']
+384,08_ODIR-5K/preprocessed_images/2162_left.jpg,[],['normal']
+385,08_ODIR-5K/preprocessed_images/2163_right.jpg,[],['cataract']
+386,08_ODIR-5K/preprocessed_images/2163_left.jpg,[],['cataract']
+387,08_ODIR-5K/preprocessed_images/2167_right.jpg,[],['cataract']
+388,08_ODIR-5K/preprocessed_images/2167_left.jpg,[],['cataract']
+389,08_ODIR-5K/preprocessed_images/2169_right.jpg,[],['cataract']
+390,08_ODIR-5K/preprocessed_images/2169_left.jpg,[],['cataract']
+391,08_ODIR-5K/preprocessed_images/2170_right.jpg,[],['cataract']
+392,08_ODIR-5K/preprocessed_images/2170_left.jpg,[],['cataract']
+393,08_ODIR-5K/preprocessed_images/2171_right.jpg,[],['normal']
+394,08_ODIR-5K/preprocessed_images/2171_left.jpg,[],['cataract']
+395,08_ODIR-5K/preprocessed_images/2172_right.jpg,[],['normal']
+396,08_ODIR-5K/preprocessed_images/2172_left.jpg,[],['cataract']
+397,08_ODIR-5K/preprocessed_images/2174_right.jpg,[],['cataract']
+398,08_ODIR-5K/preprocessed_images/2174_left.jpg,[],['normal']
+399,08_ODIR-5K/preprocessed_images/2175_right.jpg,[],['normal']
+400,08_ODIR-5K/preprocessed_images/2175_left.jpg,[],['cataract']
+401,08_ODIR-5K/preprocessed_images/2176_right.jpg,[],['cataract']
+402,08_ODIR-5K/preprocessed_images/2176_left.jpg,[],['cataract']
+403,08_ODIR-5K/preprocessed_images/2177_right.jpg,[],['cataract']
+404,08_ODIR-5K/preprocessed_images/2177_left.jpg,[],['normal']
+405,08_ODIR-5K/preprocessed_images/2178_right.jpg,[],['cataract']
+406,08_ODIR-5K/preprocessed_images/2178_left.jpg,[],['cataract']
+407,08_ODIR-5K/preprocessed_images/2179_right.jpg,[],['cataract']
+408,08_ODIR-5K/preprocessed_images/2179_left.jpg,[],['cataract']
+409,08_ODIR-5K/preprocessed_images/2180_right.jpg,[],['cataract']
+410,08_ODIR-5K/preprocessed_images/2180_left.jpg,[],['cataract']
+411,08_ODIR-5K/preprocessed_images/2181_right.jpg,[],['cataract']
+412,08_ODIR-5K/preprocessed_images/2181_left.jpg,[],['cataract']
+413,08_ODIR-5K/preprocessed_images/2182_right.jpg,[],['cataract']
+414,08_ODIR-5K/preprocessed_images/2182_left.jpg,[],['cataract']
+415,08_ODIR-5K/preprocessed_images/2183_right.jpg,[],['cataract']
+416,08_ODIR-5K/preprocessed_images/2183_left.jpg,[],['cataract']
+417,08_ODIR-5K/preprocessed_images/2184_right.jpg,[],['cataract']
+418,08_ODIR-5K/preprocessed_images/2184_left.jpg,[],['cataract']
+419,08_ODIR-5K/preprocessed_images/2187_right.jpg,[],['cataract']
+420,08_ODIR-5K/preprocessed_images/2187_left.jpg,[],['normal']
+421,08_ODIR-5K/preprocessed_images/2189_right.jpg,[],['cataract']
+422,08_ODIR-5K/preprocessed_images/2189_left.jpg,[],['cataract']
+423,08_ODIR-5K/preprocessed_images/2190_right.jpg,[],['cataract']
+424,08_ODIR-5K/preprocessed_images/2190_left.jpg,[],['normal']
+425,08_ODIR-5K/preprocessed_images/2191_right.jpg,[],['cataract']
+426,08_ODIR-5K/preprocessed_images/2191_left.jpg,[],['normal']
+427,08_ODIR-5K/preprocessed_images/2192_right.jpg,[],['cataract']
+428,08_ODIR-5K/preprocessed_images/2192_left.jpg,[],['cataract']
+429,08_ODIR-5K/preprocessed_images/2193_right.jpg,[],['cataract']
+430,08_ODIR-5K/preprocessed_images/2193_left.jpg,[],['normal']
+431,08_ODIR-5K/preprocessed_images/2194_right.jpg,[],['cataract']
+432,08_ODIR-5K/preprocessed_images/2194_left.jpg,[],['cataract']
+433,08_ODIR-5K/preprocessed_images/2195_right.jpg,[],['cataract']
+434,08_ODIR-5K/preprocessed_images/2195_left.jpg,[],['cataract']
+435,08_ODIR-5K/preprocessed_images/2196_right.jpg,[],['cataract']
+436,08_ODIR-5K/preprocessed_images/2196_left.jpg,[],['normal']
+437,08_ODIR-5K/preprocessed_images/2197_right.jpg,[],['cataract']
+438,08_ODIR-5K/preprocessed_images/2197_left.jpg,[],['normal']
+439,08_ODIR-5K/preprocessed_images/2198_right.jpg,[],['cataract']
+440,08_ODIR-5K/preprocessed_images/2198_left.jpg,[],['cataract']
+441,08_ODIR-5K/preprocessed_images/2203_right.jpg,[],['cataract']
+442,08_ODIR-5K/preprocessed_images/2203_left.jpg,[],['cataract']
+443,08_ODIR-5K/preprocessed_images/2205_right.jpg,[],['cataract']
+444,08_ODIR-5K/preprocessed_images/2205_left.jpg,[],['normal']
+445,08_ODIR-5K/preprocessed_images/2206_right.jpg,[],['cataract']
+446,08_ODIR-5K/preprocessed_images/2206_left.jpg,[],['cataract']
+447,08_ODIR-5K/preprocessed_images/2207_right.jpg,[],['cataract']
+448,08_ODIR-5K/preprocessed_images/2207_left.jpg,[],['cataract']
+449,08_ODIR-5K/preprocessed_images/2208_right.jpg,[],['cataract']
+450,08_ODIR-5K/preprocessed_images/2208_left.jpg,[],['normal']
+451,08_ODIR-5K/preprocessed_images/2209_right.jpg,[],['cataract']
+452,08_ODIR-5K/preprocessed_images/2209_left.jpg,[],['cataract']
+453,08_ODIR-5K/preprocessed_images/2210_right.jpg,[],['cataract']
+454,08_ODIR-5K/preprocessed_images/2210_left.jpg,[],['cataract']
+455,08_ODIR-5K/preprocessed_images/2211_right.jpg,[],['cataract']
+456,08_ODIR-5K/preprocessed_images/2211_left.jpg,[],['cataract']
+457,08_ODIR-5K/preprocessed_images/2212_right.jpg,[],['cataract']
+458,08_ODIR-5K/preprocessed_images/2212_left.jpg,[],['cataract']
+459,08_ODIR-5K/preprocessed_images/2213_right.jpg,[],['cataract']
+460,08_ODIR-5K/preprocessed_images/2213_left.jpg,[],['cataract']
+461,08_ODIR-5K/preprocessed_images/2215_right.jpg,[],['normal']
+462,08_ODIR-5K/preprocessed_images/2215_left.jpg,[],['cataract']
+463,08_ODIR-5K/preprocessed_images/2216_right.jpg,[],['cataract']
+464,08_ODIR-5K/preprocessed_images/2216_left.jpg,[],['cataract']
+465,08_ODIR-5K/preprocessed_images/2217_right.jpg,[],['cataract']
+466,08_ODIR-5K/preprocessed_images/2217_left.jpg,[],['cataract']
+467,08_ODIR-5K/preprocessed_images/2218_right.jpg,[],['cataract']
+468,08_ODIR-5K/preprocessed_images/2218_left.jpg,[],['normal']
+469,08_ODIR-5K/preprocessed_images/2219_right.jpg,[],['cataract']
+470,08_ODIR-5K/preprocessed_images/2219_left.jpg,[],['cataract']
+471,08_ODIR-5K/preprocessed_images/2221_right.jpg,[],['cataract']
+472,08_ODIR-5K/preprocessed_images/2221_left.jpg,[],['cataract']
+473,08_ODIR-5K/preprocessed_images/2223_right.jpg,[],['cataract']
+474,08_ODIR-5K/preprocessed_images/2223_left.jpg,[],['cataract']
+475,08_ODIR-5K/preprocessed_images/2225_right.jpg,[],['cataract']
+476,08_ODIR-5K/preprocessed_images/2225_left.jpg,[],['cataract']
+477,08_ODIR-5K/preprocessed_images/2227_right.jpg,[],['cataract']
+478,08_ODIR-5K/preprocessed_images/2227_left.jpg,[],['cataract']
+479,08_ODIR-5K/preprocessed_images/2231_right.jpg,[],['cataract']
+480,08_ODIR-5K/preprocessed_images/2232_right.jpg,[],['cataract']
+481,08_ODIR-5K/preprocessed_images/2232_left.jpg,[],['cataract']
+482,08_ODIR-5K/preprocessed_images/2233_right.jpg,[],['cataract']
+483,08_ODIR-5K/preprocessed_images/2235_right.jpg,[],['normal']
+484,08_ODIR-5K/preprocessed_images/2237_left.jpg,[],['normal']
+485,08_ODIR-5K/preprocessed_images/2329_right.jpg,[],['normal']
+486,08_ODIR-5K/preprocessed_images/2329_left.jpg,[],['normal']
+487,08_ODIR-5K/preprocessed_images/2330_right.jpg,[],['normal']
+488,08_ODIR-5K/preprocessed_images/2330_left.jpg,[],['normal']
+489,08_ODIR-5K/preprocessed_images/2331_right.jpg,[],['normal']
+490,08_ODIR-5K/preprocessed_images/2331_left.jpg,[],['normal']
+491,08_ODIR-5K/preprocessed_images/2332_right.jpg,[],['normal']
+492,08_ODIR-5K/preprocessed_images/2332_left.jpg,[],['normal']
+493,08_ODIR-5K/preprocessed_images/2333_right.jpg,[],['normal']
+494,08_ODIR-5K/preprocessed_images/2333_left.jpg,[],['normal']
+495,08_ODIR-5K/preprocessed_images/2334_right.jpg,[],['normal']
+496,08_ODIR-5K/preprocessed_images/2334_left.jpg,[],['normal']
+497,08_ODIR-5K/preprocessed_images/2335_right.jpg,[],['normal']
+498,08_ODIR-5K/preprocessed_images/2335_left.jpg,[],['normal']
+499,08_ODIR-5K/preprocessed_images/2336_right.jpg,[],['normal']
+500,08_ODIR-5K/preprocessed_images/2336_left.jpg,[],['normal']
+501,08_ODIR-5K/preprocessed_images/2337_right.jpg,[],['normal']
+502,08_ODIR-5K/preprocessed_images/2337_left.jpg,[],['normal']
+503,08_ODIR-5K/preprocessed_images/2338_right.jpg,[],['normal']
+504,08_ODIR-5K/preprocessed_images/2338_left.jpg,[],['normal']
+505,08_ODIR-5K/preprocessed_images/2339_right.jpg,[],['normal']
+506,08_ODIR-5K/preprocessed_images/2339_left.jpg,[],['normal']
+507,08_ODIR-5K/preprocessed_images/2340_right.jpg,[],['normal']
+508,08_ODIR-5K/preprocessed_images/2340_left.jpg,[],['normal']
+509,08_ODIR-5K/preprocessed_images/2341_right.jpg,[],['normal']
+510,08_ODIR-5K/preprocessed_images/2341_left.jpg,[],['normal']
+511,08_ODIR-5K/preprocessed_images/2342_right.jpg,[],['normal']
+512,08_ODIR-5K/preprocessed_images/2342_left.jpg,[],['normal']
+513,08_ODIR-5K/preprocessed_images/2343_right.jpg,[],['normal']
+514,08_ODIR-5K/preprocessed_images/2343_left.jpg,[],['normal']
+515,08_ODIR-5K/preprocessed_images/2345_right.jpg,[],['normal']
+516,08_ODIR-5K/preprocessed_images/2345_left.jpg,[],['normal']
+517,08_ODIR-5K/preprocessed_images/2346_right.jpg,[],['normal']
+518,08_ODIR-5K/preprocessed_images/2346_left.jpg,[],['normal']
+519,08_ODIR-5K/preprocessed_images/2347_right.jpg,[],['normal']
+520,08_ODIR-5K/preprocessed_images/2347_left.jpg,[],['normal']
+521,08_ODIR-5K/preprocessed_images/2348_right.jpg,[],['normal']
+522,08_ODIR-5K/preprocessed_images/2348_left.jpg,[],['normal']
+523,08_ODIR-5K/preprocessed_images/2349_right.jpg,[],['normal']
+524,08_ODIR-5K/preprocessed_images/2349_left.jpg,[],['normal']
+525,08_ODIR-5K/preprocessed_images/2351_right.jpg,[],['normal']
+526,08_ODIR-5K/preprocessed_images/2351_left.jpg,[],['normal']
+527,08_ODIR-5K/preprocessed_images/2352_right.jpg,[],['normal']
+528,08_ODIR-5K/preprocessed_images/2352_left.jpg,[],['normal']
+529,08_ODIR-5K/preprocessed_images/2353_right.jpg,[],['normal']
+530,08_ODIR-5K/preprocessed_images/2353_left.jpg,[],['normal']
+531,08_ODIR-5K/preprocessed_images/2354_right.jpg,[],['normal']
+532,08_ODIR-5K/preprocessed_images/2354_left.jpg,[],['normal']
+533,08_ODIR-5K/preprocessed_images/2355_right.jpg,[],['normal']
+534,08_ODIR-5K/preprocessed_images/2355_left.jpg,[],['normal']
+535,08_ODIR-5K/preprocessed_images/2356_right.jpg,[],['normal']
+536,08_ODIR-5K/preprocessed_images/2356_left.jpg,[],['normal']
+537,08_ODIR-5K/preprocessed_images/2357_right.jpg,[],['normal']
+538,08_ODIR-5K/preprocessed_images/2357_left.jpg,[],['normal']
+539,08_ODIR-5K/preprocessed_images/2359_right.jpg,[],['normal']
+540,08_ODIR-5K/preprocessed_images/2359_left.jpg,[],['normal']
+541,08_ODIR-5K/preprocessed_images/2360_right.jpg,[],['normal']
+542,08_ODIR-5K/preprocessed_images/2360_left.jpg,[],['normal']
+543,08_ODIR-5K/preprocessed_images/2361_right.jpg,[],['normal']
+544,08_ODIR-5K/preprocessed_images/2361_left.jpg,[],['normal']
+545,08_ODIR-5K/preprocessed_images/2362_right.jpg,[],['normal']
+546,08_ODIR-5K/preprocessed_images/2362_left.jpg,[],['normal']
+547,08_ODIR-5K/preprocessed_images/2363_right.jpg,[],['normal']
+548,08_ODIR-5K/preprocessed_images/2363_left.jpg,[],['normal']
+549,08_ODIR-5K/preprocessed_images/2364_right.jpg,[],['normal']
+550,08_ODIR-5K/preprocessed_images/2364_left.jpg,[],['normal']
+551,08_ODIR-5K/preprocessed_images/2365_right.jpg,[],['normal']
+552,08_ODIR-5K/preprocessed_images/2365_left.jpg,[],['normal']
+553,08_ODIR-5K/preprocessed_images/2366_right.jpg,[],['normal']
+554,08_ODIR-5K/preprocessed_images/2366_left.jpg,[],['normal']
+555,08_ODIR-5K/preprocessed_images/2367_right.jpg,[],['normal']
+556,08_ODIR-5K/preprocessed_images/2367_left.jpg,[],['normal']
+557,08_ODIR-5K/preprocessed_images/2368_right.jpg,[],['normal']
+558,08_ODIR-5K/preprocessed_images/2368_left.jpg,[],['normal']
+559,08_ODIR-5K/preprocessed_images/2369_right.jpg,[],['normal']
+560,08_ODIR-5K/preprocessed_images/2369_left.jpg,[],['normal']
+561,08_ODIR-5K/preprocessed_images/2370_right.jpg,[],['normal']
+562,08_ODIR-5K/preprocessed_images/2370_left.jpg,[],['normal']
+563,08_ODIR-5K/preprocessed_images/2371_right.jpg,[],['normal']
+564,08_ODIR-5K/preprocessed_images/2371_left.jpg,[],['normal']
+565,08_ODIR-5K/preprocessed_images/2372_right.jpg,[],['normal']
+566,08_ODIR-5K/preprocessed_images/2372_left.jpg,[],['normal']
+567,08_ODIR-5K/preprocessed_images/2373_right.jpg,[],['normal']
+568,08_ODIR-5K/preprocessed_images/2373_left.jpg,[],['normal']
+569,08_ODIR-5K/preprocessed_images/2374_right.jpg,[],['normal']
+570,08_ODIR-5K/preprocessed_images/2374_left.jpg,[],['normal']
+571,08_ODIR-5K/preprocessed_images/2375_right.jpg,[],['normal']
+572,08_ODIR-5K/preprocessed_images/2375_left.jpg,[],['normal']
+573,08_ODIR-5K/preprocessed_images/2376_right.jpg,[],['normal']
+574,08_ODIR-5K/preprocessed_images/2376_left.jpg,[],['normal']
+575,08_ODIR-5K/preprocessed_images/2378_right.jpg,[],['normal']
+576,08_ODIR-5K/preprocessed_images/2378_left.jpg,[],['normal']
+577,08_ODIR-5K/preprocessed_images/2379_right.jpg,[],['normal']
+578,08_ODIR-5K/preprocessed_images/2379_left.jpg,[],['normal']
+579,08_ODIR-5K/preprocessed_images/2380_right.jpg,[],['normal']
+580,08_ODIR-5K/preprocessed_images/13_right.jpg,[],['pathologic myopia']
+581,08_ODIR-5K/preprocessed_images/13_left.jpg,[],['pathologic myopia']
+582,08_ODIR-5K/preprocessed_images/16_right.jpg,[],['pathologic myopia']
+583,08_ODIR-5K/preprocessed_images/18_right.jpg,[],['pathologic myopia']
+584,08_ODIR-5K/preprocessed_images/18_left.jpg,[],['pathologic myopia']
+585,08_ODIR-5K/preprocessed_images/35_left.jpg,[],['pathologic myopia']
+586,08_ODIR-5K/preprocessed_images/39_left.jpg,[],['pathologic myopia']
+587,08_ODIR-5K/preprocessed_images/106_right.jpg,[],['pathologic myopia']
+588,08_ODIR-5K/preprocessed_images/106_left.jpg,[],['pathologic myopia']
+589,08_ODIR-5K/preprocessed_images/144_left.jpg,[],['pathologic myopia']
+590,08_ODIR-5K/preprocessed_images/145_right.jpg,[],['pathologic myopia']
+591,08_ODIR-5K/preprocessed_images/145_left.jpg,[],['pathologic myopia']
+592,08_ODIR-5K/preprocessed_images/174_left.jpg,[],['pathologic myopia']
+593,08_ODIR-5K/preprocessed_images/214_right.jpg,[],['pathologic myopia']
+594,08_ODIR-5K/preprocessed_images/233_right.jpg,[],['pathologic myopia']
+595,08_ODIR-5K/preprocessed_images/379_left.jpg,[],['pathologic myopia']
+596,08_ODIR-5K/preprocessed_images/382_right.jpg,[],['pathologic myopia']
+597,08_ODIR-5K/preprocessed_images/382_left.jpg,[],['pathologic myopia']
+598,08_ODIR-5K/preprocessed_images/390_right.jpg,[],['pathologic myopia']
+599,08_ODIR-5K/preprocessed_images/401_right.jpg,[],['pathologic myopia']

local_data/13_FIVES.csv

@@ -0,0 +1,801 @@
+,image,atributes,categories
+0,fives_retina/train/Original/139_A.png,[],['age related macular degeneration']
+1,fives_retina/train/Original/193_D.png,[],['diabetic retinopathy']
+2,fives_retina/train/Original/146_A.png,[],['age related macular degeneration']
+3,fives_retina/train/Original/329_G.png,[],['glaucoma']
+4,fives_retina/train/Original/3_A.png,[],['age related macular degeneration']
+5,fives_retina/train/Original/97_A.png,[],['age related macular degeneration']
+6,fives_retina/train/Original/361_G.png,[],['glaucoma']
+7,fives_retina/train/Original/299_D.png,[],['diabetic retinopathy']
+8,fives_retina/train/Original/378_G.png,[],['glaucoma']
+9,fives_retina/train/Original/96_A.png,[],['age related macular degeneration']
+10,fives_retina/train/Original/138_A.png,[],['age related macular degeneration']
+11,fives_retina/train/Original/536_N.png,[],['normal']
+12,fives_retina/train/Original/347_G.png,[],['glaucoma']
+13,fives_retina/train/Original/213_D.png,[],['diabetic retinopathy']
+14,fives_retina/train/Original/86_A.png,[],['age related macular degeneration']
+15,fives_retina/train/Original/437_G.png,[],['glaucoma']
+16,fives_retina/train/Original/429_G.png,[],['glaucoma']
+17,fives_retina/train/Original/534_N.png,[],['normal']
+18,fives_retina/train/Original/235_D.png,[],['diabetic retinopathy']
+19,fives_retina/train/Original/352_G.png,[],['glaucoma']
+20,fives_retina/train/Original/24_A.png,[],['age related macular degeneration']
+21,fives_retina/train/Original/135_A.png,[],['age related macular degeneration']
+22,fives_retina/train/Original/117_A.png,[],['age related macular degeneration']
+23,fives_retina/train/Original/401_G.png,[],['glaucoma']
+24,fives_retina/train/Original/476_N.png,[],['normal']
+25,fives_retina/train/Original/189_D.png,[],['diabetic retinopathy']
+26,fives_retina/train/Original/77_A.png,[],['age related macular degeneration']
+27,fives_retina/train/Original/120_A.png,[],['age related macular degeneration']
+28,fives_retina/train/Original/428_G.png,[],['glaucoma']
+29,fives_retina/train/Original/89_A.png,[],['age related macular degeneration']
+30,fives_retina/train/Original/9_A.png,[],['age related macular degeneration']
+31,fives_retina/train/Original/561_N.png,[],['normal']
+32,fives_retina/train/Original/449_G.png,[],['glaucoma']
+33,fives_retina/train/Original/508_N.png,[],['normal']
+34,fives_retina/train/Original/362_G.png,[],['glaucoma']
+35,fives_retina/train/Original/448_G.png,[],['glaucoma']
+36,fives_retina/train/Original/531_N.png,[],['normal']
+37,fives_retina/train/Original/8_A.png,[],['age related macular degeneration']
+38,fives_retina/train/Original/376_G.png,[],['glaucoma']
+39,fives_retina/train/Original/423_G.png,[],['glaucoma']
+40,fives_retina/train/Original/590_N.png,[],['normal']
+41,fives_retina/train/Original/462_N.png,[],['normal']
+42,fives_retina/train/Original/496_N.png,[],['normal']
+43,fives_retina/train/Original/207_D.png,[],['diabetic retinopathy']
+44,fives_retina/train/Original/475_N.png,[],['normal']
+45,fives_retina/train/Original/295_D.png,[],['diabetic retinopathy']
+46,fives_retina/train/Original/445_G.png,[],['glaucoma']
+47,fives_retina/train/Original/576_N.png,[],['normal']
+48,fives_retina/train/Original/257_D.png,[],['diabetic retinopathy']
+49,fives_retina/train/Original/470_N.png,[],['normal']
+50,fives_retina/train/Original/252_D.png,[],['diabetic retinopathy']
+51,fives_retina/train/Original/48_A.png,[],['age related macular degeneration']
+52,fives_retina/train/Original/390_G.png,[],['glaucoma']
+53,fives_retina/train/Original/372_G.png,[],['glaucoma']
+54,fives_retina/train/Original/479_N.png,[],['normal']
+55,fives_retina/train/Original/320_G.png,[],['glaucoma']
+56,fives_retina/train/Original/394_G.png,[],['glaucoma']
+57,fives_retina/train/Original/358_G.png,[],['glaucoma']
+58,fives_retina/train/Original/144_A.png,[],['age related macular degeneration']
+59,fives_retina/train/Original/586_N.png,[],['normal']
+60,fives_retina/train/Original/364_G.png,[],['glaucoma']
+61,fives_retina/train/Original/456_N.png,[],['normal']
+62,fives_retina/train/Original/219_D.png,[],['diabetic retinopathy']
+63,fives_retina/train/Original/230_D.png,[],['diabetic retinopathy']
+64,fives_retina/train/Original/345_G.png,[],['glaucoma']
+65,fives_retina/train/Original/579_N.png,[],['normal']
+66,fives_retina/train/Original/338_G.png,[],['glaucoma']
+67,fives_retina/train/Original/156_D.png,[],['diabetic retinopathy']
+68,fives_retina/train/Original/272_D.png,[],['diabetic retinopathy']
+69,fives_retina/train/Original/526_N.png,[],['normal']
+70,fives_retina/train/Original/261_D.png,[],['diabetic retinopathy']
+71,fives_retina/train/Original/18_A.png,[],['age related macular degeneration']
+72,fives_retina/train/Original/71_A.png,[],['age related macular degeneration']
+73,fives_retina/train/Original/350_G.png,[],['glaucoma']
+74,fives_retina/train/Original/564_N.png,[],['normal']
+75,fives_retina/train/Original/184_D.png,[],['diabetic retinopathy']
+76,fives_retina/train/Original/495_N.png,[],['normal']
+77,fives_retina/train/Original/540_N.png,[],['normal']
+78,fives_retina/train/Original/176_D.png,[],['diabetic retinopathy']
+79,fives_retina/train/Original/270_D.png,[],['diabetic retinopathy']
+80,fives_retina/train/Original/468_N.png,[],['normal']
+81,fives_retina/train/Original/181_D.png,[],['diabetic retinopathy']
+82,fives_retina/train/Original/260_D.png,[],['diabetic retinopathy']
+83,fives_retina/train/Original/205_D.png,[],['diabetic retinopathy']
+84,fives_retina/train/Original/37_A.png,[],['age related macular degeneration']
+85,fives_retina/train/Original/284_D.png,[],['diabetic retinopathy']
+86,fives_retina/train/Original/582_N.png,[],['normal']
+87,fives_retina/train/Original/99_A.png,[],['age related macular degeneration']
+88,fives_retina/train/Original/240_D.png,[],['diabetic retinopathy']
+89,fives_retina/train/Original/366_G.png,[],['glaucoma']
+90,fives_retina/train/Original/380_G.png,[],['glaucoma']
+91,fives_retina/train/Original/528_N.png,[],['normal']
+92,fives_retina/train/Original/35_A.png,[],['age related macular degeneration']
+93,fives_retina/train/Original/90_A.png,[],['age related macular degeneration']
+94,fives_retina/train/Original/182_D.png,[],['diabetic retinopathy']
+95,fives_retina/train/Original/516_N.png,[],['normal']
+96,fives_retina/train/Original/420_G.png,[],['glaucoma']
+97,fives_retina/train/Original/509_N.png,[],['normal']
+98,fives_retina/train/Original/503_N.png,[],['normal']
+99,fives_retina/train/Original/51_A.png,[],['age related macular degeneration']
+100,fives_retina/train/Original/517_N.png,[],['normal']
+101,fives_retina/train/Original/36_A.png,[],['age related macular degeneration']
+102,fives_retina/train/Original/229_D.png,[],['diabetic retinopathy']
+103,fives_retina/train/Original/430_G.png,[],['glaucoma']
+104,fives_retina/train/Original/370_G.png,[],['glaucoma']
+105,fives_retina/train/Original/94_A.png,[],['age related macular degeneration']
+106,fives_retina/train/Original/485_N.png,[],['normal']
+107,fives_retina/train/Original/149_A.png,[],['age related macular degeneration']
+108,fives_retina/train/Original/55_A.png,[],['age related macular degeneration']
+109,fives_retina/train/Original/123_A.png,[],['age related macular degeneration']
+110,fives_retina/train/Original/40_A.png,[],['age related macular degeneration']
+111,fives_retina/train/Original/537_N.png,[],['normal']
+112,fives_retina/train/Original/173_D.png,[],['diabetic retinopathy']
+113,fives_retina/train/Original/481_N.png,[],['normal']
+114,fives_retina/train/Original/131_A.png,[],['age related macular degeneration']
+115,fives_retina/train/Original/23_A.png,[],['age related macular degeneration']
+116,fives_retina/train/Original/141_A.png,[],['age related macular degeneration']
+117,fives_retina/train/Original/487_N.png,[],['normal']
+118,fives_retina/train/Original/554_N.png,[],['normal']
+119,fives_retina/train/Original/346_G.png,[],['glaucoma']
+120,fives_retina/train/Original/552_N.png,[],['normal']
+121,fives_retina/train/Original/87_A.png,[],['age related macular degeneration']
+122,fives_retina/train/Original/4_A.png,[],['age related macular degeneration']
+123,fives_retina/train/Original/559_N.png,[],['normal']
+124,fives_retina/train/Original/7_A.png,[],['age related macular degeneration']
+125,fives_retina/train/Original/593_N.png,[],['normal']
+126,fives_retina/train/Original/326_G.png,[],['glaucoma']
+127,fives_retina/train/Original/298_D.png,[],['diabetic retinopathy']
+128,fives_retina/train/Original/243_D.png,[],['diabetic retinopathy']
+129,fives_retina/train/Original/507_N.png,[],['normal']
+130,fives_retina/train/Original/497_N.png,[],['normal']
+131,fives_retina/train/Original/391_G.png,[],['glaucoma']
+132,fives_retina/train/Original/575_N.png,[],['normal']
+133,fives_retina/train/Original/274_D.png,[],['diabetic retinopathy']
+134,fives_retina/train/Original/399_G.png,[],['glaucoma']
+135,fives_retina/train/Original/533_N.png,[],['normal']
+136,fives_retina/train/Original/285_D.png,[],['diabetic retinopathy']
+137,fives_retina/train/Original/489_N.png,[],['normal']
+138,fives_retina/train/Original/147_A.png,[],['age related macular degeneration']
+139,fives_retina/train/Original/332_G.png,[],['glaucoma']
+140,fives_retina/train/Original/418_G.png,[],['glaucoma']
+141,fives_retina/train/Original/404_G.png,[],['glaucoma']
+142,fives_retina/train/Original/467_N.png,[],['normal']
+143,fives_retina/train/Original/64_A.png,[],['age related macular degeneration']
+144,fives_retina/train/Original/563_N.png,[],['normal']
+145,fives_retina/train/Original/340_G.png,[],['glaucoma']
+146,fives_retina/train/Original/30_A.png,[],['age related macular degeneration']
+147,fives_retina/train/Original/520_N.png,[],['normal']
+148,fives_retina/train/Original/175_D.png,[],['diabetic retinopathy']
+149,fives_retina/train/Original/58_A.png,[],['age related macular degeneration']
+150,fives_retina/train/Original/494_N.png,[],['normal']
+151,fives_retina/train/Original/227_D.png,[],['diabetic retinopathy']
+152,fives_retina/train/Original/162_D.png,[],['diabetic retinopathy']
+153,fives_retina/train/Original/150_A.png,[],['age related macular degeneration']
+154,fives_retina/train/Original/377_G.png,[],['glaucoma']
+155,fives_retina/train/Original/98_A.png,[],['age related macular degeneration']
+156,fives_retina/train/Original/385_G.png,[],['glaucoma']
+157,fives_retina/train/Original/59_A.png,[],['age related macular degeneration']
+158,fives_retina/train/Original/91_A.png,[],['age related macular degeneration']
+159,fives_retina/train/Original/192_D.png,[],['diabetic retinopathy']
+160,fives_retina/train/Original/38_A.png,[],['age related macular degeneration']
+161,fives_retina/train/Original/486_N.png,[],['normal']
+162,fives_retina/train/Original/568_N.png,[],['normal']
+163,fives_retina/train/Original/281_D.png,[],['diabetic retinopathy']
+164,fives_retina/train/Original/194_D.png,[],['diabetic retinopathy']
+165,fives_retina/train/Original/501_N.png,[],['normal']
+166,fives_retina/train/Original/291_D.png,[],['diabetic retinopathy']
+167,fives_retina/train/Original/178_D.png,[],['diabetic retinopathy']
+168,fives_retina/train/Original/588_N.png,[],['normal']
+169,fives_retina/train/Original/256_D.png,[],['diabetic retinopathy']
+170,fives_retina/train/Original/311_G.png,[],['glaucoma']
+171,fives_retina/train/Original/572_N.png,[],['normal']
+172,fives_retina/train/Original/196_D.png,[],['diabetic retinopathy']
+173,fives_retina/train/Original/239_D.png,[],['diabetic retinopathy']
+174,fives_retina/train/Original/553_N.png,[],['normal']
+175,fives_retina/train/Original/502_N.png,[],['normal']
+176,fives_retina/train/Original/5_A.png,[],['age related macular degeneration']
+177,fives_retina/train/Original/324_G.png,[],['glaucoma']
+178,fives_retina/train/Original/165_D.png,[],['diabetic retinopathy']
+179,fives_retina/train/Original/43_A.png,[],['age related macular degeneration']
+180,fives_retina/train/Original/212_D.png,[],['diabetic retinopathy']
+181,fives_retina/train/Original/143_A.png,[],['age related macular degeneration']
+182,fives_retina/train/Original/434_G.png,[],['glaucoma']
+183,fives_retina/train/Original/152_D.png,[],['diabetic retinopathy']
+184,fives_retina/train/Original/413_G.png,[],['glaucoma']
+185,fives_retina/train/Original/136_A.png,[],['age related macular degeneration']
+186,fives_retina/train/Original/483_N.png,[],['normal']
+187,fives_retina/train/Original/168_D.png,[],['diabetic retinopathy']
+188,fives_retina/train/Original/218_D.png,[],['diabetic retinopathy']
+189,fives_retina/train/Original/339_G.png,[],['glaucoma']
+190,fives_retina/train/Original/241_D.png,[],['diabetic retinopathy']
+191,fives_retina/train/Original/68_A.png,[],['age related macular degeneration']
+192,fives_retina/train/Original/354_G.png,[],['glaucoma']
+193,fives_retina/train/Original/515_N.png,[],['normal']
+194,fives_retina/train/Original/145_A.png,[],['age related macular degeneration']
+195,fives_retina/train/Original/337_G.png,[],['glaucoma']
+196,fives_retina/train/Original/381_G.png,[],['glaucoma']
+197,fives_retina/train/Original/191_D.png,[],['diabetic retinopathy']
+198,fives_retina/train/Original/511_N.png,[],['normal']
+199,fives_retina/train/Original/327_G.png,[],['glaucoma']
+200,fives_retina/train/Original/454_N.png,[],['normal']
+201,fives_retina/train/Original/523_N.png,[],['normal']
+202,fives_retina/train/Original/532_N.png,[],['normal']
+203,fives_retina/train/Original/221_D.png,[],['diabetic retinopathy']
+204,fives_retina/train/Original/125_A.png,[],['age related macular degeneration']
+205,fives_retina/train/Original/78_A.png,[],['age related macular degeneration']
+206,fives_retina/train/Original/126_A.png,[],['age related macular degeneration']
+207,fives_retina/train/Original/480_N.png,[],['normal']
+208,fives_retina/train/Original/180_D.png,[],['diabetic retinopathy']
+209,fives_retina/train/Original/341_G.png,[],['glaucoma']
+210,fives_retina/train/Original/539_N.png,[],['normal']
+211,fives_retina/train/Original/47_A.png,[],['age related macular degeneration']
+212,fives_retina/train/Original/353_G.png,[],['glaucoma']
+213,fives_retina/train/Original/277_D.png,[],['diabetic retinopathy']
+214,fives_retina/train/Original/204_D.png,[],['diabetic retinopathy']
+215,fives_retina/train/Original/160_D.png,[],['diabetic retinopathy']
+216,fives_retina/train/Original/414_G.png,[],['glaucoma']
+217,fives_retina/train/Original/546_N.png,[],['normal']
+218,fives_retina/train/Original/464_N.png,[],['normal']
+219,fives_retina/train/Original/542_N.png,[],['normal']
+220,fives_retina/train/Original/22_A.png,[],['age related macular degeneration']
+221,fives_retina/train/Original/57_A.png,[],['age related macular degeneration']
+222,fives_retina/train/Original/121_A.png,[],['age related macular degeneration']
+223,fives_retina/train/Original/166_D.png,[],['diabetic retinopathy']
+224,fives_retina/train/Original/259_D.png,[],['diabetic retinopathy']
+225,fives_retina/train/Original/313_G.png,[],['glaucoma']
+226,fives_retina/train/Original/174_D.png,[],['diabetic retinopathy']
+227,fives_retina/train/Original/1_A.png,[],['age related macular degeneration']
+228,fives_retina/train/Original/551_N.png,[],['normal']
+229,fives_retina/train/Original/50_A.png,[],['age related macular degeneration']
+230,fives_retina/train/Original/379_G.png,[],['glaucoma']
+231,fives_retina/train/Original/122_A.png,[],['age related macular degeneration']
+232,fives_retina/train/Original/493_N.png,[],['normal']
+233,fives_retina/train/Original/548_N.png,[],['normal']
+234,fives_retina/train/Original/109_A.png,[],['age related macular degeneration']
+235,fives_retina/train/Original/518_N.png,[],['normal']
+236,fives_retina/train/Original/231_D.png,[],['diabetic retinopathy']
+237,fives_retina/train/Original/183_D.png,[],['diabetic retinopathy']
+238,fives_retina/train/Original/421_G.png,[],['glaucoma']
+239,fives_retina/train/Original/81_A.png,[],['age related macular degeneration']
+240,fives_retina/train/Original/398_G.png,[],['glaucoma']
+241,fives_retina/train/Original/280_D.png,[],['diabetic retinopathy']
+242,fives_retina/train/Original/524_N.png,[],['normal']
+243,fives_retina/train/Original/305_G.png,[],['glaucoma']
+244,fives_retina/train/Original/471_N.png,[],['normal']
+245,fives_retina/train/Original/75_A.png,[],['age related macular degeneration']
+246,fives_retina/train/Original/54_A.png,[],['age related macular degeneration']
+247,fives_retina/train/Original/83_A.png,[],['age related macular degeneration']
+248,fives_retina/train/Original/484_N.png,[],['normal']
+249,fives_retina/train/Original/514_N.png,[],['normal']
+250,fives_retina/train/Original/415_G.png,[],['glaucoma']
+251,fives_retina/train/Original/153_D.png,[],['diabetic retinopathy']
+252,fives_retina/train/Original/41_A.png,[],['age related macular degeneration']
+253,fives_retina/train/Original/451_N.png,[],['normal']
+254,fives_retina/train/Original/186_D.png,[],['diabetic retinopathy']
+255,fives_retina/train/Original/583_N.png,[],['normal']
+256,fives_retina/train/Original/488_N.png,[],['normal']
+257,fives_retina/train/Original/142_A.png,[],['age related macular degeneration']
+258,fives_retina/train/Original/116_A.png,[],['age related macular degeneration']
+259,fives_retina/train/Original/72_A.png,[],['age related macular degeneration']
+260,fives_retina/train/Original/288_D.png,[],['diabetic retinopathy']
+261,fives_retina/train/Original/375_G.png,[],['glaucoma']
+262,fives_retina/train/Original/236_D.png,[],['diabetic retinopathy']
+263,fives_retina/train/Original/14_A.png,[],['age related macular degeneration']
+264,fives_retina/train/Original/108_A.png,[],['age related macular degeneration']
+265,fives_retina/train/Original/124_A.png,[],['age related macular degeneration']
+266,fives_retina/train/Original/315_G.png,[],['glaucoma']
+267,fives_retina/train/Original/294_D.png,[],['diabetic retinopathy']
+268,fives_retina/train/Original/13_A.png,[],['age related macular degeneration']
+269,fives_retina/train/Original/88_A.png,[],['age related macular degeneration']
+270,fives_retina/train/Original/76_A.png,[],['age related macular degeneration']
+271,fives_retina/train/Original/458_N.png,[],['normal']
+272,fives_retina/train/Original/512_N.png,[],['normal']
+273,fives_retina/train/Original/558_N.png,[],['normal']
+274,fives_retina/train/Original/459_N.png,[],['normal']
+275,fives_retina/train/Original/308_G.png,[],['glaucoma']
+276,fives_retina/train/Original/132_A.png,[],['age related macular degeneration']
+277,fives_retina/train/Original/334_G.png,[],['glaucoma']
+278,fives_retina/train/Original/102_A.png,[],['age related macular degeneration']
+279,fives_retina/train/Original/465_N.png,[],['normal']
+280,fives_retina/train/Original/594_N.png,[],['normal']
+281,fives_retina/train/Original/478_N.png,[],['normal']
+282,fives_retina/train/Original/290_D.png,[],['diabetic retinopathy']
+283,fives_retina/train/Original/359_G.png,[],['glaucoma']
+284,fives_retina/train/Original/384_G.png,[],['glaucoma']
+285,fives_retina/train/Original/65_A.png,[],['age related macular degeneration']
+286,fives_retina/train/Original/439_G.png,[],['glaucoma']
+287,fives_retina/train/Original/450_G.png,[],['glaucoma']
+288,fives_retina/train/Original/129_A.png,[],['age related macular degeneration']
+289,fives_retina/train/Original/73_A.png,[],['age related macular degeneration']
+290,fives_retina/train/Original/578_N.png,[],['normal']
+291,fives_retina/train/Original/573_N.png,[],['normal']
+292,fives_retina/train/Original/276_D.png,[],['diabetic retinopathy']
+293,fives_retina/train/Original/251_D.png,[],['diabetic retinopathy']
+294,fives_retina/train/Original/419_G.png,[],['glaucoma']
+295,fives_retina/train/Original/422_G.png,[],['glaucoma']
+296,fives_retina/train/Original/278_D.png,[],['diabetic retinopathy']
+297,fives_retina/train/Original/265_D.png,[],['diabetic retinopathy']
+298,fives_retina/train/Original/19_A.png,[],['age related macular degeneration']
+299,fives_retina/train/Original/177_D.png,[],['diabetic retinopathy']
+300,fives_retina/train/Original/69_A.png,[],['age related macular degeneration']
+301,fives_retina/train/Original/119_A.png,[],['age related macular degeneration']
+302,fives_retina/train/Original/333_G.png,[],['glaucoma']
+303,fives_retina/train/Original/248_D.png,[],['diabetic retinopathy']
+304,fives_retina/train/Original/80_A.png,[],['age related macular degeneration']
+305,fives_retina/train/Original/360_G.png,[],['glaucoma']
+306,fives_retina/train/Original/107_A.png,[],['age related macular degeneration']
+307,fives_retina/train/Original/469_N.png,[],['normal']
+308,fives_retina/train/Original/267_D.png,[],['diabetic retinopathy']
+309,fives_retina/train/Original/541_N.png,[],['normal']
+310,fives_retina/train/Original/198_D.png,[],['diabetic retinopathy']
+311,fives_retina/train/Original/171_D.png,[],['diabetic retinopathy']
+312,fives_retina/train/Original/70_A.png,[],['age related macular degeneration']
+313,fives_retina/train/Original/100_A.png,[],['age related macular degeneration']
+314,fives_retina/train/Original/45_A.png,[],['age related macular degeneration']
+315,fives_retina/train/Original/190_D.png,[],['diabetic retinopathy']
+316,fives_retina/train/Original/110_A.png,[],['age related macular degeneration']
+317,fives_retina/train/Original/169_D.png,[],['diabetic retinopathy']
+318,fives_retina/train/Original/599_N.png,[],['normal']
+319,fives_retina/train/Original/356_G.png,[],['glaucoma']
+320,fives_retina/train/Original/530_N.png,[],['normal']
+321,fives_retina/train/Original/155_D.png,[],['diabetic retinopathy']
+322,fives_retina/train/Original/237_D.png,[],['diabetic retinopathy']
+323,fives_retina/train/Original/426_G.png,[],['glaucoma']
+324,fives_retina/train/Original/61_A.png,[],['age related macular degeneration']
+325,fives_retina/train/Original/200_D.png,[],['diabetic retinopathy']
+326,fives_retina/train/Original/314_G.png,[],['glaucoma']
+327,fives_retina/train/Original/319_G.png,[],['glaucoma']
+328,fives_retina/train/Original/570_N.png,[],['normal']
+329,fives_retina/train/Original/397_G.png,[],['glaucoma']
+330,fives_retina/train/Original/457_N.png,[],['normal']
+331,fives_retina/train/Original/304_G.png,[],['glaucoma']
+332,fives_retina/train/Original/444_G.png,[],['glaucoma']
+333,fives_retina/train/Original/442_G.png,[],['glaucoma']
+334,fives_retina/train/Original/555_N.png,[],['normal']
+335,fives_retina/train/Original/29_A.png,[],['age related macular degeneration']
+336,fives_retina/train/Original/473_N.png,[],['normal']
+337,fives_retina/train/Original/297_D.png,[],['diabetic retinopathy']
+338,fives_retina/train/Original/154_D.png,[],['diabetic retinopathy']
+339,fives_retina/train/Original/101_A.png,[],['age related macular degeneration']
+340,fives_retina/train/Original/130_A.png,[],['age related macular degeneration']
+341,fives_retina/train/Original/405_G.png,[],['glaucoma']
+342,fives_retina/train/Original/492_N.png,[],['normal']
+343,fives_retina/train/Original/574_N.png,[],['normal']
+344,fives_retina/train/Original/490_N.png,[],['normal']
+345,fives_retina/train/Original/571_N.png,[],['normal']
+346,fives_retina/train/Original/301_G.png,[],['glaucoma']
+347,fives_retina/train/Original/289_D.png,[],['diabetic retinopathy']
+348,fives_retina/train/Original/63_A.png,[],['age related macular degeneration']
+349,fives_retina/train/Original/408_G.png,[],['glaucoma']
+350,fives_retina/train/Original/238_D.png,[],['diabetic retinopathy']
+351,fives_retina/train/Original/371_G.png,[],['glaucoma']
+352,fives_retina/train/Original/82_A.png,[],['age related macular degeneration']
+353,fives_retina/train/Original/368_G.png,[],['glaucoma']
+354,fives_retina/train/Original/374_G.png,[],['glaucoma']
+355,fives_retina/train/Original/323_G.png,[],['glaucoma']
+356,fives_retina/train/Original/225_D.png,[],['diabetic retinopathy']
+357,fives_retina/train/Original/477_N.png,[],['normal']
+358,fives_retina/train/Original/2_A.png,[],['age related macular degeneration']
+359,fives_retina/train/Original/127_A.png,[],['age related macular degeneration']
+360,fives_retina/train/Original/113_A.png,[],['age related macular degeneration']
+361,fives_retina/train/Original/211_D.png,[],['diabetic retinopathy']
+362,fives_retina/train/Original/137_A.png,[],['age related macular degeneration']
+363,fives_retina/train/Original/208_D.png,[],['diabetic retinopathy']
+364,fives_retina/train/Original/163_D.png,[],['diabetic retinopathy']
+365,fives_retina/train/Original/206_D.png,[],['diabetic retinopathy']
+366,fives_retina/train/Original/440_G.png,[],['glaucoma']
+367,fives_retina/train/Original/74_A.png,[],['age related macular degeneration']
+368,fives_retina/train/Original/425_G.png,[],['glaucoma']
+369,fives_retina/train/Original/275_D.png,[],['diabetic retinopathy']
+370,fives_retina/train/Original/62_A.png,[],['age related macular degeneration']
+371,fives_retina/train/Original/567_N.png,[],['normal']
+372,fives_retina/train/Original/53_A.png,[],['age related macular degeneration']
+373,fives_retina/train/Original/56_A.png,[],['age related macular degeneration']
+374,fives_retina/train/Original/27_A.png,[],['age related macular degeneration']
+375,fives_retina/train/Original/26_A.png,[],['age related macular degeneration']
+376,fives_retina/train/Original/565_N.png,[],['normal']
+377,fives_retina/train/Original/220_D.png,[],['diabetic retinopathy']
+378,fives_retina/train/Original/226_D.png,[],['diabetic retinopathy']
+379,fives_retina/train/Original/233_D.png,[],['diabetic retinopathy']
+380,fives_retina/train/Original/447_G.png,[],['glaucoma']
+381,fives_retina/train/Original/202_D.png,[],['diabetic retinopathy']
+382,fives_retina/train/Original/106_A.png,[],['age related macular degeneration']
+383,fives_retina/train/Original/255_D.png,[],['diabetic retinopathy']
+384,fives_retina/train/Original/264_D.png,[],['diabetic retinopathy']
+385,fives_retina/train/Original/577_N.png,[],['normal']
+386,fives_retina/train/Original/342_G.png,[],['glaucoma']
+387,fives_retina/train/Original/111_A.png,[],['age related macular degeneration']
+388,fives_retina/train/Original/10_A.png,[],['age related macular degeneration']
+389,fives_retina/train/Original/6_A.png,[],['age related macular degeneration']
+390,fives_retina/train/Original/336_G.png,[],['glaucoma']
+391,fives_retina/train/Original/547_N.png,[],['normal']
+392,fives_retina/train/Original/34_A.png,[],['age related macular degeneration']
+393,fives_retina/train/Original/93_A.png,[],['age related macular degeneration']
+394,fives_retina/train/Original/49_A.png,[],['age related macular degeneration']
+395,fives_retina/train/Original/365_G.png,[],['glaucoma']
+396,fives_retina/train/Original/21_A.png,[],['age related macular degeneration']
+397,fives_retina/train/Original/543_N.png,[],['normal']
+398,fives_retina/train/Original/148_A.png,[],['age related macular degeneration']
+399,fives_retina/train/Original/351_G.png,[],['glaucoma']
+400,fives_retina/train/Original/292_D.png,[],['diabetic retinopathy']
+401,fives_retina/train/Original/263_D.png,[],['diabetic retinopathy']
+402,fives_retina/train/Original/112_A.png,[],['age related macular degeneration']
+403,fives_retina/train/Original/435_G.png,[],['glaucoma']
+404,fives_retina/train/Original/460_N.png,[],['normal']
+405,fives_retina/train/Original/187_D.png,[],['diabetic retinopathy']
+406,fives_retina/train/Original/386_G.png,[],['glaucoma']
+407,fives_retina/train/Original/151_D.png,[],['diabetic retinopathy']
+408,fives_retina/train/Original/268_D.png,[],['diabetic retinopathy']
+409,fives_retina/train/Original/506_N.png,[],['normal']
+410,fives_retina/train/Original/282_D.png,[],['diabetic retinopathy']
+411,fives_retina/train/Original/269_D.png,[],['diabetic retinopathy']
+412,fives_retina/train/Original/410_G.png,[],['glaucoma']
+413,fives_retina/train/Original/441_G.png,[],['glaucoma']
+414,fives_retina/train/Original/16_A.png,[],['age related macular degeneration']
+415,fives_retina/train/Original/32_A.png,[],['age related macular degeneration']
+416,fives_retina/train/Original/302_G.png,[],['glaucoma']
+417,fives_retina/train/Original/396_G.png,[],['glaucoma']
+418,fives_retina/train/Original/170_D.png,[],['diabetic retinopathy']
+419,fives_retina/train/Original/322_G.png,[],['glaucoma']
+420,fives_retina/train/Original/92_A.png,[],['age related macular degeneration']
+421,fives_retina/train/Original/556_N.png,[],['normal']
+422,fives_retina/train/Original/79_A.png,[],['age related macular degeneration']
+423,fives_retina/train/Original/25_A.png,[],['age related macular degeneration']
+424,fives_retina/train/Original/436_G.png,[],['glaucoma']
+425,fives_retina/train/Original/245_D.png,[],['diabetic retinopathy']
+426,fives_retina/train/Original/214_D.png,[],['diabetic retinopathy']
+427,fives_retina/train/Original/452_N.png,[],['normal']
+428,fives_retina/train/Original/330_G.png,[],['glaucoma']
+429,fives_retina/train/Original/201_D.png,[],['diabetic retinopathy']
+430,fives_retina/train/Original/203_D.png,[],['diabetic retinopathy']
+431,fives_retina/train/Original/161_D.png,[],['diabetic retinopathy']
+432,fives_retina/train/Original/505_N.png,[],['normal']
+433,fives_retina/train/Original/557_N.png,[],['normal']
+434,fives_retina/train/Original/140_A.png,[],['age related macular degeneration']
+435,fives_retina/train/Original/566_N.png,[],['normal']
+436,fives_retina/train/Original/453_N.png,[],['normal']
+437,fives_retina/train/Original/427_G.png,[],['glaucoma']
+438,fives_retina/train/Original/580_N.png,[],['normal']
+439,fives_retina/train/Original/409_G.png,[],['glaucoma']
+440,fives_retina/train/Original/402_G.png,[],['glaucoma']
+441,fives_retina/train/Original/393_G.png,[],['glaucoma']
+442,fives_retina/train/Original/17_A.png,[],['age related macular degeneration']
+443,fives_retina/train/Original/400_G.png,[],['glaucoma']
+444,fives_retina/train/Original/254_D.png,[],['diabetic retinopathy']
+445,fives_retina/train/Original/115_A.png,[],['age related macular degeneration']
+446,fives_retina/train/Original/585_N.png,[],['normal']
+447,fives_retina/train/Original/389_G.png,[],['glaucoma']
+448,fives_retina/train/Original/521_N.png,[],['normal']
+449,fives_retina/train/Original/179_D.png,[],['diabetic retinopathy']
+450,fives_retina/train/Original/266_D.png,[],['diabetic retinopathy']
+451,fives_retina/train/Original/303_G.png,[],['glaucoma']
+452,fives_retina/train/Original/262_D.png,[],['diabetic retinopathy']
+453,fives_retina/train/Original/20_A.png,[],['age related macular degeneration']
+454,fives_retina/train/Original/172_D.png,[],['diabetic retinopathy']
+455,fives_retina/train/Original/504_N.png,[],['normal']
+456,fives_retina/train/Original/312_G.png,[],['glaucoma']
+457,fives_retina/train/Original/188_D.png,[],['diabetic retinopathy']
+458,fives_retina/train/Original/438_G.png,[],['glaucoma']
+459,fives_retina/train/Original/412_G.png,[],['glaucoma']
+460,fives_retina/train/Original/66_A.png,[],['age related macular degeneration']
+461,fives_retina/train/Original/222_D.png,[],['diabetic retinopathy']
+462,fives_retina/train/Original/310_G.png,[],['glaucoma']
+463,fives_retina/train/Original/134_A.png,[],['age related macular degeneration']
+464,fives_retina/train/Original/584_N.png,[],['normal']
+465,fives_retina/train/Original/242_D.png,[],['diabetic retinopathy']
+466,fives_retina/train/Original/466_N.png,[],['normal']
+467,fives_retina/train/Original/592_N.png,[],['normal']
+468,fives_retina/train/Original/84_A.png,[],['age related macular degeneration']
+469,fives_retina/train/Original/114_A.png,[],['age related macular degeneration']
+470,fives_retina/train/Original/387_G.png,[],['glaucoma']
+471,fives_retina/train/Original/217_D.png,[],['diabetic retinopathy']
+472,fives_retina/train/Original/42_A.png,[],['age related macular degeneration']
+473,fives_retina/train/Original/461_N.png,[],['normal']
+474,fives_retina/train/Original/587_N.png,[],['normal']
+475,fives_retina/train/Original/463_N.png,[],['normal']
+476,fives_retina/train/Original/395_G.png,[],['glaucoma']
+477,fives_retina/train/Original/28_A.png,[],['age related macular degeneration']
+478,fives_retina/train/Original/33_A.png,[],['age related macular degeneration']
+479,fives_retina/train/Original/482_N.png,[],['normal']
+480,fives_retina/train/Original/271_D.png,[],['diabetic retinopathy']
+481,fives_retina/train/Original/407_G.png,[],['glaucoma']
+482,fives_retina/train/Original/224_D.png,[],['diabetic retinopathy']
+483,fives_retina/train/Original/249_D.png,[],['diabetic retinopathy']
+484,fives_retina/train/Original/598_N.png,[],['normal']
+485,fives_retina/train/Original/273_D.png,[],['diabetic retinopathy']
+486,fives_retina/train/Original/357_G.png,[],['glaucoma']
+487,fives_retina/train/Original/433_G.png,[],['glaucoma']
+488,fives_retina/train/Original/424_G.png,[],['glaucoma']
+489,fives_retina/train/Original/39_A.png,[],['age related macular degeneration']
+490,fives_retina/train/Original/529_N.png,[],['normal']
+491,fives_retina/train/Original/388_G.png,[],['glaucoma']
+492,fives_retina/train/Original/216_D.png,[],['diabetic retinopathy']
+493,fives_retina/train/Original/287_D.png,[],['diabetic retinopathy']
+494,fives_retina/train/Original/331_G.png,[],['glaucoma']
+495,fives_retina/train/Original/367_G.png,[],['glaucoma']
+496,fives_retina/train/Original/510_N.png,[],['normal']
+497,fives_retina/train/Original/133_A.png,[],['age related macular degeneration']
+498,fives_retina/train/Original/215_D.png,[],['diabetic retinopathy']
+499,fives_retina/train/Original/569_N.png,[],['normal']
+500,fives_retina/train/Original/431_G.png,[],['glaucoma']
+501,fives_retina/train/Original/522_N.png,[],['normal']
+502,fives_retina/train/Original/545_N.png,[],['normal']
+503,fives_retina/train/Original/369_G.png,[],['glaucoma']
+504,fives_retina/train/Original/328_G.png,[],['glaucoma']
+505,fives_retina/train/Original/250_D.png,[],['diabetic retinopathy']
+506,fives_retina/train/Original/549_N.png,[],['normal']
+507,fives_retina/train/Original/195_D.png,[],['diabetic retinopathy']
+508,fives_retina/train/Original/95_A.png,[],['age related macular degeneration']
+509,fives_retina/train/Original/197_D.png,[],['diabetic retinopathy']
+510,fives_retina/train/Original/597_N.png,[],['normal']
+511,fives_retina/train/Original/286_D.png,[],['diabetic retinopathy']
+512,fives_retina/train/Original/513_N.png,[],['normal']
+513,fives_retina/train/Original/383_G.png,[],['glaucoma']
+514,fives_retina/train/Original/246_D.png,[],['diabetic retinopathy']
+515,fives_retina/train/Original/283_D.png,[],['diabetic retinopathy']
+516,fives_retina/train/Original/527_N.png,[],['normal']
+517,fives_retina/train/Original/15_A.png,[],['age related macular degeneration']
+518,fives_retina/train/Original/432_G.png,[],['glaucoma']
+519,fives_retina/train/Original/562_N.png,[],['normal']
+520,fives_retina/train/Original/12_A.png,[],['age related macular degeneration']
+521,fives_retina/train/Original/105_A.png,[],['age related macular degeneration']
+522,fives_retina/train/Original/309_G.png,[],['glaucoma']
+523,fives_retina/train/Original/348_G.png,[],['glaucoma']
+524,fives_retina/train/Original/128_A.png,[],['age related macular degeneration']
+525,fives_retina/train/Original/11_A.png,[],['age related macular degeneration']
+526,fives_retina/train/Original/158_D.png,[],['diabetic retinopathy']
+527,fives_retina/train/Original/544_N.png,[],['normal']
+528,fives_retina/train/Original/596_N.png,[],['normal']
+529,fives_retina/train/Original/167_D.png,[],['diabetic retinopathy']
+530,fives_retina/train/Original/406_G.png,[],['glaucoma']
+531,fives_retina/train/Original/525_N.png,[],['normal']
+532,fives_retina/train/Original/279_D.png,[],['diabetic retinopathy']
+533,fives_retina/train/Original/500_N.png,[],['normal']
+534,fives_retina/train/Original/491_N.png,[],['normal']
+535,fives_retina/train/Original/589_N.png,[],['normal']
+536,fives_retina/train/Original/349_G.png,[],['glaucoma']
+537,fives_retina/train/Original/307_G.png,[],['glaucoma']
+538,fives_retina/train/Original/210_D.png,[],['diabetic retinopathy']
+539,fives_retina/train/Original/538_N.png,[],['normal']
+540,fives_retina/train/Original/104_A.png,[],['age related macular degeneration']
+541,fives_retina/train/Original/455_N.png,[],['normal']
+542,fives_retina/train/Original/31_A.png,[],['age related macular degeneration']
+543,fives_retina/train/Original/600_N.png,[],['normal']
+544,fives_retina/train/Original/46_A.png,[],['age related macular degeneration']
+545,fives_retina/train/Original/318_G.png,[],['glaucoma']
+546,fives_retina/train/Original/416_G.png,[],['glaucoma']
+547,fives_retina/train/Original/392_G.png,[],['glaucoma']
+548,fives_retina/train/Original/550_N.png,[],['normal']
+549,fives_retina/train/Original/595_N.png,[],['normal']
+550,fives_retina/train/Original/85_A.png,[],['age related macular degeneration']
+551,fives_retina/train/Original/317_G.png,[],['glaucoma']
+552,fives_retina/train/Original/335_G.png,[],['glaucoma']
+553,fives_retina/train/Original/560_N.png,[],['normal']
+554,fives_retina/train/Original/306_G.png,[],['glaucoma']
+555,fives_retina/train/Original/472_N.png,[],['normal']
+556,fives_retina/train/Original/343_G.png,[],['glaucoma']
+557,fives_retina/train/Original/247_D.png,[],['diabetic retinopathy']
+558,fives_retina/train/Original/321_G.png,[],['glaucoma']
+559,fives_retina/train/Original/591_N.png,[],['normal']
+560,fives_retina/train/Original/44_A.png,[],['age related macular degeneration']
+561,fives_retina/train/Original/253_D.png,[],['diabetic retinopathy']
+562,fives_retina/train/Original/474_N.png,[],['normal']
+563,fives_retina/train/Original/223_D.png,[],['diabetic retinopathy']
+564,fives_retina/train/Original/157_D.png,[],['diabetic retinopathy']
+565,fives_retina/train/Original/52_A.png,[],['age related macular degeneration']
+566,fives_retina/train/Original/103_A.png,[],['age related macular degeneration']
+567,fives_retina/train/Original/159_D.png,[],['diabetic retinopathy']
+568,fives_retina/train/Original/344_G.png,[],['glaucoma']
+569,fives_retina/train/Original/209_D.png,[],['diabetic retinopathy']
+570,fives_retina/train/Original/185_D.png,[],['diabetic retinopathy']
+571,fives_retina/train/Original/446_G.png,[],['glaucoma']
+572,fives_retina/train/Original/325_G.png,[],['glaucoma']
+573,fives_retina/train/Original/411_G.png,[],['glaucoma']
+574,fives_retina/train/Original/382_G.png,[],['glaucoma']
+575,fives_retina/train/Original/581_N.png,[],['normal']
+576,fives_retina/train/Original/228_D.png,[],['diabetic retinopathy']
+577,fives_retina/train/Original/363_G.png,[],['glaucoma']
+578,fives_retina/train/Original/258_D.png,[],['diabetic retinopathy']
+579,fives_retina/train/Original/293_D.png,[],['diabetic retinopathy']
+580,fives_retina/train/Original/443_G.png,[],['glaucoma']
+581,fives_retina/train/Original/234_D.png,[],['diabetic retinopathy']
+582,fives_retina/train/Original/403_G.png,[],['glaucoma']
+583,fives_retina/train/Original/164_D.png,[],['diabetic retinopathy']
+584,fives_retina/train/Original/316_G.png,[],['glaucoma']
+585,fives_retina/train/Original/498_N.png,[],['normal']
+586,fives_retina/train/Original/199_D.png,[],['diabetic retinopathy']
+587,fives_retina/train/Original/118_A.png,[],['age related macular degeneration']
+588,fives_retina/train/Original/417_G.png,[],['glaucoma']
+589,fives_retina/train/Original/60_A.png,[],['age related macular degeneration']
+590,fives_retina/train/Original/373_G.png,[],['glaucoma']
+591,fives_retina/train/Original/244_D.png,[],['diabetic retinopathy']
+592,fives_retina/train/Original/519_N.png,[],['normal']
+593,fives_retina/train/Original/355_G.png,[],['glaucoma']
+594,fives_retina/train/Original/67_A.png,[],['age related macular degeneration']
+595,fives_retina/train/Original/499_N.png,[],['normal']
+596,fives_retina/train/Original/232_D.png,[],['diabetic retinopathy']
+597,fives_retina/train/Original/296_D.png,[],['diabetic retinopathy']
+598,fives_retina/train/Original/535_N.png,[],['normal']
+599,fives_retina/train/Original/300_D.png,[],['diabetic retinopathy']
+600,fives_retina/test/Original/117_G.png,[],['glaucoma']
+601,fives_retina/test/Original/3_A.png,[],['age related macular degeneration']
+602,fives_retina/test/Original/138_G.png,[],['glaucoma']
+603,fives_retina/test/Original/58_D.png,[],['diabetic retinopathy']
+604,fives_retina/test/Original/112_G.png,[],['glaucoma']
+605,fives_retina/test/Original/114_G.png,[],['glaucoma']
+606,fives_retina/test/Original/188_N.png,[],['normal']
+607,fives_retina/test/Original/24_A.png,[],['age related macular degeneration']
+608,fives_retina/test/Original/79_D.png,[],['diabetic retinopathy']
+609,fives_retina/test/Original/69_D.png,[],['diabetic retinopathy']
+610,fives_retina/test/Original/9_A.png,[],['age related macular degeneration']
+611,fives_retina/test/Original/103_G.png,[],['glaucoma']
+612,fives_retina/test/Original/8_A.png,[],['age related macular degeneration']
+613,fives_retina/test/Original/48_A.png,[],['age related macular degeneration']
+614,fives_retina/test/Original/155_N.png,[],['normal']
+615,fives_retina/test/Original/78_D.png,[],['diabetic retinopathy']
+616,fives_retina/test/Original/127_G.png,[],['glaucoma']
+617,fives_retina/test/Original/165_N.png,[],['normal']
+618,fives_retina/test/Original/194_N.png,[],['normal']
+619,fives_retina/test/Original/190_N.png,[],['normal']
+620,fives_retina/test/Original/168_N.png,[],['normal']
+621,fives_retina/test/Original/183_N.png,[],['normal']
+622,fives_retina/test/Original/18_A.png,[],['age related macular degeneration']
+623,fives_retina/test/Original/158_N.png,[],['normal']
+624,fives_retina/test/Original/55_D.png,[],['diabetic retinopathy']
+625,fives_retina/test/Original/179_N.png,[],['normal']
+626,fives_retina/test/Original/129_G.png,[],['glaucoma']
+627,fives_retina/test/Original/37_A.png,[],['age related macular degeneration']
+628,fives_retina/test/Original/35_A.png,[],['age related macular degeneration']
+629,fives_retina/test/Original/161_N.png,[],['normal']
+630,fives_retina/test/Original/99_D.png,[],['diabetic retinopathy']
+631,fives_retina/test/Original/36_A.png,[],['age related macular degeneration']
+632,fives_retina/test/Original/71_D.png,[],['diabetic retinopathy']
+633,fives_retina/test/Original/139_G.png,[],['glaucoma']
+634,fives_retina/test/Original/149_G.png,[],['glaucoma']
+635,fives_retina/test/Original/40_A.png,[],['age related macular degeneration']
+636,fives_retina/test/Original/118_G.png,[],['glaucoma']
+637,fives_retina/test/Original/23_A.png,[],['age related macular degeneration']
+638,fives_retina/test/Original/4_A.png,[],['age related macular degeneration']
+639,fives_retina/test/Original/7_A.png,[],['age related macular degeneration']
+640,fives_retina/test/Original/65_D.png,[],['diabetic retinopathy']
+641,fives_retina/test/Original/141_G.png,[],['glaucoma']
+642,fives_retina/test/Original/172_N.png,[],['normal']
+643,fives_retina/test/Original/70_D.png,[],['diabetic retinopathy']
+644,fives_retina/test/Original/196_N.png,[],['normal']
+645,fives_retina/test/Original/128_G.png,[],['glaucoma']
+646,fives_retina/test/Original/101_G.png,[],['glaucoma']
+647,fives_retina/test/Original/30_A.png,[],['age related macular degeneration']
+648,fives_retina/test/Original/63_D.png,[],['diabetic retinopathy']
+649,fives_retina/test/Original/100_D.png,[],['diabetic retinopathy']
+650,fives_retina/test/Original/169_N.png,[],['normal']
+651,fives_retina/test/Original/150_G.png,[],['glaucoma']
+652,fives_retina/test/Original/52_D.png,[],['diabetic retinopathy']
+653,fives_retina/test/Original/154_N.png,[],['normal']
+654,fives_retina/test/Original/38_A.png,[],['age related macular degeneration']
+655,fives_retina/test/Original/171_N.png,[],['normal']
+656,fives_retina/test/Original/200_N.png,[],['normal']
+657,fives_retina/test/Original/135_G.png,[],['glaucoma']
+658,fives_retina/test/Original/160_N.png,[],['normal']
+659,fives_retina/test/Original/5_A.png,[],['age related macular degeneration']
+660,fives_retina/test/Original/140_G.png,[],['glaucoma']
+661,fives_retina/test/Original/43_A.png,[],['age related macular degeneration']
+662,fives_retina/test/Original/120_G.png,[],['glaucoma']
+663,fives_retina/test/Original/110_G.png,[],['glaucoma']
+664,fives_retina/test/Original/178_N.png,[],['normal']
+665,fives_retina/test/Original/93_D.png,[],['diabetic retinopathy']
+666,fives_retina/test/Original/198_N.png,[],['normal']
+667,fives_retina/test/Original/56_D.png,[],['diabetic retinopathy']
+668,fives_retina/test/Original/98_D.png,[],['diabetic retinopathy']
+669,fives_retina/test/Original/95_D.png,[],['diabetic retinopathy']
+670,fives_retina/test/Original/159_N.png,[],['normal']
+671,fives_retina/test/Original/175_N.png,[],['normal']
+672,fives_retina/test/Original/47_A.png,[],['age related macular degeneration']
+673,fives_retina/test/Original/180_N.png,[],['normal']
+674,fives_retina/test/Original/22_A.png,[],['age related macular degeneration']
+675,fives_retina/test/Original/106_G.png,[],['glaucoma']
+676,fives_retina/test/Original/84_D.png,[],['diabetic retinopathy']
+677,fives_retina/test/Original/1_A.png,[],['age related macular degeneration']
+678,fives_retina/test/Original/50_A.png,[],['age related macular degeneration']
+679,fives_retina/test/Original/54_D.png,[],['diabetic retinopathy']
+680,fives_retina/test/Original/60_D.png,[],['diabetic retinopathy']
+681,fives_retina/test/Original/132_G.png,[],['glaucoma']
+682,fives_retina/test/Original/67_D.png,[],['diabetic retinopathy']
+683,fives_retina/test/Original/195_N.png,[],['normal']
+684,fives_retina/test/Original/64_D.png,[],['diabetic retinopathy']
+685,fives_retina/test/Original/182_N.png,[],['normal']
+686,fives_retina/test/Original/41_A.png,[],['age related macular degeneration']
+687,fives_retina/test/Original/156_N.png,[],['normal']
+688,fives_retina/test/Original/72_D.png,[],['diabetic retinopathy']
+689,fives_retina/test/Original/14_A.png,[],['age related macular degeneration']
+690,fives_retina/test/Original/13_A.png,[],['age related macular degeneration']
+691,fives_retina/test/Original/148_G.png,[],['glaucoma']
+692,fives_retina/test/Original/62_D.png,[],['diabetic retinopathy']
+693,fives_retina/test/Original/57_D.png,[],['diabetic retinopathy']
+694,fives_retina/test/Original/136_G.png,[],['glaucoma']
+695,fives_retina/test/Original/143_G.png,[],['glaucoma']
+696,fives_retina/test/Original/96_D.png,[],['diabetic retinopathy']
+697,fives_retina/test/Original/170_N.png,[],['normal']
+698,fives_retina/test/Original/68_D.png,[],['diabetic retinopathy']
+699,fives_retina/test/Original/122_G.png,[],['glaucoma']
+700,fives_retina/test/Original/77_D.png,[],['diabetic retinopathy']
+701,fives_retina/test/Original/19_A.png,[],['age related macular degeneration']
+702,fives_retina/test/Original/108_G.png,[],['glaucoma']
+703,fives_retina/test/Original/193_N.png,[],['normal']
+704,fives_retina/test/Original/176_N.png,[],['normal']
+705,fives_retina/test/Original/199_N.png,[],['normal']
+706,fives_retina/test/Original/174_N.png,[],['normal']
+707,fives_retina/test/Original/45_A.png,[],['age related macular degeneration']
+708,fives_retina/test/Original/81_D.png,[],['diabetic retinopathy']
+709,fives_retina/test/Original/124_G.png,[],['glaucoma']
+710,fives_retina/test/Original/185_N.png,[],['normal']
+711,fives_retina/test/Original/94_D.png,[],['diabetic retinopathy']
+712,fives_retina/test/Original/86_D.png,[],['diabetic retinopathy']
+713,fives_retina/test/Original/89_D.png,[],['diabetic retinopathy']
+714,fives_retina/test/Original/163_N.png,[],['normal']
+715,fives_retina/test/Original/173_N.png,[],['normal']
+716,fives_retina/test/Original/162_N.png,[],['normal']
+717,fives_retina/test/Original/29_A.png,[],['age related macular degeneration']
+718,fives_retina/test/Original/115_G.png,[],['glaucoma']
+719,fives_retina/test/Original/189_N.png,[],['normal']
+720,fives_retina/test/Original/119_G.png,[],['glaucoma']
+721,fives_retina/test/Original/82_D.png,[],['diabetic retinopathy']
+722,fives_retina/test/Original/83_D.png,[],['diabetic retinopathy']
+723,fives_retina/test/Original/131_G.png,[],['glaucoma']
+724,fives_retina/test/Original/144_G.png,[],['glaucoma']
+725,fives_retina/test/Original/2_A.png,[],['age related macular degeneration']
+726,fives_retina/test/Original/85_D.png,[],['diabetic retinopathy']
+727,fives_retina/test/Original/152_N.png,[],['normal']
+728,fives_retina/test/Original/90_D.png,[],['diabetic retinopathy']
+729,fives_retina/test/Original/27_A.png,[],['age related macular degeneration']
+730,fives_retina/test/Original/87_D.png,[],['diabetic retinopathy']
+731,fives_retina/test/Original/97_D.png,[],['diabetic retinopathy']
+732,fives_retina/test/Original/26_A.png,[],['age related macular degeneration']
+733,fives_retina/test/Original/186_N.png,[],['normal']
+734,fives_retina/test/Original/177_N.png,[],['normal']
+735,fives_retina/test/Original/142_G.png,[],['glaucoma']
+736,fives_retina/test/Original/109_G.png,[],['glaucoma']
+737,fives_retina/test/Original/76_D.png,[],['diabetic retinopathy']
+738,fives_retina/test/Original/10_A.png,[],['age related macular degeneration']
+739,fives_retina/test/Original/6_A.png,[],['age related macular degeneration']
+740,fives_retina/test/Original/116_G.png,[],['glaucoma']
+741,fives_retina/test/Original/126_G.png,[],['glaucoma']
+742,fives_retina/test/Original/34_A.png,[],['age related macular degeneration']
+743,fives_retina/test/Original/49_A.png,[],['age related macular degeneration']
+744,fives_retina/test/Original/21_A.png,[],['age related macular degeneration']
+745,fives_retina/test/Original/102_G.png,[],['glaucoma']
+746,fives_retina/test/Original/151_N.png,[],['normal']
+747,fives_retina/test/Original/16_A.png,[],['age related macular degeneration']
+748,fives_retina/test/Original/32_A.png,[],['age related macular degeneration']
+749,fives_retina/test/Original/51_D.png,[],['diabetic retinopathy']
+750,fives_retina/test/Original/197_N.png,[],['normal']
+751,fives_retina/test/Original/66_D.png,[],['diabetic retinopathy']
+752,fives_retina/test/Original/184_N.png,[],['normal']
+753,fives_retina/test/Original/25_A.png,[],['age related macular degeneration']
+754,fives_retina/test/Original/157_N.png,[],['normal']
+755,fives_retina/test/Original/164_N.png,[],['normal']
+756,fives_retina/test/Original/130_G.png,[],['glaucoma']
+757,fives_retina/test/Original/17_A.png,[],['age related macular degeneration']
+758,fives_retina/test/Original/123_G.png,[],['glaucoma']
+759,fives_retina/test/Original/147_G.png,[],['glaucoma']
+760,fives_retina/test/Original/20_A.png,[],['age related macular degeneration']
+761,fives_retina/test/Original/111_G.png,[],['glaucoma']
+762,fives_retina/test/Original/53_D.png,[],['diabetic retinopathy']
+763,fives_retina/test/Original/191_N.png,[],['normal']
+764,fives_retina/test/Original/104_G.png,[],['glaucoma']
+765,fives_retina/test/Original/146_G.png,[],['glaucoma']
+766,fives_retina/test/Original/42_A.png,[],['age related macular degeneration']
+767,fives_retina/test/Original/28_A.png,[],['age related macular degeneration']
+768,fives_retina/test/Original/33_A.png,[],['age related macular degeneration']
+769,fives_retina/test/Original/133_G.png,[],['glaucoma']
+770,fives_retina/test/Original/153_N.png,[],['normal']
+771,fives_retina/test/Original/39_A.png,[],['age related macular degeneration']
+772,fives_retina/test/Original/74_D.png,[],['diabetic retinopathy']
+773,fives_retina/test/Original/137_G.png,[],['glaucoma']
+774,fives_retina/test/Original/73_D.png,[],['diabetic retinopathy']
+775,fives_retina/test/Original/166_N.png,[],['normal']
+776,fives_retina/test/Original/134_G.png,[],['glaucoma']
+777,fives_retina/test/Original/167_N.png,[],['normal']
+778,fives_retina/test/Original/15_A.png,[],['age related macular degeneration']
+779,fives_retina/test/Original/91_D.png,[],['diabetic retinopathy']
+780,fives_retina/test/Original/12_A.png,[],['age related macular degeneration']
+781,fives_retina/test/Original/145_G.png,[],['glaucoma']
+782,fives_retina/test/Original/75_D.png,[],['diabetic retinopathy']
+783,fives_retina/test/Original/11_A.png,[],['age related macular degeneration']
+784,fives_retina/test/Original/107_G.png,[],['glaucoma']
+785,fives_retina/test/Original/80_D.png,[],['diabetic retinopathy']
+786,fives_retina/test/Original/192_N.png,[],['normal']
+787,fives_retina/test/Original/187_N.png,[],['normal']
+788,fives_retina/test/Original/31_A.png,[],['age related macular degeneration']
+789,fives_retina/test/Original/59_D.png,[],['diabetic retinopathy']
+790,fives_retina/test/Original/46_A.png,[],['age related macular degeneration']
+791,fives_retina/test/Original/125_G.png,[],['glaucoma']
+792,fives_retina/test/Original/181_N.png,[],['normal']
+793,fives_retina/test/Original/105_G.png,[],['glaucoma']
+794,fives_retina/test/Original/44_A.png,[],['age related macular degeneration']
+795,fives_retina/test/Original/61_D.png,[],['diabetic retinopathy']
+796,fives_retina/test/Original/88_D.png,[],['diabetic retinopathy']
+797,fives_retina/test/Original/121_G.png,[],['glaucoma']
+798,fives_retina/test/Original/113_G.png,[],['glaucoma']
+799,fives_retina/test/Original/92_D.png,[],['diabetic retinopathy']

local_data/25_REFUGE.csv

@@ -0,0 +1,401 @@
+,image,atributes,categories
+0,REFUGE-Training400/g0001.jpg,[],['glaucoma']
+1,REFUGE-Training400/g0002.jpg,[],['glaucoma']
+2,REFUGE-Training400/g0003.jpg,[],['glaucoma']
+3,REFUGE-Training400/g0004.jpg,[],['glaucoma']
+4,REFUGE-Training400/g0005.jpg,[],['glaucoma']
+5,REFUGE-Training400/g0006.jpg,[],['glaucoma']
+6,REFUGE-Training400/g0007.jpg,[],['glaucoma']
+7,REFUGE-Training400/g0008.jpg,[],['glaucoma']
+8,REFUGE-Training400/g0009.jpg,[],['glaucoma']
+9,REFUGE-Training400/g0010.jpg,[],['glaucoma']
+10,REFUGE-Training400/g0011.jpg,[],['glaucoma']
+11,REFUGE-Training400/g0012.jpg,[],['glaucoma']
+12,REFUGE-Training400/g0013.jpg,[],['glaucoma']
+13,REFUGE-Training400/g0014.jpg,[],['glaucoma']
+14,REFUGE-Training400/g0015.jpg,[],['glaucoma']
+15,REFUGE-Training400/g0016.jpg,[],['glaucoma']
+16,REFUGE-Training400/g0017.jpg,[],['glaucoma']
+17,REFUGE-Training400/g0018.jpg,[],['glaucoma']
+18,REFUGE-Training400/g0019.jpg,[],['glaucoma']
+19,REFUGE-Training400/g0020.jpg,[],['glaucoma']
+20,REFUGE-Training400/g0021.jpg,[],['glaucoma']
+21,REFUGE-Training400/g0022.jpg,[],['glaucoma']
+22,REFUGE-Training400/g0023.jpg,[],['glaucoma']
+23,REFUGE-Training400/g0024.jpg,[],['glaucoma']
+24,REFUGE-Training400/g0025.jpg,[],['glaucoma']
+25,REFUGE-Training400/g0026.jpg,[],['glaucoma']
+26,REFUGE-Training400/g0027.jpg,[],['glaucoma']
+27,REFUGE-Training400/g0028.jpg,[],['glaucoma']
+28,REFUGE-Training400/g0029.jpg,[],['glaucoma']
+29,REFUGE-Training400/g0030.jpg,[],['glaucoma']
+30,REFUGE-Training400/g0031.jpg,[],['glaucoma']
+31,REFUGE-Training400/g0032.jpg,[],['glaucoma']
+32,REFUGE-Training400/g0033.jpg,[],['glaucoma']
+33,REFUGE-Training400/g0034.jpg,[],['glaucoma']
+34,REFUGE-Training400/g0035.jpg,[],['glaucoma']
+35,REFUGE-Training400/g0036.jpg,[],['glaucoma']
+36,REFUGE-Training400/g0037.jpg,[],['glaucoma']
+37,REFUGE-Training400/g0038.jpg,[],['glaucoma']
+38,REFUGE-Training400/g0039.jpg,[],['glaucoma']
+39,REFUGE-Training400/g0040.jpg,[],['glaucoma']
+40,REFUGE-Training400/n0001.jpg,[],['no glaucoma']
+41,REFUGE-Training400/n0002.jpg,[],['no glaucoma']
+42,REFUGE-Training400/n0003.jpg,[],['no glaucoma']
+43,REFUGE-Training400/n0004.jpg,[],['no glaucoma']
+44,REFUGE-Training400/n0005.jpg,[],['no glaucoma']
+45,REFUGE-Training400/n0006.jpg,[],['no glaucoma']
+46,REFUGE-Training400/n0007.jpg,[],['no glaucoma']
+47,REFUGE-Training400/n0008.jpg,[],['no glaucoma']
+48,REFUGE-Training400/n0009.jpg,[],['no glaucoma']
+49,REFUGE-Training400/n0010.jpg,[],['no glaucoma']
+50,REFUGE-Training400/n0011.jpg,[],['no glaucoma']
+51,REFUGE-Training400/n0012.jpg,[],['no glaucoma']
+52,REFUGE-Training400/n0013.jpg,[],['no glaucoma']
+53,REFUGE-Training400/n0014.jpg,[],['no glaucoma']
+54,REFUGE-Training400/n0015.jpg,[],['no glaucoma']
+55,REFUGE-Training400/n0016.jpg,[],['no glaucoma']
+56,REFUGE-Training400/n0017.jpg,[],['no glaucoma']
+57,REFUGE-Training400/n0018.jpg,[],['no glaucoma']
+58,REFUGE-Training400/n0019.jpg,[],['no glaucoma']
+59,REFUGE-Training400/n0020.jpg,[],['no glaucoma']
+60,REFUGE-Training400/n0021.jpg,[],['no glaucoma']
+61,REFUGE-Training400/n0022.jpg,[],['no glaucoma']
+62,REFUGE-Training400/n0023.jpg,[],['no glaucoma']
+63,REFUGE-Training400/n0024.jpg,[],['no glaucoma']
+64,REFUGE-Training400/n0025.jpg,[],['no glaucoma']
+65,REFUGE-Training400/n0026.jpg,[],['no glaucoma']
+66,REFUGE-Training400/n0027.jpg,[],['no glaucoma']
+67,REFUGE-Training400/n0028.jpg,[],['no glaucoma']
+68,REFUGE-Training400/n0029.jpg,[],['no glaucoma']
+69,REFUGE-Training400/n0030.jpg,[],['no glaucoma']
+70,REFUGE-Training400/n0031.jpg,[],['no glaucoma']
+71,REFUGE-Training400/n0032.jpg,[],['no glaucoma']
+72,REFUGE-Training400/n0033.jpg,[],['no glaucoma']
+73,REFUGE-Training400/n0034.jpg,[],['no glaucoma']
+74,REFUGE-Training400/n0035.jpg,[],['no glaucoma']
+75,REFUGE-Training400/n0036.jpg,[],['no glaucoma']
+76,REFUGE-Training400/n0037.jpg,[],['no glaucoma']
+77,REFUGE-Training400/n0038.jpg,[],['no glaucoma']
+78,REFUGE-Training400/n0039.jpg,[],['no glaucoma']
+79,REFUGE-Training400/n0040.jpg,[],['no glaucoma']
+80,REFUGE-Training400/n0041.jpg,[],['no glaucoma']
+81,REFUGE-Training400/n0042.jpg,[],['no glaucoma']
+82,REFUGE-Training400/n0043.jpg,[],['no glaucoma']
+83,REFUGE-Training400/n0044.jpg,[],['no glaucoma']
+84,REFUGE-Training400/n0045.jpg,[],['no glaucoma']
+85,REFUGE-Training400/n0046.jpg,[],['no glaucoma']
+86,REFUGE-Training400/n0047.jpg,[],['no glaucoma']
+87,REFUGE-Training400/n0048.jpg,[],['no glaucoma']
+88,REFUGE-Training400/n0049.jpg,[],['no glaucoma']
+89,REFUGE-Training400/n0050.jpg,[],['no glaucoma']
+90,REFUGE-Training400/n0051.jpg,[],['no glaucoma']
+91,REFUGE-Training400/n0052.jpg,[],['no glaucoma']
+92,REFUGE-Training400/n0053.jpg,[],['no glaucoma']
+93,REFUGE-Training400/n0054.jpg,[],['no glaucoma']
+94,REFUGE-Training400/n0055.jpg,[],['no glaucoma']
+95,REFUGE-Training400/n0056.jpg,[],['no glaucoma']
+96,REFUGE-Training400/n0057.jpg,[],['no glaucoma']
+97,REFUGE-Training400/n0058.jpg,[],['no glaucoma']
+98,REFUGE-Training400/n0059.jpg,[],['no glaucoma']
+99,REFUGE-Training400/n0060.jpg,[],['no glaucoma']
+100,REFUGE-Training400/n0061.jpg,[],['no glaucoma']
+101,REFUGE-Training400/n0062.jpg,[],['no glaucoma']
+102,REFUGE-Training400/n0063.jpg,[],['no glaucoma']
+103,REFUGE-Training400/n0064.jpg,[],['no glaucoma']
+104,REFUGE-Training400/n0065.jpg,[],['no glaucoma']
+105,REFUGE-Training400/n0066.jpg,[],['no glaucoma']
+106,REFUGE-Training400/n0067.jpg,[],['no glaucoma']
+107,REFUGE-Training400/n0068.jpg,[],['no glaucoma']
+108,REFUGE-Training400/n0069.jpg,[],['no glaucoma']
+109,REFUGE-Training400/n0070.jpg,[],['no glaucoma']
+110,REFUGE-Training400/n0071.jpg,[],['no glaucoma']
+111,REFUGE-Training400/n0072.jpg,[],['no glaucoma']
+112,REFUGE-Training400/n0073.jpg,[],['no glaucoma']
+113,REFUGE-Training400/n0074.jpg,[],['no glaucoma']
+114,REFUGE-Training400/n0075.jpg,[],['no glaucoma']
+115,REFUGE-Training400/n0076.jpg,[],['no glaucoma']
+116,REFUGE-Training400/n0077.jpg,[],['no glaucoma']
+117,REFUGE-Training400/n0078.jpg,[],['no glaucoma']
+118,REFUGE-Training400/n0079.jpg,[],['no glaucoma']
+119,REFUGE-Training400/n0080.jpg,[],['no glaucoma']
+120,REFUGE-Training400/n0081.jpg,[],['no glaucoma']
+121,REFUGE-Training400/n0082.jpg,[],['no glaucoma']
+122,REFUGE-Training400/n0083.jpg,[],['no glaucoma']
+123,REFUGE-Training400/n0084.jpg,[],['no glaucoma']
+124,REFUGE-Training400/n0085.jpg,[],['no glaucoma']
+125,REFUGE-Training400/n0086.jpg,[],['no glaucoma']
+126,REFUGE-Training400/n0087.jpg,[],['no glaucoma']
+127,REFUGE-Training400/n0088.jpg,[],['no glaucoma']
+128,REFUGE-Training400/n0089.jpg,[],['no glaucoma']
+129,REFUGE-Training400/n0090.jpg,[],['no glaucoma']
+130,REFUGE-Training400/n0091.jpg,[],['no glaucoma']
+131,REFUGE-Training400/n0092.jpg,[],['no glaucoma']
+132,REFUGE-Training400/n0093.jpg,[],['no glaucoma']
+133,REFUGE-Training400/n0094.jpg,[],['no glaucoma']
+134,REFUGE-Training400/n0095.jpg,[],['no glaucoma']
+135,REFUGE-Training400/n0096.jpg,[],['no glaucoma']
+136,REFUGE-Training400/n0097.jpg,[],['no glaucoma']
+137,REFUGE-Training400/n0098.jpg,[],['no glaucoma']
+138,REFUGE-Training400/n0099.jpg,[],['no glaucoma']
+139,REFUGE-Training400/n0100.jpg,[],['no glaucoma']
+140,REFUGE-Training400/n0101.jpg,[],['no glaucoma']
+141,REFUGE-Training400/n0102.jpg,[],['no glaucoma']
+142,REFUGE-Training400/n0103.jpg,[],['no glaucoma']
+143,REFUGE-Training400/n0104.jpg,[],['no glaucoma']
+144,REFUGE-Training400/n0105.jpg,[],['no glaucoma']
+145,REFUGE-Training400/n0106.jpg,[],['no glaucoma']
+146,REFUGE-Training400/n0107.jpg,[],['no glaucoma']
+147,REFUGE-Training400/n0108.jpg,[],['no glaucoma']
+148,REFUGE-Training400/n0109.jpg,[],['no glaucoma']
+149,REFUGE-Training400/n0110.jpg,[],['no glaucoma']
+150,REFUGE-Training400/n0111.jpg,[],['no glaucoma']
+151,REFUGE-Training400/n0112.jpg,[],['no glaucoma']
+152,REFUGE-Training400/n0113.jpg,[],['no glaucoma']
+153,REFUGE-Training400/n0114.jpg,[],['no glaucoma']
+154,REFUGE-Training400/n0115.jpg,[],['no glaucoma']
+155,REFUGE-Training400/n0116.jpg,[],['no glaucoma']
+156,REFUGE-Training400/n0117.jpg,[],['no glaucoma']
+157,REFUGE-Training400/n0118.jpg,[],['no glaucoma']
+158,REFUGE-Training400/n0119.jpg,[],['no glaucoma']
+159,REFUGE-Training400/n0120.jpg,[],['no glaucoma']
+160,REFUGE-Training400/n0121.jpg,[],['no glaucoma']
+161,REFUGE-Training400/n0122.jpg,[],['no glaucoma']
+162,REFUGE-Training400/n0123.jpg,[],['no glaucoma']
+163,REFUGE-Training400/n0124.jpg,[],['no glaucoma']
+164,REFUGE-Training400/n0125.jpg,[],['no glaucoma']
+165,REFUGE-Training400/n0126.jpg,[],['no glaucoma']
+166,REFUGE-Training400/n0127.jpg,[],['no glaucoma']
+167,REFUGE-Training400/n0128.jpg,[],['no glaucoma']
+168,REFUGE-Training400/n0129.jpg,[],['no glaucoma']
+169,REFUGE-Training400/n0130.jpg,[],['no glaucoma']
+170,REFUGE-Training400/n0131.jpg,[],['no glaucoma']
+171,REFUGE-Training400/n0132.jpg,[],['no glaucoma']
+172,REFUGE-Training400/n0133.jpg,[],['no glaucoma']
+173,REFUGE-Training400/n0134.jpg,[],['no glaucoma']
+174,REFUGE-Training400/n0135.jpg,[],['no glaucoma']
+175,REFUGE-Training400/n0136.jpg,[],['no glaucoma']
+176,REFUGE-Training400/n0137.jpg,[],['no glaucoma']
+177,REFUGE-Training400/n0138.jpg,[],['no glaucoma']
+178,REFUGE-Training400/n0139.jpg,[],['no glaucoma']
+179,REFUGE-Training400/n0140.jpg,[],['no glaucoma']
+180,REFUGE-Training400/n0141.jpg,[],['no glaucoma']
+181,REFUGE-Training400/n0142.jpg,[],['no glaucoma']
+182,REFUGE-Training400/n0143.jpg,[],['no glaucoma']
+183,REFUGE-Training400/n0144.jpg,[],['no glaucoma']
+184,REFUGE-Training400/n0145.jpg,[],['no glaucoma']
+185,REFUGE-Training400/n0146.jpg,[],['no glaucoma']
+186,REFUGE-Training400/n0147.jpg,[],['no glaucoma']
+187,REFUGE-Training400/n0148.jpg,[],['no glaucoma']
+188,REFUGE-Training400/n0149.jpg,[],['no glaucoma']
+189,REFUGE-Training400/n0150.jpg,[],['no glaucoma']
+190,REFUGE-Training400/n0151.jpg,[],['no glaucoma']
+191,REFUGE-Training400/n0152.jpg,[],['no glaucoma']
+192,REFUGE-Training400/n0153.jpg,[],['no glaucoma']
+193,REFUGE-Training400/n0154.jpg,[],['no glaucoma']
+194,REFUGE-Training400/n0155.jpg,[],['no glaucoma']
+195,REFUGE-Training400/n0156.jpg,[],['no glaucoma']
+196,REFUGE-Training400/n0157.jpg,[],['no glaucoma']
+197,REFUGE-Training400/n0158.jpg,[],['no glaucoma']
+198,REFUGE-Training400/n0159.jpg,[],['no glaucoma']
+199,REFUGE-Training400/n0160.jpg,[],['no glaucoma']
+200,REFUGE-Training400/n0161.jpg,[],['no glaucoma']
+201,REFUGE-Training400/n0162.jpg,[],['no glaucoma']
+202,REFUGE-Training400/n0163.jpg,[],['no glaucoma']
+203,REFUGE-Training400/n0164.jpg,[],['no glaucoma']
+204,REFUGE-Training400/n0165.jpg,[],['no glaucoma']
+205,REFUGE-Training400/n0166.jpg,[],['no glaucoma']
+206,REFUGE-Training400/n0167.jpg,[],['no glaucoma']
+207,REFUGE-Training400/n0168.jpg,[],['no glaucoma']
+208,REFUGE-Training400/n0169.jpg,[],['no glaucoma']
+209,REFUGE-Training400/n0170.jpg,[],['no glaucoma']
+210,REFUGE-Training400/n0171.jpg,[],['no glaucoma']
+211,REFUGE-Training400/n0172.jpg,[],['no glaucoma']
+212,REFUGE-Training400/n0173.jpg,[],['no glaucoma']
+213,REFUGE-Training400/n0174.jpg,[],['no glaucoma']
+214,REFUGE-Training400/n0175.jpg,[],['no glaucoma']
+215,REFUGE-Training400/n0176.jpg,[],['no glaucoma']
+216,REFUGE-Training400/n0177.jpg,[],['no glaucoma']
+217,REFUGE-Training400/n0178.jpg,[],['no glaucoma']
+218,REFUGE-Training400/n0179.jpg,[],['no glaucoma']
+219,REFUGE-Training400/n0180.jpg,[],['no glaucoma']
+220,REFUGE-Training400/n0181.jpg,[],['no glaucoma']
+221,REFUGE-Training400/n0182.jpg,[],['no glaucoma']
+222,REFUGE-Training400/n0183.jpg,[],['no glaucoma']
+223,REFUGE-Training400/n0184.jpg,[],['no glaucoma']
+224,REFUGE-Training400/n0185.jpg,[],['no glaucoma']
+225,REFUGE-Training400/n0186.jpg,[],['no glaucoma']
+226,REFUGE-Training400/n0187.jpg,[],['no glaucoma']
+227,REFUGE-Training400/n0188.jpg,[],['no glaucoma']
+228,REFUGE-Training400/n0189.jpg,[],['no glaucoma']
+229,REFUGE-Training400/n0190.jpg,[],['no glaucoma']
+230,REFUGE-Training400/n0191.jpg,[],['no glaucoma']
+231,REFUGE-Training400/n0192.jpg,[],['no glaucoma']
+232,REFUGE-Training400/n0193.jpg,[],['no glaucoma']
+233,REFUGE-Training400/n0194.jpg,[],['no glaucoma']
+234,REFUGE-Training400/n0195.jpg,[],['no glaucoma']
+235,REFUGE-Training400/n0196.jpg,[],['no glaucoma']
+236,REFUGE-Training400/n0197.jpg,[],['no glaucoma']
+237,REFUGE-Training400/n0198.jpg,[],['no glaucoma']
+238,REFUGE-Training400/n0199.jpg,[],['no glaucoma']
+239,REFUGE-Training400/n0200.jpg,[],['no glaucoma']
+240,REFUGE-Training400/n0201.jpg,[],['no glaucoma']
+241,REFUGE-Training400/n0202.jpg,[],['no glaucoma']
+242,REFUGE-Training400/n0203.jpg,[],['no glaucoma']
+243,REFUGE-Training400/n0204.jpg,[],['no glaucoma']
+244,REFUGE-Training400/n0205.jpg,[],['no glaucoma']
+245,REFUGE-Training400/n0206.jpg,[],['no glaucoma']
+246,REFUGE-Training400/n0207.jpg,[],['no glaucoma']
+247,REFUGE-Training400/n0208.jpg,[],['no glaucoma']
+248,REFUGE-Training400/n0209.jpg,[],['no glaucoma']
+249,REFUGE-Training400/n0210.jpg,[],['no glaucoma']
+250,REFUGE-Training400/n0211.jpg,[],['no glaucoma']
+251,REFUGE-Training400/n0212.jpg,[],['no glaucoma']
+252,REFUGE-Training400/n0213.jpg,[],['no glaucoma']
+253,REFUGE-Training400/n0214.jpg,[],['no glaucoma']
+254,REFUGE-Training400/n0215.jpg,[],['no glaucoma']
+255,REFUGE-Training400/n0216.jpg,[],['no glaucoma']
+256,REFUGE-Training400/n0217.jpg,[],['no glaucoma']
+257,REFUGE-Training400/n0218.jpg,[],['no glaucoma']
+258,REFUGE-Training400/n0219.jpg,[],['no glaucoma']
+259,REFUGE-Training400/n0220.jpg,[],['no glaucoma']
+260,REFUGE-Training400/n0221.jpg,[],['no glaucoma']
+261,REFUGE-Training400/n0222.jpg,[],['no glaucoma']
+262,REFUGE-Training400/n0223.jpg,[],['no glaucoma']
+263,REFUGE-Training400/n0224.jpg,[],['no glaucoma']
+264,REFUGE-Training400/n0225.jpg,[],['no glaucoma']
+265,REFUGE-Training400/n0226.jpg,[],['no glaucoma']
+266,REFUGE-Training400/n0227.jpg,[],['no glaucoma']
+267,REFUGE-Training400/n0228.jpg,[],['no glaucoma']
+268,REFUGE-Training400/n0229.jpg,[],['no glaucoma']
+269,REFUGE-Training400/n0230.jpg,[],['no glaucoma']
+270,REFUGE-Training400/n0231.jpg,[],['no glaucoma']
+271,REFUGE-Training400/n0232.jpg,[],['no glaucoma']
+272,REFUGE-Training400/n0233.jpg,[],['no glaucoma']
+273,REFUGE-Training400/n0234.jpg,[],['no glaucoma']
+274,REFUGE-Training400/n0235.jpg,[],['no glaucoma']
+275,REFUGE-Training400/n0236.jpg,[],['no glaucoma']
+276,REFUGE-Training400/n0237.jpg,[],['no glaucoma']
+277,REFUGE-Training400/n0238.jpg,[],['no glaucoma']
+278,REFUGE-Training400/n0239.jpg,[],['no glaucoma']
+279,REFUGE-Training400/n0240.jpg,[],['no glaucoma']
+280,REFUGE-Training400/n0241.jpg,[],['no glaucoma']
+281,REFUGE-Training400/n0242.jpg,[],['no glaucoma']
+282,REFUGE-Training400/n0243.jpg,[],['no glaucoma']
+283,REFUGE-Training400/n0244.jpg,[],['no glaucoma']
+284,REFUGE-Training400/n0245.jpg,[],['no glaucoma']
+285,REFUGE-Training400/n0246.jpg,[],['no glaucoma']
+286,REFUGE-Training400/n0247.jpg,[],['no glaucoma']
+287,REFUGE-Training400/n0248.jpg,[],['no glaucoma']
+288,REFUGE-Training400/n0249.jpg,[],['no glaucoma']
+289,REFUGE-Training400/n0250.jpg,[],['no glaucoma']
+290,REFUGE-Training400/n0251.jpg,[],['no glaucoma']
+291,REFUGE-Training400/n0252.jpg,[],['no glaucoma']
+292,REFUGE-Training400/n0253.jpg,[],['no glaucoma']
+293,REFUGE-Training400/n0254.jpg,[],['no glaucoma']
+294,REFUGE-Training400/n0255.jpg,[],['no glaucoma']
+295,REFUGE-Training400/n0256.jpg,[],['no glaucoma']
+296,REFUGE-Training400/n0257.jpg,[],['no glaucoma']
+297,REFUGE-Training400/n0258.jpg,[],['no glaucoma']
+298,REFUGE-Training400/n0259.jpg,[],['no glaucoma']
+299,REFUGE-Training400/n0260.jpg,[],['no glaucoma']
+300,REFUGE-Training400/n0261.jpg,[],['no glaucoma']
+301,REFUGE-Training400/n0262.jpg,[],['no glaucoma']
+302,REFUGE-Training400/n0263.jpg,[],['no glaucoma']
+303,REFUGE-Training400/n0264.jpg,[],['no glaucoma']
+304,REFUGE-Training400/n0265.jpg,[],['no glaucoma']
+305,REFUGE-Training400/n0266.jpg,[],['no glaucoma']
+306,REFUGE-Training400/n0267.jpg,[],['no glaucoma']
+307,REFUGE-Training400/n0268.jpg,[],['no glaucoma']
+308,REFUGE-Training400/n0269.jpg,[],['no glaucoma']
+309,REFUGE-Training400/n0270.jpg,[],['no glaucoma']
+310,REFUGE-Training400/n0271.jpg,[],['no glaucoma']
+311,REFUGE-Training400/n0272.jpg,[],['no glaucoma']
+312,REFUGE-Training400/n0273.jpg,[],['no glaucoma']
+313,REFUGE-Training400/n0274.jpg,[],['no glaucoma']
+314,REFUGE-Training400/n0275.jpg,[],['no glaucoma']
+315,REFUGE-Training400/n0276.jpg,[],['no glaucoma']
+316,REFUGE-Training400/n0277.jpg,[],['no glaucoma']
+317,REFUGE-Training400/n0278.jpg,[],['no glaucoma']
+318,REFUGE-Training400/n0279.jpg,[],['no glaucoma']
+319,REFUGE-Training400/n0280.jpg,[],['no glaucoma']
+320,REFUGE-Training400/n0281.jpg,[],['no glaucoma']
+321,REFUGE-Training400/n0282.jpg,[],['no glaucoma']
+322,REFUGE-Training400/n0283.jpg,[],['no glaucoma']
+323,REFUGE-Training400/n0284.jpg,[],['no glaucoma']
+324,REFUGE-Training400/n0285.jpg,[],['no glaucoma']
+325,REFUGE-Training400/n0286.jpg,[],['no glaucoma']
+326,REFUGE-Training400/n0287.jpg,[],['no glaucoma']
+327,REFUGE-Training400/n0288.jpg,[],['no glaucoma']
+328,REFUGE-Training400/n0289.jpg,[],['no glaucoma']
+329,REFUGE-Training400/n0290.jpg,[],['no glaucoma']
+330,REFUGE-Training400/n0291.jpg,[],['no glaucoma']
+331,REFUGE-Training400/n0292.jpg,[],['no glaucoma']
+332,REFUGE-Training400/n0293.jpg,[],['no glaucoma']
+333,REFUGE-Training400/n0294.jpg,[],['no glaucoma']
+334,REFUGE-Training400/n0295.jpg,[],['no glaucoma']
+335,REFUGE-Training400/n0296.jpg,[],['no glaucoma']
+336,REFUGE-Training400/n0297.jpg,[],['no glaucoma']
+337,REFUGE-Training400/n0298.jpg,[],['no glaucoma']
+338,REFUGE-Training400/n0299.jpg,[],['no glaucoma']
+339,REFUGE-Training400/n0300.jpg,[],['no glaucoma']
+340,REFUGE-Training400/n0301.jpg,[],['no glaucoma']
+341,REFUGE-Training400/n0302.jpg,[],['no glaucoma']
+342,REFUGE-Training400/n0303.jpg,[],['no glaucoma']
+343,REFUGE-Training400/n0304.jpg,[],['no glaucoma']
+344,REFUGE-Training400/n0305.jpg,[],['no glaucoma']
+345,REFUGE-Training400/n0306.jpg,[],['no glaucoma']
+346,REFUGE-Training400/n0307.jpg,[],['no glaucoma']
+347,REFUGE-Training400/n0308.jpg,[],['no glaucoma']
+348,REFUGE-Training400/n0309.jpg,[],['no glaucoma']
+349,REFUGE-Training400/n0310.jpg,[],['no glaucoma']
+350,REFUGE-Training400/n0311.jpg,[],['no glaucoma']
+351,REFUGE-Training400/n0312.jpg,[],['no glaucoma']
+352,REFUGE-Training400/n0313.jpg,[],['no glaucoma']
+353,REFUGE-Training400/n0314.jpg,[],['no glaucoma']
+354,REFUGE-Training400/n0315.jpg,[],['no glaucoma']
+355,REFUGE-Training400/n0316.jpg,[],['no glaucoma']
+356,REFUGE-Training400/n0317.jpg,[],['no glaucoma']
+357,REFUGE-Training400/n0318.jpg,[],['no glaucoma']
+358,REFUGE-Training400/n0319.jpg,[],['no glaucoma']
+359,REFUGE-Training400/n0320.jpg,[],['no glaucoma']
+360,REFUGE-Training400/n0321.jpg,[],['no glaucoma']
+361,REFUGE-Training400/n0322.jpg,[],['no glaucoma']
+362,REFUGE-Training400/n0323.jpg,[],['no glaucoma']
+363,REFUGE-Training400/n0324.jpg,[],['no glaucoma']
+364,REFUGE-Training400/n0325.jpg,[],['no glaucoma']
+365,REFUGE-Training400/n0326.jpg,[],['no glaucoma']
+366,REFUGE-Training400/n0327.jpg,[],['no glaucoma']
+367,REFUGE-Training400/n0328.jpg,[],['no glaucoma']
+368,REFUGE-Training400/n0329.jpg,[],['no glaucoma']
+369,REFUGE-Training400/n0330.jpg,[],['no glaucoma']
+370,REFUGE-Training400/n0331.jpg,[],['no glaucoma']
+371,REFUGE-Training400/n0332.jpg,[],['no glaucoma']
+372,REFUGE-Training400/n0333.jpg,[],['no glaucoma']
+373,REFUGE-Training400/n0334.jpg,[],['no glaucoma']
+374,REFUGE-Training400/n0335.jpg,[],['no glaucoma']
+375,REFUGE-Training400/n0336.jpg,[],['no glaucoma']
+376,REFUGE-Training400/n0337.jpg,[],['no glaucoma']
+377,REFUGE-Training400/n0338.jpg,[],['no glaucoma']
+378,REFUGE-Training400/n0339.jpg,[],['no glaucoma']
+379,REFUGE-Training400/n0340.jpg,[],['no glaucoma']
+380,REFUGE-Training400/n0341.jpg,[],['no glaucoma']
+381,REFUGE-Training400/n0342.jpg,[],['no glaucoma']
+382,REFUGE-Training400/n0343.jpg,[],['no glaucoma']
+383,REFUGE-Training400/n0344.jpg,[],['no glaucoma']
+384,REFUGE-Training400/n0345.jpg,[],['no glaucoma']
+385,REFUGE-Training400/n0346.jpg,[],['no glaucoma']
+386,REFUGE-Training400/n0347.jpg,[],['no glaucoma']
+387,REFUGE-Training400/n0348.jpg,[],['no glaucoma']
+388,REFUGE-Training400/n0349.jpg,[],['no glaucoma']
+389,REFUGE-Training400/n0350.jpg,[],['no glaucoma']
+390,REFUGE-Training400/n0351.jpg,[],['no glaucoma']
+391,REFUGE-Training400/n0352.jpg,[],['no glaucoma']
+392,REFUGE-Training400/n0353.jpg,[],['no glaucoma']
+393,REFUGE-Training400/n0354.jpg,[],['no glaucoma']
+394,REFUGE-Training400/n0355.jpg,[],['no glaucoma']
+395,REFUGE-Training400/n0356.jpg,[],['no glaucoma']
+396,REFUGE-Training400/n0357.jpg,[],['no glaucoma']
+397,REFUGE-Training400/n0358.jpg,[],['no glaucoma']
+398,REFUGE-Training400/n0359.jpg,[],['no glaucoma']
+399,REFUGE-Training400/n0360.jpg,[],['no glaucoma']

openclip_CITATION.cff

@@ -0,0 +1,36 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+
+cff-version: 1.1.0
+message: If you use this software, please cite it as below.
+authors:
+  - family-names: Ilharco
+    given-names: Gabriel
+  - family-names: Wortsman
+    given-names: Mitchell
+  - family-names: Wightman
+    given-names: Ross
+  - family-names: Gordon
+    given-names: Cade   
+  - family-names: Carlini
+    given-names: Nicholas
+  - family-names: Taori
+    given-names: Rohan
+  - family-names: Dave
+    given-names: Achal
+  - family-names: Shankar
+    given-names: Vaishaal
+  - family-names: Namkoong
+    given-names: Hongseok
+  - family-names: Miller
+    given-names: John
+  - family-names: Hajishirzi
+    given-names: Hannaneh
+  - family-names: Farhadi
+    given-names: Ali
+  - family-names: Schmidt
+    given-names: Ludwig
+title: OpenCLIP
+version: v0.1
+doi: 10.5281/zenodo.5143773
+date-released: 2021-07-28

openclip_LICENSE

@@ -0,0 +1,23 @@
+Copyright (c) 2012-2021 Gabriel Ilharco, Mitchell Wortsman, 
+Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, 
+John Miller, Hongseok Namkoong, Hannaneh Hajishirzi, Ali Farhadi, 
+Ludwig Schmidt
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

requirements.txt

@@ -0,0 +1,9 @@
+torch>=2.4.0
+torchvision
+regex
+ftfy
+tqdm
+webdataset>=0.2.5
+webdataset>=0.2.5
+pandas
+braceexpand

run_configs_400m.py

@@ -0,0 +1,136 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates
+
+# usage:
+# torchrun --nproc_per_node=4 src/training/main.py b16_400m my-experiment-name <path-to-metaclip-pretrained-checkpoint>
+from dataclasses import dataclass
+from configs import Config
+
+
+@dataclass
+class b32_400m(Config):
+    inmem=True
+    engine="train_one_epoch_ex"
+    eval_steps=5000
+    save_frequency=1
+    # First prepare UniMed-Dataset using instructions in the docs/PREPARE-UniMed-DATA.md and then,
+    # provide paths for each sub-part of UniMed dataset below.
+    train_data="/<dataset-path>/radimagenet_webdataset/dataset-{000001..001049}.tar::/<dataset-path>/chexpert_webdataset/dataset-{000001..000212}.tar::/<dataset-path>/openi_webdataset/dataset-{000001..000007}.tar::/<dataset-path>/chest_xray8_webdataset/dataset-{000001..000113}.tar::/<dataset-path>/mimic_cxr/dataset-{000001..000270}.tar::/<dataset-path>/roco_webdataset/dataset-{000001..000061}.tar::/<dataset-path>/pmc_clip_webdataset/dataset-{000001..001645}.tar::/<dataset-path>/llava_med_alignment_set_webdataset/dataset-{000001..000468}.tar::/<dataset-path>/llava_med_hq_60k_set_webdataset/dataset-{000001..000265}.tar::/<dataset-path>/quilt_webdataset/dataset-{000001..001018}.tar::/<dataset-path>/retina_part1_webdataset/dataset-{000001..000155}.tar::/<dataset-path>/retina_part2_webdataset/dataset-{000001..000013}.tar::/<dataset-path>/retina_part3_webdataset/dataset-{000001..000006}.tar"
+    # train_num_samples = 1049000 (radimagenet) + 212000 (chexpert) + 7000 (openi) + 113000 (chest-xray8) + 270000 (mimic-cxr) + 61000 (rocov2) + 1645000 (pmc-clip) + 468000 (llavamed-alignment) + 265000 (llava-medhq) + 1018000 (quilt) + 155000 (retina part 1) + 13000 (retina part 2) + 6000 (retina part 3)
+    # Total training samples must equal total dataset size
+    train_num_samples = 5282000
+    # By default, we provide equal weightage to all dataset parts
+    train_data_upsampling_factors = "1::1::1::1::1::1::1::1::1::1::1::1"
+    # ----------------------------------------
+    workers=8
+    batch_size=128
+    epochs= 10
+    eval_freq = 1
+    model="ViT-B-32-quickgelu"
+    name="ViT-B-32"
+    force_quick_gelu=True
+    warmup=2000
+    seed=0
+    local_loss=True
+    gather_with_grad=True
+    nodes=16
+    ngpus=4
+    imagenet_val = None
+    report_to = 'wandb'
+    tokenizer_context_length = 256
+    text_encode_model_name = 'microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract'
+
+@dataclass
+class b32_400m_eval(Config):
+    inmem=True
+    engine="train_one_epoch_ex"
+    eval_steps=5000
+    save_frequency=1
+    train_data=""
+    workers=8
+    eval_freq = 1
+    train_num_samples=400000000
+    batch_size=512
+    epochs=10
+    model="ViT-B-32-quickgelu"
+    name="ViT-B-32"
+    force_quick_gelu=True
+    warmup=2000
+    seed=0
+    local_loss=True
+    gather_with_grad=True
+    nodes=16
+    ngpus=4
+    imagenet_val = None
+    pretrained = '<path-to-metaclip-pretrained-weights-file>/b16_400m.pt'
+    text_encode_model_name = 'microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract'
+    tokenizer_context_length = 256
+
+
+@dataclass
+class b16_400m(b32_400m):
+    model="ViT-B-16-quickgelu"
+    name="ViT-B-16"
+    grad_checkpointing=True
+    # Change below
+    pretrained = '<path-to-metaclip-pretrained-weights-file>/b16_400m.pt'
+    text_encode_model_name = 'microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract'
+
+@dataclass
+class b16_400m_eval(b32_400m_eval):
+    model="ViT-B-16-quickgelu"
+    name="ViT-B-16"
+    grad_checkpointing=True
+    pretrained = '<path-to-metaclip-pretrained-weights-file>/b16_400m.pt'
+    text_encoder_model_name = 'microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract'
+
+
+@dataclass
+class l14_400m(b32_400m): 
+    model="ViT-L-14-336-quickgelu"
+    name="ViT-L-14"
+    lr=0.0004
+    grad_checkpointing=True
+    batch_size=128
+    nodes=16
+    ngpus=8
+    text_encoder_model_name = 'microsoft/BiomedNLP-BiomedBERT-large-uncased-abstract'
+
+@dataclass
+class l14_400m_eval(b32_400m_eval):
+    model="ViT-L-14-336-quickgelu"
+    name="ViT-L-14"
+    lr=0.0004
+    grad_checkpointing=True
+    batch_size=256
+    nodes=16
+    ngpus=8
+    text_encoder_model_name = 'microsoft/BiomedNLP-BiomedBERT-large-uncased-abstract'
+
+@dataclass
+class l14_400m_base_text_encoder(b32_400m):
+    model="ViT-L-14-336-quickgelu"
+    name="ViT-L-14"
+    lr=0.0004
+    grad_checkpointing=True
+    batch_size=128
+    nodes=16
+    ngpus=8
+    text_encoder_model_name = 'microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract'
+
+@dataclass
+class l14_400m_base_text_encoder_eval(b32_400m_eval):
+    model="ViT-L-14-336-quickgelu"
+    name="ViT-L-14"
+    lr=0.0004
+    grad_checkpointing=True
+    batch_size=256
+    nodes=16
+    ngpus=8
+    text_encode_model_name = 'microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract'
+
+if __name__ == "__main__":
+    import inspect
+    import sys
+    for name, obj in inspect.getmembers(sys.modules[__name__]):
+        if inspect.isfunction(obj):
+            print(name)

setup.py

@@ -0,0 +1,59 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+# All rights reserved.
+
+""" Setup
+"""
+from setuptools import setup, find_packages
+from codecs import open
+from os import path
+
+here = path.abspath(path.dirname(__file__))
+
+# Get the long description from the README file
+with open(path.join(here, 'README.md'), encoding='utf-8') as f:
+    long_description = f.read()
+
+exec(open('src/open_clip/version.py').read())
+setup(
+    name='open_clip_torch',
+    version=__version__,
+    description='OpenCLIP',
+    long_description=long_description,
+    long_description_content_type='text/markdown',
+    url='https://github.com/mlfoundations/open_clip',
+    author='',
+    author_email='',
+    classifiers=[
+        # How mature is this project? Common values are
+        #   3 - Alpha
+        #   4 - Beta
+        #   5 - Production/Stable
+        'Development Status :: 3 - Alpha',
+        'Intended Audience :: Education',
+        'Intended Audience :: Science/Research',
+        'License :: OSI Approved :: Apache Software License',
+        'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
+        'Topic :: Scientific/Engineering',
+        'Topic :: Scientific/Engineering :: Artificial Intelligence',
+        'Topic :: Software Development',
+        'Topic :: Software Development :: Libraries',
+        'Topic :: Software Development :: Libraries :: Python Modules',
+    ],
+
+    # Note that this is a string of words separated by whitespace, not a list.
+    keywords='CLIP pretrained',
+    package_dir={'': 'src'},
+    packages=find_packages(where='src', exclude=['training']),
+    include_package_data=True,
+    install_requires=[
+        'torch >= 1.9',
+        'torchvision',
+        'ftfy',
+        'regex',
+        'tqdm',
+    ],
+    python_requires='>=3.7',
+)

src/open_clip/__init__.py

@@ -0,0 +1,8 @@
+from .factory import list_models, create_model, create_model_and_transforms, add_model_config
+from .loss import ClipLoss
+from .model import CLIP, CLIPTextCfg, CLIPVisionCfg, convert_weights_to_fp16, trace_model
+from .openai import load_openai_model, list_openai_models
+from .pretrained import list_pretrained, list_pretrained_tag_models, list_pretrained_model_tags,\
+    get_pretrained_url, download_pretrained
+from .tokenizer import SimpleTokenizer, tokenize, HFTokenizer
+from .transform import image_transform, get_mean_std

src/open_clip/bpe_simple_vocab_16e6.txt.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:924691ac288e54409236115652ad4aa250f48203de50a9e4722a6ecd48d6804a
+size 1356917

src/open_clip/factory.py

@@ -0,0 +1,175 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates
+
+import json
+import logging
+import os
+import pathlib
+import re
+from copy import deepcopy
+from pathlib import Path
+from typing import Optional, Tuple
+
+import torch
+
+from . import model as model_zoo
+from .model import CLIP, convert_weights_to_fp16, resize_pos_embed
+from .openai import load_openai_model
+from .pretrained import get_pretrained_url, download_pretrained
+from .transform import image_transform
+
+
+_MODEL_CONFIG_PATHS = [Path(__file__).parent / f"model_configs/"]
+_MODEL_CONFIGS = {}  # directory (model_name: config) of model architecture configs
+
+
+def _natural_key(string_):
+    return [int(s) if s.isdigit() else s for s in re.split(r'(\d+)', string_.lower())]
+
+
+def _rescan_model_configs():
+    global _MODEL_CONFIGS
+
+    config_ext = ('.json',)
+    config_files = []
+    for config_path in _MODEL_CONFIG_PATHS:
+        if config_path.is_file() and config_path.suffix in config_ext:
+            config_files.append(config_path)
+        elif config_path.is_dir():
+            for ext in config_ext:
+                config_files.extend(config_path.glob(f'*{ext}'))
+
+    for cf in config_files:
+        with open(cf, 'r') as f:
+            model_cfg = json.load(f)
+            if all(a in model_cfg for a in ('embed_dim', 'vision_cfg', 'text_cfg')):
+                _MODEL_CONFIGS[cf.stem] = model_cfg
+
+    _MODEL_CONFIGS = {k: v for k, v in sorted(_MODEL_CONFIGS.items(), key=lambda x: _natural_key(x[0]))}
+
+
+_rescan_model_configs()  # initial populate of model config registry
+
+
+def load_state_dict(checkpoint_path: str, map_location='cpu'):
+    checkpoint = torch.load(checkpoint_path, map_location=map_location)
+    if isinstance(checkpoint, dict) and 'state_dict' in checkpoint:
+        state_dict = checkpoint['state_dict']
+    else:
+        state_dict = checkpoint
+    if next(iter(state_dict.items()))[0].startswith('module'):
+        state_dict = {k[7:]: v for k, v in state_dict.items()}
+    return state_dict
+
+
+def load_checkpoint(model, checkpoint_path, strict=False):
+    state_dict = load_state_dict(checkpoint_path)
+    resize_pos_embed(state_dict, model)
+    incompatible_keys = model.load_state_dict(state_dict, strict=strict)
+    return incompatible_keys
+
+
+def create_model(
+        model_name: str,
+        pretrained: str = '',
+        precision: str = 'fp32',
+        device: torch.device = torch.device('cpu'),
+        jit: bool = False,
+        force_quick_gelu: bool = False,
+        pretrained_image: bool = False,
+        clip_model: str = "CLIP",
+        text_encoder_name=None,
+):
+    model_name = model_name.replace('/', '-')  # for callers using old naming with / in ViT names
+
+    if pretrained.lower() == 'openai':
+        logging.info(f'Loading pretrained {model_name} from OpenAI.')
+        model = load_openai_model(model_name, device=device, jit=jit)
+        # See https://discuss.pytorch.org/t/valueerror-attemting-to-unscale-fp16-gradients/81372
+        if precision == "amp" or precision == "fp32":
+            model = model.float()
+    else:
+        if model_name in _MODEL_CONFIGS:
+            logging.info(f'Loading {model_name} model config.')
+            model_cfg = deepcopy(_MODEL_CONFIGS[model_name])
+        else:
+            logging.error(f'Model config for {model_name} not found; available models {list_models()}.')
+            raise RuntimeError(f'Model config for {model_name} not found.')
+
+        if force_quick_gelu:
+            # override for use of QuickGELU on non-OpenAI transformer models
+            model_cfg["quick_gelu"] = True
+
+        if pretrained_image:
+            if 'timm_model_name' in model_cfg.get('vision_cfg', {}):
+                # pretrained weight loading for timm models set via vision_cfg
+                model_cfg['vision_cfg']['timm_model_pretrained'] = True
+            else:
+                assert False, 'pretrained image towers currently only supported for timm models'
+        model_cfg['text_encoder_name'] = text_encoder_name
+        model_cls = getattr(model_zoo, clip_model)
+        model = model_cls(**model_cfg)  # via multiple CLIP models.
+
+        if pretrained:
+            checkpoint_path = ''
+            url = get_pretrained_url(model_name, pretrained)
+            if url:
+                checkpoint_path = download_pretrained(url)
+            elif os.path.exists(pretrained):
+                checkpoint_path = pretrained
+
+            if checkpoint_path:
+                logging.info(f'Loading pretrained {model_name} weights ({pretrained}).')
+                msg = load_checkpoint(model, checkpoint_path)
+                logging.info(f'{msg}')
+            else:
+                logging.warning(f'Pretrained weights ({pretrained}) not found for model {model_name}.')
+                raise RuntimeError(f'Pretrained weights ({pretrained}) not found for model {model_name}.')
+
+        model.to(device=device)
+        if precision == "fp16":
+            assert device.type != 'cpu'
+            convert_weights_to_fp16(model)
+
+        if jit:
+            model = torch.jit.script(model)
+
+    return model
+
+
+def create_model_and_transforms(
+        model_name: str,
+        pretrained: str = '',
+        precision: str = 'fp32',
+        device: torch.device = torch.device('cpu'),
+        jit: bool = False,
+        force_quick_gelu: bool = False,
+        pretrained_image: bool = False,
+        mean: Optional[Tuple[float, ...]] = None,
+        std: Optional[Tuple[float, ...]] = None,
+        inmem = False,
+        clip_model: str = "CLIP",
+        text_encoder_name=None,
+):
+    model = create_model(
+        model_name, pretrained, precision, device, jit,
+        force_quick_gelu=force_quick_gelu,
+        pretrained_image=pretrained_image,
+        clip_model=clip_model,
+        text_encoder_name=text_encoder_name,
+    )
+    preprocess_train = image_transform(model.visual.image_size, is_train=True, mean=mean, std=std, inmem=inmem)
+    preprocess_val = image_transform(model.visual.image_size, is_train=False, mean=mean, std=std)
+    return model, preprocess_train, preprocess_val
+
+
+def list_models():
+    """ enumerate available model architectures based on config files """
+    return list(_MODEL_CONFIGS.keys())
+
+
+def add_model_config(path):
+    """ add model config path or file and update registry """
+    if not isinstance(path, Path):
+        path = Path(path)
+    _MODEL_CONFIG_PATHS.append(path)
+    _rescan_model_configs()

src/open_clip/hf_configs.py

@@ -0,0 +1,67 @@
+# HF architecture dict:
+arch_dict = {
+    # https://huggingface.co/docs/transformers/model_doc/roberta#roberta
+    "roberta": {
+        "config_names": {
+            "context_length": "max_position_embeddings",
+            "vocab_size": "vocab_size",
+            "width": "hidden_size",
+            "heads": "num_attention_heads",
+            "layers": "num_hidden_layers",
+            "layer_attr": "layer",
+            "token_embeddings_attr": "embeddings"
+        },
+        "pooler": "mean_pooler",
+    },
+    # https://huggingface.co/docs/transformers/model_doc/xlm-roberta#transformers.XLMRobertaConfig
+    "xlm-roberta": {
+        "config_names": {
+            "context_length": "max_position_embeddings",
+            "vocab_size": "vocab_size",
+            "width": "hidden_size",
+            "heads": "num_attention_heads",
+            "layers": "num_hidden_layers",
+            "layer_attr": "layer",
+            "token_embeddings_attr": "embeddings"
+        },
+        "pooler": "mean_pooler",
+    },
+    # https://huggingface.co/docs/transformers/model_doc/mt5#mt5
+    "mt5": {
+        "config_names": {
+            # unlimited seqlen
+            # https://github.com/google-research/text-to-text-transfer-transformer/issues/273
+            # https://github.com/huggingface/transformers/blob/v4.24.0/src/transformers/models/t5/modeling_t5.py#L374
+            "context_length": "",
+            "vocab_size": "vocab_size",
+            "width": "d_model",
+            "heads": "num_heads",
+            "layers": "num_layers",
+            "layer_attr": "block",
+            "token_embeddings_attr": "embed_tokens"
+        },
+        "pooler": "mean_pooler",
+    },
+    # https://huggingface.co/docs/transformers/model_doc/bert
+    "bert": {
+        "config_names": {
+            "context_length": "max_position_embeddings",
+            "vocab_size": "vocab_size",
+            "width": "hidden_size",
+            "heads": "num_attention_heads",
+            "layers": "num_hidden_layers",
+        },
+        "pooler": "cls_pooler",
+    },
+    # https://huggingface.co/docs/transformers/model_doc/m2m_100
+    "m2m_100": {
+        "config_names": {
+            "context_length": "max_position_embeddings",
+            "vocab_size": "vocab_size",
+            "width": "d_model",
+            "heads": "encoder_attention_heads",
+            "layers": "encoder_layers",
+        },
+        "pooler": "cls_pooler",
+    },
+}

src/open_clip/hf_model.py

@@ -0,0 +1,193 @@
+""" huggingface model adapter
+
+Wraps HuggingFace transformers (https://github.com/huggingface/transformers) models for use as a text tower in CLIP model.
+"""
+import re
+
+import torch
+import torch.nn as nn
+from torch import TensorType
+
+try:
+    import transformers
+    from transformers import AutoModel, AutoTokenizer, AutoConfig, PretrainedConfig
+    from transformers.modeling_outputs import BaseModelOutput, BaseModelOutputWithPooling, \
+        BaseModelOutputWithPoolingAndCrossAttentions
+except ImportError as e:
+    transformers = None
+
+
+    class BaseModelOutput:
+        pass
+
+
+    class PretrainedConfig:
+        pass
+
+from .hf_configs import arch_dict
+
+
+# utils
+def _camel2snake(s):
+    return re.sub(r'(?<!^)(?=[A-Z])', '_', s).lower()
+
+
+# TODO: ?last - for gpt-like models
+_POOLERS = {}
+
+
+def register_pooler(cls):
+    """Decorator registering pooler class"""
+    _POOLERS[_camel2snake(cls.__name__)] = cls
+    return cls
+
+
+@register_pooler
+class MeanPooler(nn.Module):
+    """Mean pooling"""
+
+    def forward(self, x: BaseModelOutput, attention_mask: TensorType):
+        masked_output = x.last_hidden_state * attention_mask.unsqueeze(-1)
+        return masked_output.sum(dim=1) / attention_mask.sum(-1, keepdim=True)
+
+
+@register_pooler
+class MaxPooler(nn.Module):
+    """Max pooling"""
+
+    def forward(self, x: BaseModelOutput, attention_mask: TensorType):
+        masked_output = x.last_hidden_state.masked_fill(attention_mask.unsqueeze(-1), -torch.inf)
+        return masked_output.max(1).values
+
+
+@register_pooler
+class ClsPooler(nn.Module):
+    """CLS token pooling"""
+
+    def __init__(self, use_pooler_output=True):
+        super().__init__()
+        self.cls_token_position = 0
+        self.use_pooler_output = use_pooler_output
+
+    def forward(self, x: BaseModelOutput, attention_mask: TensorType):
+        if (self.use_pooler_output and
+                isinstance(x, (BaseModelOutputWithPooling, BaseModelOutputWithPoolingAndCrossAttentions)) and
+                (x.pooler_output is not None)
+        ):
+            return x.pooler_output
+
+        return x.last_hidden_state[:, self.cls_token_position, :]
+
+
+@register_pooler
+class ClsLastHiddenStatePooler(nn.Module):
+    """CLS token pooling
+    NOTE: this is equivalent to ClsPooler above with use_pooler_output=False
+    """
+
+    def __init__(self):
+        super().__init__()
+        self.cls_token_position = 0
+
+    def forward(self, x: BaseModelOutput, attention_mask: TensorType):
+        return x.last_hidden_state[:, self.cls_token_position, :]
+
+
+class HFTextEncoder(nn.Module):
+    """HuggingFace model adapter"""
+    output_tokens: torch.jit.Final[bool]
+
+    def __init__(
+            self,
+            model_name_or_path: str,
+            output_dim: int,
+            config: PretrainedConfig = None,
+            pooler_type: str = None,
+            proj_type: str = None,
+            pretrained: bool = True,
+            output_tokens: bool = False,
+    ):
+        super().__init__()
+        self.output_tokens = output_tokens
+        self.output_dim = output_dim
+
+        # TODO: find better way to get this information
+        uses_transformer_pooler = (pooler_type == "cls_pooler")
+
+        if transformers is None:
+            raise RuntimeError("Please `pip install transformers` to use pre-trained HuggingFace models")
+        if config is None:
+            self.config = AutoConfig.from_pretrained(model_name_or_path)
+            create_func, model_args = (AutoModel.from_pretrained, model_name_or_path) if pretrained else (
+                AutoModel.from_config, self.config)
+            # TODO: do all model configs have this attribute? PretrainedConfig does so yes??
+            if hasattr(self.config, "is_encoder_decoder") and self.config.is_encoder_decoder:
+                self.transformer = create_func(model_args)
+                self.transformer = self.transformer.encoder
+            else:
+                self.transformer = create_func(model_args, add_pooling_layer=uses_transformer_pooler)
+        else:
+            self.config = config
+            self.transformer = AutoModel.from_config(config)
+        if pooler_type is None:  # get default arch pooler
+            pooler_type = (arch_dict[self.config.model_type]["pooler"])
+
+        # FIXME downstream users of OpenCLIP models use these attr, need to verify valid across all models
+        self.vocab_size = getattr(self.config, 'vocab_size', 0)
+        self.context_length = getattr(self.config, 'max_position_embeddings', 0)
+
+        self.pooler = _POOLERS[pooler_type]()
+
+        d_model = getattr(self.config, arch_dict[self.config.model_type]["config_names"]["width"])
+        if (d_model == output_dim) and (proj_type is None):  # do we always need a proj?
+            self.proj = nn.Identity()
+        elif proj_type == 'linear':
+            self.proj = nn.Linear(d_model, output_dim, bias=False)
+        elif proj_type == 'mlp':
+            hidden_size = (d_model + output_dim) // 2
+            self.proj = nn.Sequential(
+                nn.Linear(d_model, hidden_size, bias=False),
+                nn.GELU(),
+                nn.Linear(hidden_size, output_dim, bias=False),
+            )
+
+    def forward(self, x: TensorType):
+        attn_mask = (x != self.config.pad_token_id).long()
+        out = self.transformer(input_ids=x, attention_mask=attn_mask)
+        pooled_out = self.pooler(out, attn_mask)
+        projected = self.proj(pooled_out)
+
+        seq_len = out.last_hidden_state.shape[1]
+        tokens = (
+            out.last_hidden_state[:, torch.arange(seq_len) != self.pooler.cls_token_position, :]
+            if type(self.pooler) == ClsPooler
+            else out.last_hidden_state
+        )
+
+        if self.output_tokens:
+            return projected, tokens
+        return projected
+
+    def lock(self, unlocked_layers: int = 0, freeze_layer_norm: bool = True):
+        if not unlocked_layers:  # full freezing
+            for n, p in self.transformer.named_parameters():
+                p.requires_grad = (not freeze_layer_norm) if "LayerNorm" in n.split(".") else False
+            return
+
+        encoder = self.transformer.encoder if hasattr(self.transformer, 'encoder') else self.transformer
+        layer_list = getattr(encoder, arch_dict[self.config.model_type]["config_names"]["layer_attr"])
+        print(f"Unlocking {unlocked_layers}/{len(layer_list) + 1} layers of hf model")
+        embeddings = getattr(
+            self.transformer, arch_dict[self.config.model_type]["config_names"]["token_embeddings_attr"])
+        modules = [embeddings, *layer_list][:-unlocked_layers]
+        # freeze layers
+        for module in modules:
+            for n, p in module.named_parameters():
+                p.requires_grad = (not freeze_layer_norm) if "LayerNorm" in n.split(".") else False
+
+    @torch.jit.ignore
+    def set_grad_checkpointing(self, enable=True):
+        self.transformer.gradient_checkpointing_enable()
+
+    def init_parameters(self):
+        pass

src/open_clip/loss.py

@@ -0,0 +1,123 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates
+
+import torch
+import torch.nn as nn
+from torch.nn import functional as F
+
+try:
+    import torch.distributed.nn
+    from torch import distributed as dist
+    has_distributed = True
+except ImportError:
+    has_distributed = False
+
+try:
+    import horovod.torch as hvd
+except ImportError:
+    hvd = None
+
+
+def gather_features(
+        image_features,
+        text_features,
+        local_loss=False,
+        gather_with_grad=False,
+        rank=0,
+        world_size=1,
+        use_horovod=False
+):
+    assert has_distributed, 'torch.distributed did not import correctly, please use a PyTorch version with support.'
+    if use_horovod:
+        assert hvd is not None, 'Please install horovod'
+        if gather_with_grad:
+            all_image_features = hvd.allgather(image_features)
+            all_text_features = hvd.allgather(text_features)
+        else:
+            with torch.no_grad():
+                all_image_features = hvd.allgather(image_features)
+                all_text_features = hvd.allgather(text_features)
+            if not local_loss:
+                # ensure grads for local rank when all_* features don't have a gradient
+                gathered_image_features = list(all_image_features.chunk(world_size, dim=0))
+                gathered_text_features = list(all_text_features.chunk(world_size, dim=0))
+                gathered_image_features[rank] = image_features
+                gathered_text_features[rank] = text_features
+                all_image_features = torch.cat(gathered_image_features, dim=0)
+                all_text_features = torch.cat(gathered_text_features, dim=0)
+    else:
+        # We gather tensors from all gpus
+        if gather_with_grad:
+            all_image_features = torch.cat(torch.distributed.nn.all_gather(image_features), dim=0)
+            all_text_features = torch.cat(torch.distributed.nn.all_gather(text_features), dim=0)
+        else:
+            gathered_image_features = [torch.zeros_like(image_features) for _ in range(world_size)]
+            gathered_text_features = [torch.zeros_like(text_features) for _ in range(world_size)]
+            dist.all_gather(gathered_image_features, image_features)
+            dist.all_gather(gathered_text_features, text_features)
+            if not local_loss:
+                # ensure grads for local rank when all_* features don't have a gradient
+                gathered_image_features[rank] = image_features
+                gathered_text_features[rank] = text_features
+            all_image_features = torch.cat(gathered_image_features, dim=0)
+            all_text_features = torch.cat(gathered_text_features, dim=0)
+
+    return all_image_features, all_text_features
+
+
+class ClipLoss(nn.Module):
+
+    def __init__(
+            self,
+            local_loss=False,
+            gather_with_grad=False,
+            cache_labels=False,
+            rank=0,
+            world_size=1,
+            use_horovod=False,
+    ):
+        super().__init__()
+        self.local_loss = local_loss
+        self.gather_with_grad = gather_with_grad
+        self.cache_labels = cache_labels
+        self.rank = rank
+        self.world_size = world_size
+        self.use_horovod = use_horovod
+
+        # cache state
+        self.prev_num_logits = 0
+        self.labels = {}
+
+    def forward(self, image_features, text_features, logit_scale):
+        device = image_features.device
+        if self.world_size > 1:
+            all_image_features, all_text_features = gather_features(
+                image_features, text_features,
+                self.local_loss, self.gather_with_grad, self.rank, self.world_size, self.use_horovod)
+
+            if self.local_loss:
+                logits_per_image = logit_scale * image_features @ all_text_features.T
+                logits_per_text = logit_scale * text_features @ all_image_features.T
+            else:
+                logits_per_image = logit_scale * all_image_features @ all_text_features.T
+                logits_per_text = logits_per_image.T
+        else:
+            logits_per_image = logit_scale * image_features @ text_features.T
+            logits_per_text = logit_scale * text_features @ image_features.T
+
+        # calculated ground-truth and cache if enabled
+        num_logits = logits_per_image.shape[0]
+        if self.prev_num_logits != num_logits or device not in self.labels:
+            labels = torch.arange(num_logits, device=device, dtype=torch.long)
+            if self.world_size > 1 and self.local_loss:
+                labels = labels + num_logits * self.rank
+            if self.cache_labels:
+                self.labels[device] = labels
+                self.prev_num_logits = num_logits
+        else:
+            labels = self.labels[device]
+
+        total_loss = (
+            F.cross_entropy(logits_per_image, labels) +
+            F.cross_entropy(logits_per_text, labels)
+            ) / 2
+        return total_loss

src/open_clip/model.py

@@ -0,0 +1,574 @@
+""" CLIP Model
+
+Adapted from https://github.com/openai/CLIP. Originally MIT License, Copyright (c) 2021 OpenAI.
+"""
+# Copyright (c) Meta Platforms, Inc. and affiliates
+from .hf_model import HFTextEncoder
+from collections import OrderedDict
+from dataclasses import dataclass
+import logging
+import math
+from typing import Tuple, Union, Callable, Optional
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+from torch import nn
+from torch.utils.checkpoint import checkpoint
+
+from .timm_model import TimmModel
+from .utils import freeze_batch_norm_2d, to_2tuple
+
+
+class Bottleneck(nn.Module):
+    expansion = 4
+
+    def __init__(self, inplanes, planes, stride=1):
+        super().__init__()
+
+        # all conv layers have stride 1. an avgpool is performed after the second convolution when stride > 1
+        self.conv1 = nn.Conv2d(inplanes, planes, 1, bias=False)
+        self.bn1 = nn.BatchNorm2d(planes)
+        self.relu1 = nn.ReLU(inplace=True)
+
+        self.conv2 = nn.Conv2d(planes, planes, 3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(planes)
+        self.relu2 = nn.ReLU(inplace=True)
+
+        self.avgpool = nn.AvgPool2d(stride) if stride > 1 else nn.Identity()
+
+        self.conv3 = nn.Conv2d(planes, planes * self.expansion, 1, bias=False)
+        self.bn3 = nn.BatchNorm2d(planes * self.expansion)
+        self.relu3 = nn.ReLU(inplace=True)
+
+        self.downsample = None
+        self.stride = stride
+
+        if stride > 1 or inplanes != planes * Bottleneck.expansion:
+            # downsampling layer is prepended with an avgpool, and the subsequent convolution has stride 1
+            self.downsample = nn.Sequential(OrderedDict([
+                ("-1", nn.AvgPool2d(stride)),
+                ("0", nn.Conv2d(inplanes, planes * self.expansion, 1, stride=1, bias=False)),
+                ("1", nn.BatchNorm2d(planes * self.expansion))
+            ]))
+
+    def forward(self, x: torch.Tensor):
+        identity = x
+
+        out = self.relu1(self.bn1(self.conv1(x)))
+        out = self.relu2(self.bn2(self.conv2(out)))
+        out = self.avgpool(out)
+        out = self.bn3(self.conv3(out))
+
+        if self.downsample is not None:
+            identity = self.downsample(x)
+
+        out += identity
+        out = self.relu3(out)
+        return out
+
+
+class AttentionPool2d(nn.Module):
+    def __init__(self, spacial_dim: int, embed_dim: int, num_heads: int, output_dim: int = None):
+        super().__init__()
+        self.positional_embedding = nn.Parameter(torch.randn(spacial_dim ** 2 + 1, embed_dim) / embed_dim ** 0.5)
+        self.k_proj = nn.Linear(embed_dim, embed_dim)
+        self.q_proj = nn.Linear(embed_dim, embed_dim)
+        self.v_proj = nn.Linear(embed_dim, embed_dim)
+        self.c_proj = nn.Linear(embed_dim, output_dim or embed_dim)
+        self.num_heads = num_heads
+
+    def forward(self, x):
+        x = x.reshape(x.shape[0], x.shape[1], x.shape[2] * x.shape[3]).permute(2, 0, 1)  # NCHW -> (HW)NC
+        x = torch.cat([x.mean(dim=0, keepdim=True), x], dim=0)  # (HW+1)NC
+        x = x + self.positional_embedding[:, None, :].to(x.dtype)  # (HW+1)NC
+        x, _ = F.multi_head_attention_forward(
+            query=x, key=x, value=x,
+            embed_dim_to_check=x.shape[-1],
+            num_heads=self.num_heads,
+            q_proj_weight=self.q_proj.weight,
+            k_proj_weight=self.k_proj.weight,
+            v_proj_weight=self.v_proj.weight,
+            in_proj_weight=None,
+            in_proj_bias=torch.cat([self.q_proj.bias, self.k_proj.bias, self.v_proj.bias]),
+            bias_k=None,
+            bias_v=None,
+            add_zero_attn=False,
+            dropout_p=0,
+            out_proj_weight=self.c_proj.weight,
+            out_proj_bias=self.c_proj.bias,
+            use_separate_proj_weight=True,
+            training=self.training,
+            need_weights=False
+        )
+
+        return x[0]
+
+
+class ModifiedResNet(nn.Module):
+    """
+    A ResNet class that is similar to torchvision's but contains the following changes:
+    - There are now 3 "stem" convolutions as opposed to 1, with an average pool instead of a max pool.
+    - Performs anti-aliasing strided convolutions, where an avgpool is prepended to convolutions with stride > 1
+    - The final pooling layer is a QKV attention instead of an average pool
+    """
+
+    def __init__(self, layers, output_dim, heads, image_size=224, width=64):
+        super().__init__()
+        self.output_dim = output_dim
+        self.image_size = image_size
+
+        # the 3-layer stem
+        self.conv1 = nn.Conv2d(3, width // 2, kernel_size=3, stride=2, padding=1, bias=False)
+        self.bn1 = nn.BatchNorm2d(width // 2)
+        self.relu1 = nn.ReLU(inplace=True)
+        self.conv2 = nn.Conv2d(width // 2, width // 2, kernel_size=3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(width // 2)
+        self.relu2 = nn.ReLU(inplace=True)
+        self.conv3 = nn.Conv2d(width // 2, width, kernel_size=3, padding=1, bias=False)
+        self.bn3 = nn.BatchNorm2d(width)
+        self.relu3 = nn.ReLU(inplace=True)
+        self.avgpool = nn.AvgPool2d(2)
+
+        # residual layers
+        self._inplanes = width  # this is a *mutable* variable used during construction
+        self.layer1 = self._make_layer(width, layers[0])
+        self.layer2 = self._make_layer(width * 2, layers[1], stride=2)
+        self.layer3 = self._make_layer(width * 4, layers[2], stride=2)
+        self.layer4 = self._make_layer(width * 8, layers[3], stride=2)
+
+        embed_dim = width * 32  # the ResNet feature dimension
+        self.attnpool = AttentionPool2d(image_size // 32, embed_dim, heads, output_dim)
+
+        self.init_parameters()
+
+    def _make_layer(self, planes, blocks, stride=1):
+        layers = [Bottleneck(self._inplanes, planes, stride)]
+
+        self._inplanes = planes * Bottleneck.expansion
+        for _ in range(1, blocks):
+            layers.append(Bottleneck(self._inplanes, planes))
+
+        return nn.Sequential(*layers)
+
+    def init_parameters(self):
+        if self.attnpool is not None:
+            std = self.attnpool.c_proj.in_features ** -0.5
+            nn.init.normal_(self.attnpool.q_proj.weight, std=std)
+            nn.init.normal_(self.attnpool.k_proj.weight, std=std)
+            nn.init.normal_(self.attnpool.v_proj.weight, std=std)
+            nn.init.normal_(self.attnpool.c_proj.weight, std=std)
+
+        for resnet_block in [self.layer1, self.layer2, self.layer3, self.layer4]:
+            for name, param in resnet_block.named_parameters():
+                if name.endswith("bn3.weight"):
+                    nn.init.zeros_(param)
+
+    def lock(self, unlocked_groups=0, freeze_bn_stats=False):
+        assert unlocked_groups == 0, 'partial locking not currently supported for this model'
+        for param in self.parameters():
+            param.requires_grad = False
+        if freeze_bn_stats:
+            freeze_batch_norm_2d(self)
+
+    @torch.jit.ignore
+    def set_grad_checkpointing(self, enable=True):
+        # FIXME support for non-transformer
+        pass
+
+    def stem(self, x):
+        x = self.relu1(self.bn1(self.conv1(x)))
+        x = self.relu2(self.bn2(self.conv2(x)))
+        x = self.relu3(self.bn3(self.conv3(x)))
+        x = self.avgpool(x)
+        return x
+
+    def forward(self, x):
+        x = self.stem(x)
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.layer3(x)
+        x = self.layer4(x)
+        x = self.attnpool(x)
+
+        return x
+
+
+class LayerNorm(nn.LayerNorm):
+    """Subclass torch's LayerNorm to handle fp16."""
+
+    def forward(self, x: torch.Tensor):
+        orig_type = x.dtype
+        x = F.layer_norm(x, self.normalized_shape, self.weight, self.bias, self.eps)
+        return x.to(orig_type)
+
+
+class QuickGELU(nn.Module):
+    # NOTE This is slower than nn.GELU or nn.SiLU and uses more GPU memory
+    def forward(self, x: torch.Tensor):
+        return x * torch.sigmoid(1.702 * x)
+
+
+class ResidualAttentionBlock(nn.Module):
+    def __init__(self, d_model: int, n_head: int, mlp_ratio: float = 4.0, act_layer: Callable = nn.GELU):
+        super().__init__()
+
+        self.attn = nn.MultiheadAttention(d_model, n_head)
+        self.ln_1 = LayerNorm(d_model)
+        mlp_width = int(d_model * mlp_ratio)
+        self.mlp = nn.Sequential(OrderedDict([
+            ("c_fc", nn.Linear(d_model, mlp_width)),
+            ("gelu", act_layer()),
+            ("c_proj", nn.Linear(mlp_width, d_model))
+        ]))
+        self.ln_2 = LayerNorm(d_model)
+
+    def attention(self, x: torch.Tensor, attn_mask: Optional[torch.Tensor] = None):
+        return self.attn(x, x, x, need_weights=False, attn_mask=attn_mask)[0]
+
+    def forward(self, x: torch.Tensor, attn_mask: Optional[torch.Tensor] = None):
+        x = x + self.attention(self.ln_1(x), attn_mask=attn_mask)
+        x = x + self.mlp(self.ln_2(x))
+        return x
+
+
+class Transformer(nn.Module):
+    def __init__(self, width: int, layers: int, heads: int,  mlp_ratio: float = 4.0, act_layer: Callable = nn.GELU):
+        super().__init__()
+        self.width = width
+        self.layers = layers
+        self.grad_checkpointing = False
+
+        self.resblocks = nn.ModuleList([
+            ResidualAttentionBlock(width, heads, mlp_ratio, act_layer=act_layer)
+            for _ in range(layers)
+        ])
+
+    def forward(self, x: torch.Tensor, attn_mask: Optional[torch.Tensor] = None):
+        for r in self.resblocks:
+            if self.grad_checkpointing and not torch.jit.is_scripting():
+                x = checkpoint(r, x, attn_mask)
+            else:
+                x = r(x, attn_mask=attn_mask)
+        return x
+
+
+class VisualTransformer(nn.Module):
+    def __init__(
+            self, image_size: int, patch_size: int, width: int, layers: int, heads: int, mlp_ratio: float,
+            output_dim: int, act_layer: Callable = nn.GELU):
+        super().__init__()
+        self.image_size = to_2tuple(image_size)
+        self.patch_size = to_2tuple(patch_size)
+        self.grid_size = (self.image_size[0] // self.patch_size[0], self.image_size[1] // self.patch_size[1])
+        self.output_dim = output_dim
+        self.conv1 = nn.Conv2d(in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, bias=False)
+
+        scale = width ** -0.5
+        self.class_embedding = nn.Parameter(scale * torch.randn(width))
+        self.positional_embedding = nn.Parameter(scale * torch.randn(self.grid_size[0] * self.grid_size[1] + 1, width))
+        self.ln_pre = LayerNorm(width)
+
+        self.transformer = Transformer(width, layers, heads, mlp_ratio, act_layer=act_layer)
+
+        self.ln_post = LayerNorm(width)
+        self.proj = nn.Parameter(scale * torch.randn(width, output_dim))
+
+    def lock(self, unlocked_groups=0, freeze_bn_stats=False):
+        assert unlocked_groups == 0, 'partial locking not currently supported for this model'
+        for param in self.parameters():
+            param.requires_grad = False
+
+    @torch.jit.ignore
+    def set_grad_checkpointing(self, enable=True):
+        self.transformer.grad_checkpointing = enable
+
+    def forward(self, x: torch.Tensor):
+        x = self.conv1(x)  # shape = [*, width, grid, grid]
+        x = x.reshape(x.shape[0], x.shape[1], -1)  # shape = [*, width, grid ** 2]
+        x = x.permute(0, 2, 1)  # shape = [*, grid ** 2, width]
+        x = torch.cat(
+            [self.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device),
+             x], dim=1)  # shape = [*, grid ** 2 + 1, width]
+        x = x + self.positional_embedding.to(x.dtype)
+        x = self.ln_pre(x)
+
+        x = x.permute(1, 0, 2)  # NLD -> LND
+        x = self.transformer(x)
+        x = x.permute(1, 0, 2)  # LND -> NLD
+
+        x = self.ln_post(x[:, 0, :])
+
+        if self.proj is not None:
+            x = x @ self.proj
+
+        return x
+
+
+@dataclass
+class CLIPVisionCfg:
+    layers: Union[Tuple[int, int, int, int], int] = 12
+    width: int = 768
+    head_width: int = 64
+    mlp_ratio: float = 4.0
+    patch_size: int = 16
+    image_size: Union[Tuple[int, int], int] = 224
+    timm_model_name: str = None  # a valid model name overrides layers, width, patch_size
+    timm_model_pretrained: bool = False  # use (imagenet) pretrained weights for named model
+    timm_pool: str = 'avg'  # feature pooling for timm model ('abs_attn', 'rot_attn', 'avg', '')
+    timm_proj: str = 'linear'  # linear projection for timm model output ('linear', 'mlp', '')
+
+
+@dataclass
+class CLIPTextCfg:
+    context_length: int = 77
+    vocab_size: int = 49408
+    width: int = 512
+    heads: int = 8
+    layers: int = 12
+
+
+class CLIP(nn.Module):
+    def __init__(
+            self,
+            embed_dim: int,
+            vision_cfg: CLIPVisionCfg,
+            text_cfg: CLIPTextCfg,
+            quick_gelu: bool = False,
+            text_encoder_name = None,
+    ):
+        super().__init__()
+        if isinstance(vision_cfg, dict):
+            vision_cfg = CLIPVisionCfg(**vision_cfg)
+        if isinstance(text_cfg, dict):
+            text_cfg = CLIPTextCfg(**text_cfg)
+
+        self.context_length = text_cfg.context_length
+
+        # OpenAI models are pretrained w/ QuickGELU 
+        # NOTE: timm models always use native GELU regardless of quick_gelu flag.
+        act_layer = QuickGELU if quick_gelu else nn.GELU
+
+        if vision_cfg.timm_model_name:
+            self.visual = TimmModel(
+                vision_cfg.timm_model_name,
+                pretrained=vision_cfg.timm_model_pretrained,
+                pool=vision_cfg.timm_pool,
+                proj=vision_cfg.timm_proj,
+                embed_dim=embed_dim,
+                image_size=vision_cfg.image_size
+            )
+            act_layer = nn.GELU  # so that text transformer doesn't use QuickGELU w/ timm models
+        elif isinstance(vision_cfg.layers, (tuple, list)):
+            vision_heads = vision_cfg.width * 32 // vision_cfg.head_width
+            self.visual = ModifiedResNet(
+                layers=vision_cfg.layers,
+                output_dim=embed_dim,
+                heads=vision_heads,
+                image_size=vision_cfg.image_size,
+                width=vision_cfg.width
+            )
+        else:
+            vision_heads = vision_cfg.width // vision_cfg.head_width
+            self.visual = VisualTransformer(
+                image_size=vision_cfg.image_size,
+                patch_size=vision_cfg.patch_size,
+                width=vision_cfg.width,
+                layers=vision_cfg.layers,
+                heads=vision_heads,
+                mlp_ratio=vision_cfg.mlp_ratio,
+                output_dim=embed_dim,
+                act_layer=act_layer,
+            )
+
+        self.text_encoder = HFTextEncoder(
+            text_encoder_name,
+            output_dim=embed_dim,
+            proj_type='mlp',
+            pooler_type='cls_last_hidden_state_pooler',
+            pretrained=True,
+            output_tokens=False,
+        )
+
+        self.logit_scale = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))
+
+        self.init_parameters()
+
+    def init_parameters(self):
+        # nn.init.normal_(self.token_embedding.weight, std=0.02)
+        # nn.init.normal_(self.positional_embedding, std=0.01)
+        nn.init.constant_(self.logit_scale, np.log(1 / 0.07))
+
+        if hasattr(self.visual, 'init_parameters'):
+            self.visual.init_parameters()
+
+
+    def build_attention_mask(self):
+        # lazily create causal attention mask, with full attention between the vision tokens
+        # pytorch uses additive attention mask; fill with -inf
+        mask = torch.empty(self.context_length, self.context_length)
+        mask.fill_(float("-inf"))
+        mask.triu_(1)  # zero out the lower diagonal
+        return mask
+
+    def lock_image_tower(self, unlocked_groups=0, freeze_bn_stats=False):
+        # lock image tower as per LiT - https://arxiv.org/abs/2111.07991
+        self.visual.lock(unlocked_groups=unlocked_groups, freeze_bn_stats=freeze_bn_stats)
+
+    @torch.jit.ignore
+    def set_grad_checkpointing(self, enable=True):
+        self.visual.set_grad_checkpointing(enable)
+        self.text_encoder.grad_checkpointing = enable
+
+    def encode_image(self, image):
+        return self.visual(image)
+
+
+    def encode_text(self, text):
+        text_features = self.text_encoder(text)  # For BioMedCLIP text encoder
+        return text_features
+
+    def forward(self, image, text, clamp_logit_scale_to=None):
+        if image is not None:
+            image_features = self.encode_image(image)
+            image_features = F.normalize(image_features, dim=-1)
+        else:
+            image_features = None
+        if text is not None:
+            # text_features = self.encode_text(text) # For normal CLIP text encoder
+            text_features = self.text_encoder(text)  # For BioMedCLIP text encoder
+            text_features = F.normalize(text_features, dim=-1)
+        else:
+            text_features = None
+        if clamp_logit_scale_to is not None:
+            with torch.no_grad():
+                self.logit_scale.data.clamp_(0, clamp_logit_scale_to)
+        return image_features, text_features, self.logit_scale.exp()
+
+
+def convert_weights_to_fp16(model: nn.Module):
+    """Convert applicable model parameters to fp16"""
+
+    def _convert_weights_to_fp16(l):
+        if isinstance(l, (nn.Conv1d, nn.Conv2d, nn.Linear)):
+            l.weight.data = l.weight.data.half()
+            if l.bias is not None:
+                l.bias.data = l.bias.data.half()
+
+        if isinstance(l, nn.MultiheadAttention):
+            for attr in [*[f"{s}_proj_weight" for s in ["in", "q", "k", "v"]], "in_proj_bias", "bias_k", "bias_v"]:
+                tensor = getattr(l, attr)
+                if tensor is not None:
+                    tensor.data = tensor.data.half()
+
+        for name in ["text_projection", "proj"]:
+            if hasattr(l, name):
+                attr = getattr(l, name)
+                if attr is not None:
+                    attr.data = attr.data.half()
+
+    model.apply(_convert_weights_to_fp16)
+
+
+def build_model_from_openai_state_dict(state_dict: dict):
+    vit = "visual.proj" in state_dict
+
+    if vit:
+        vision_width = state_dict["visual.conv1.weight"].shape[0]
+        vision_layers = len(
+            [k for k in state_dict.keys() if k.startswith("visual.") and k.endswith(".attn.in_proj_weight")])
+        vision_patch_size = state_dict["visual.conv1.weight"].shape[-1]
+        grid_size = round((state_dict["visual.positional_embedding"].shape[0] - 1) ** 0.5)
+        image_size = vision_patch_size * grid_size
+    else:
+        counts: list = [
+            len(set(k.split(".")[2] for k in state_dict if k.startswith(f"visual.layer{b}"))) for b in [1, 2, 3, 4]]
+        vision_layers = tuple(counts)
+        vision_width = state_dict["visual.layer1.0.conv1.weight"].shape[0]
+        output_width = round((state_dict["visual.attnpool.positional_embedding"].shape[0] - 1) ** 0.5)
+        vision_patch_size = None
+        assert output_width ** 2 + 1 == state_dict["visual.attnpool.positional_embedding"].shape[0]
+        image_size = output_width * 32
+
+    embed_dim = state_dict["text_projection"].shape[1]
+    context_length = state_dict["positional_embedding"].shape[0]
+    vocab_size = state_dict["token_embedding.weight"].shape[0]
+    transformer_width = state_dict["ln_final.weight"].shape[0]
+    transformer_heads = transformer_width // 64
+    transformer_layers = len(set(k.split(".")[2] for k in state_dict if k.startswith(f"transformer.resblocks")))
+
+    vision_cfg = CLIPVisionCfg(
+        layers=vision_layers,
+        width=vision_width,
+        patch_size=vision_patch_size,
+        image_size=image_size,
+    )
+    text_cfg = CLIPTextCfg(
+        context_length=context_length,
+        vocab_size=vocab_size,
+        width=transformer_width,
+        heads=transformer_heads,
+        layers=transformer_layers
+    )
+    model = CLIP(
+        embed_dim,
+        vision_cfg=vision_cfg,
+        text_cfg=text_cfg,
+        quick_gelu=True,  # OpenAI models were trained with QuickGELU
+    )
+
+    for key in ["input_resolution", "context_length", "vocab_size"]:
+        state_dict.pop(key, None)
+
+    convert_weights_to_fp16(model)
+    model.load_state_dict(state_dict)
+    return model.eval()
+
+
+def trace_model(model, batch_size=256, device=torch.device('cpu')):
+    model.eval()
+    image_size = model.visual.image_size
+    example_images = torch.ones((batch_size, 3, image_size, image_size), device=device)
+    example_text = torch.zeros((batch_size, model.context_length), dtype=torch.int, device=device)
+    model = torch.jit.trace_module(
+        model,
+        inputs=dict(
+            forward=(example_images, example_text),
+            encode_text=(example_text,),
+            encode_image=(example_images,)
+        ))
+    model.visual.image_size = image_size
+    return model
+
+
+def resize_pos_embed(state_dict, model, interpolation: str = 'bicubic', seq_dim=1):
+    # Rescale the grid of position embeddings when loading from state_dict
+    old_pos_embed = state_dict.get('visual.positional_embedding', None)
+    if old_pos_embed is None or not hasattr(model.visual, 'grid_size'):
+        return
+    grid_size = to_2tuple(model.visual.grid_size)
+    extra_tokens = 1  # FIXME detect different token configs (ie no class token, or more)
+    new_seq_len = grid_size[0] * grid_size[1] + extra_tokens
+    if new_seq_len == old_pos_embed.shape[0]:
+        return
+
+    if extra_tokens:
+        pos_emb_tok, pos_emb_img = old_pos_embed[:extra_tokens], old_pos_embed[extra_tokens:]
+    else:
+        pos_emb_tok, pos_emb_img = None, old_pos_embed
+    old_grid_size = to_2tuple(int(math.sqrt(len(pos_emb_img))))
+
+    logging.info('Resizing position embedding grid-size from %s to %s', old_grid_size, grid_size)
+    pos_emb_img = pos_emb_img.reshape(1, old_grid_size[0], old_grid_size[1], -1).permute(0, 3, 1, 2)
+    pos_emb_img = F.interpolate(
+        pos_emb_img,
+        size=grid_size,
+        mode=interpolation,
+        align_corners=True,
+    )
+    pos_emb_img = pos_emb_img.permute(0, 2, 3, 1).reshape(1, grid_size[0] * grid_size[1], -1)[0]
+    if pos_emb_tok is not None:
+        new_pos_embed = torch.cat([pos_emb_tok, pos_emb_img], dim=0)
+    else:
+        new_pos_embed = pos_emb_img
+    state_dict['visual.positional_embedding'] = new_pos_embed

src/open_clip/model_configs/RN101-quickgelu.json

@@ -0,0 +1,22 @@
+{
+    "embed_dim": 512,
+    "quick_gelu": true,
+    "vision_cfg": {
+        "image_size": 224,
+        "layers": [
+            3,
+            4,
+            23,
+            3
+        ],
+        "width": 64,
+        "patch_size": null
+    },
+    "text_cfg": {
+        "context_length": 77,
+        "vocab_size": 49408,
+        "width": 512,
+        "heads": 8,
+        "layers": 12
+    }
+}

src/open_clip/model_configs/RN101.json

@@ -0,0 +1,21 @@
+{
+    "embed_dim": 512,
+    "vision_cfg": {
+        "image_size": 224,
+        "layers": [
+            3,
+            4,
+            23,
+            3
+        ],
+        "width": 64,
+        "patch_size": null
+    },
+    "text_cfg": {
+        "context_length": 77,
+        "vocab_size": 49408,
+        "width": 512,
+        "heads": 8,
+        "layers": 12
+    }
+}

src/open_clip/model_configs/RN50-quickgelu.json

@@ -0,0 +1,22 @@
+{
+    "embed_dim": 1024,
+    "quick_gelu": true,
+    "vision_cfg": {
+        "image_size": 224,
+        "layers": [
+            3,
+            4,
+            6,
+            3
+        ],
+        "width": 64,
+        "patch_size": null
+    },
+    "text_cfg": {
+        "context_length": 77,
+        "vocab_size": 49408,
+        "width": 512,
+        "heads": 8,
+        "layers": 12
+    }
+}

src/open_clip/model_configs/RN50.json

@@ -0,0 +1,21 @@
+{
+    "embed_dim": 1024,
+    "vision_cfg": {
+        "image_size": 224,
+        "layers": [
+            3,
+            4,
+            6,
+            3
+        ],
+        "width": 64,
+        "patch_size": null
+    },
+    "text_cfg": {
+        "context_length": 77,
+        "vocab_size": 49408,
+        "width": 512,
+        "heads": 8,
+        "layers": 12
+    }
+}