Spaces:
Running
Running
| # Appendix: ML OSS Evaluation Rubric | |
| The following checklist provides an evaluation rubric for open source code related to machine learning research. | |
| For any given code repository, tally a score based on these questions: | |
| - **Q1:** Does the repository use a business-friendly license? | |
| - **Q2:** Does the code install correctly with either `pip` or `conda` package managers? | |
| - **Q3:** Are the library dependencies reasonably current, not using pinned versions for popular libraries? | |
| - **Q4:** Has the project provided sample code which runs without exceptions? | |
| - **Q5:** Can the sample code reproduce the published results of the research? | |
| - **Q6:** Does the library provide affordances for data integration, i.e., it's not optimized for a particular benchmark? | |
| - **Q7:** Can the code be called programmatically as a library, i.e., not run primarily through a command line interface (CLI), and not requiring container/microservice orchestration? | |
| - **Q8:** Will the library and its dependencies pass a reasonable level of security audit without structural changes? | |
| - **Q9:** Does the code support concurrency and parallelization? | |
| - **Q10:** Has the repo been maintained within the past six months? | |
| ## Dependency Evaluations | |
| Based on this checklist, the dependencies integrated within this project scores as follows: | |
| rubric | `OpenNRE` | `pulp` | `qwikidata` | `REBEL` | `spaCy` | `Spotlight` | `SpanMarker` | `transformers` | |
| --- | --- | --- | --- | --- | --- | --- | --- | |
| Q1 | x | x | x | x | x | x | x | x | |
| Q2 | x | x | x | x | x | x | x | x | |
| Q3 | x | x | x | x | x | x | x | x | |
| Q4 | x | x | x | x | x | x | x | x | |
| Q5 | x | x | x | x | x | x | x | x | |
| Q6 | x | x | x | x | x | x | x | x | |
| Q7 | x | x | x | x | x | x | x | x | |
| Q8 | x | x | x | x | x | x | x | x | |
| Q9 | x | x | x | x | x | x | x | x | |
| Q10 | x | x | x | x | x | x | x | x | |
| [`OpenNRE`](https://github.com/thunlp/OpenNRE/) | |
| [`pulp`](https://github.com/coin-or/pulp) | |
| [`qwikidata`](https://github.com/kensho-technologies/qwikidata) | |
| [`REBEL`](https://github.com/Babelscape/rebel) | |
| [`spaCy`](https://spacy.io/) | |
| [`spaCy-DBpedia-Spotlight`](https://github.com/MartinoMensio/spacy-dbpedia-spotlight) | |
| [`SpanMarker`](https://github.com/tomaarsen/SpanMarkerNER/) | |
| [`transformers`](https://github.com/huggingface/transformers/) | |
| There were many other open source code projects which were evaluated | |
| but scored < 8 and were therefore considered unusable for our work. | |