Upload folder using huggingface_hub

README.md

@@ -183,7 +183,7 @@ model-index:
       name: Coverage (XY)
 ---
 
-# DWDSmor – German morphology
+# DWDSmor – German Morphology
 
 
 
@@ -205,14 +205,33 @@ The automata are compiled and traversed via
 library and toolbox for finite-state transducers (FSTs). Their
 coverage of the German language depends on
 
-1. the DWDSmor grammar, defining the rules by which word formation happens, and
-1. a lexicon, assigning inflection classes to lexical words.
+1. the DWDSmor grammar, defining the rules by which word formation
+   happens, and
+1. a lexicon, declaring inflection classes and other morphological
+   properties for covered lexical words.
 
-While the grammar, derived from
+The grammar, derived from
 [SMORLemma](https://github.com/rsennrich/SMORLemma) and providing the
 morphology for building automata from lexica, is common to all DWDSmor
-installations and published as open source, there are **2 lexicon
-editions**:
+installations and published as open source. In contrast we provide
+**multiple lexica** resulting in different editions of DWDSmor:
+
+1. the **DWDS Edition**, derived from the complete lexical dataset of
+   the [DWDS dictionary](https://www.dwds.de/) and available upon
+   request for research purposes,
+1. the **Open Edition**, based on a subset of the DWDS, covering the
+   most common word forms and released freely with the grammar for
+   general use and experiments.
+
+Depending on the edition and word class, coverage ranges from 70 to
+100% with the notable exceptions of foreign language words and named
+entities: Generally, both classes are not part of the underlying DWDS
+dictionary and thus barely covered by DWDSmor. Current overall
+coverage measured against the [German Universal Dependencies
+treebank](https://universaldependencies.org/treebanks/de_hdt/index.html)
+is documented on the respective [Hugging Face Hub
+page](https://huggingface.co/zentrum-lexikographie) of each edition.
+
 
 ## Usage
 
@@ -222,7 +241,7 @@ DWDSmor as a Python library is available via the package index PyPI:
 pip install dwdsmor
 ```
 
-For lemmatisation:
+The library can be used for lemmatisation:
 
 ``` python-console
 >>> import dwsdmor
@@ -231,13 +250,51 @@ For lemmatisation:
 >>> assert lemmatizer("getestet", pos={"+ADJ"}) == "getestet"
 ```
 
-…
+Next to the Python API, the package provides a simple command line
+interface named `dwdsmor`. To analyze a word form, pass it as an
+argument:
+
+```plaintext
+$ dwdsmor getestet
+| Wordform   | Lemma    | Analysis                            | POS   | Degree   | Function   | Nonfinite   | Tense   | Auxiliary   |
+|------------|----------|-------------------------------------|-------|----------|------------|-------------|---------|-------------|
+| getestet   | getestet | ge<~>test<~>et<+ADJ><Pos><Pred/Adv> | +ADJ  | Pos      | Pred/Adv   |             |         |             |
+| getestet   | testen   | test<~>en<+V><Part><Perf><haben>    | +V    |          |            | Part        | Perf    | haben       |
+```
+
+To generate all word forms for a lexical word, pass it (or a form
+which can be analyzed as the lexical word) as an argument together
+with the option `-g`:
+
+``` plaintext
+$ dwdsmor -g getestet
+[…]
+| Wordform   | Lemma    | Analysis                                                    | POS   | Subcategory   | Degree   | Function   |   Person | Gender   | Case   | Number   | Nonfinite   | Tense   | Mood   | Auxiliary   | Inflection   |
+|------------|----------|-------------------------------------------------------------|-------|---------------|----------|------------|----------|----------|--------|----------|-------------|---------|--------|-------------|--------------|
+| getestete  | getestet | ge<~>test<~>et<+ADJ><Pos><Attr/Subst><Fem><Acc><Sg><St>     | +ADJ  |               | Pos      | Attr/Subst |          | Fem      | Acc    | Sg       |             |         |        |             | St           |
+| getestete  | getestet | ge<~>test<~>et<+ADJ><Pos><Attr/Subst><Fem><Acc><Sg><Wk>     | +ADJ  |               | Pos      | Attr/Subst |          | Fem      | Acc    | Sg       |             |         |        |             | Wk           |
+| getesteter | getestet | ge<~>test<~>et<+ADJ><Pos><Attr/Subst><Fem><Dat><Sg><St>     | +ADJ  |               | Pos      | Attr/Subst |          | Fem      | Dat    | Sg       |             |         |        |             | St           |
+| getesteten | getestet | ge<~>test<~>et<+ADJ><Pos><Attr/Subst><Fem><Dat><Sg><Wk>     | +ADJ  |               | Pos      | Attr/Subst |          | Fem      | Dat    | Sg       |             |         |        |             | Wk           |
+| getesteter | getestet | ge<~>test<~>et<+ADJ><Pos><Attr/Subst><Fem><Gen><Sg><St>     | +ADJ  |               | Pos      | Attr/Subst |          | Fem      | Gen    | Sg       |             |         |        |             | St           |
+| getesteten | getestet | ge<~>test<~>et<+ADJ><Pos><Attr/Subst><Fem><Gen><Sg><Wk>     | +ADJ  |               | Pos      | Attr/Subst |          | Fem      | Gen    | Sg       |             |         |        |             | Wk           |
+[…]
+| testeten   | testen   | test<~>en<+V><1><Pl><Past><Ind>                             | +V    |               |          |            |        1 |          |        | Pl       |             | Past    | Ind    |             |              |
+| testeten   | testen   | test<~>en<+V><1><Pl><Past><Subj>                            | +V    |               |          |            |        1 |          |        | Pl       |             | Past    | Subj   |             |              |
+| testen     | testen   | test<~>en<+V><1><Pl><Pres><Ind>                             | +V    |               |          |            |        1 |          |        | Pl       |             | Pres    | Ind    |             |              |
+| testen     | testen   | test<~>en<+V><1><Pl><Pres><Subj>                            | +V    |               |          |            |        1 |          |        | Pl       |             | Pres    | Subj   |             |              |
+| testete    | testen   | test<~>en<+V><1><Sg><Past><Ind>                             | +V    |               |          |            |        1 |          |        | Sg       |             | Past    | Ind    |             |              |
+| testete    | testen   | test<~>en<+V><1><Sg><Past><Subj>                            | +V    |               |          |            |        1 |          |        | Sg       |             | Past    | Subj   |             |              |
+| teste      | testen   | test<~>en<+V><1><Sg><Pres><Ind>                             | +V    |               |          |            |        1 |          |        | Sg       |             | Pres    | Ind    |             |              |
+| teste      | testen   | test<~>en<+V><1><Sg><Pres><Subj>                            | +V    |               |          |            |        1 |          |        | Sg       |             | Pres    | Subj   |             |              |
+| testetet   | testen   | test<~>en<+V><2><Pl><Past><Ind>                             | +V    |               |          |            |        2 |          |        | Pl       |             | Past    | Ind    |             |              |
+[…]
+```
 
 ## Development
 
-DWDSmor is in active development. In its current stage, DWDSmor
-supports most inflection classes and some productive word-formation
-patterns of written German.
+DWDSmor is in active development. In its current stage, it supports
+most inflection classes and some productive word-formation patterns of
+written German.
 
 
 ### Prerequisites
@@ -266,7 +323,7 @@ patterns of written German.
 On a Debian-based distribution, the following command install the
 required software:
 
-```sh
+```plaintext
 apt-get install python3 default-jdk libsaxonhe-java sfst
 ```
 
@@ -275,66 +332,69 @@ apt-get install python3 default-jdk libsaxonhe-java sfst
 Optionally, set up a Python virtual environment for project builds,
 i. e. via Python's `venv`:
 
-```sh
+```plaintext
 python3 -m venv .venv
 source .venv/bin/activate
 ```
 
 Then install DWDSmor, including development dependencies:
 
-```sh
+```plaintext
 pip install -U pip setuptools && pip install -e '.[dev]'
 ```
 
 
 ### Building lexica and automata
 
-Building different editions is facilitated via a build script:
+Building different editions is facilitated via the script `build-dwdsmor`:
 
 
+```plaintext
+$ ./build-dwdsmor --help
+usage: cli.py [-h] [--automaton AUTOMATON] [--force] [--with-metrics] [--release] [--tag]
+              [editions ...]
 
-```sh
-make all
-```
+Build DWDSmor.
 
-Alternatively, you can run:
+positional arguments:
+  editions              Editions to build (all by default)
 
-```sh
-make dwds && make dwds-install && make dwdsmor
+options:
+  -h, --help            show this help message and exit
+  --automaton AUTOMATON
+                        Automaton type to build (all by default)
+  --force               Force building (also current targets)
+  --with-metrics        Measure UD/de-hdt coverage
+  --release             Push automata to HF hub
+  --tag                 Tag HF hub release with current version
 ```
 
-Note that these commands require DWDS sources in `lexicon/dwds/wb/`, which are
-not part of this repository.
+To build all editions available in the current git checkout, run:
 
-Alternatively, you can build sample DWDSmor lexica and transducers from the
-sample lexicon in `lexicon/sample/wb/` by running:
-
-```sh
-make sample && make sample-install && make dwdsmor
+```plaintext
+./build-dwdsmor
 ```
 
-After building DWDSmor transducers, install them into `lib/`, where the
-Python scripts `dwdsmor` and `dwdsmor-paradigm` expect them by default:
-
-```sh
-make install
-```
+The build result can be found in `build/` with one subdirectory per
+edition. Each edition contains several automata types in standard and
+compact format:
 
-The installed DWDSmor transducers are:
 
-* `lib/dwdsmor.{a,ca}`: transducer with inflection and word-formation
-  components, for lemmatisation and morphological analysis of word forms in
-  terms of grammatical categories
-* `lib/dwdsmor-morph.{a,ca}`: transducer with inflection and word-formation
-  components, for the generation of morphologically segmented word forms
-* `lib/dwdsmor-finite.{a,ca}`: transducer with an inflection component and a
+* `lemma.{a,ca}`: transducer with inflection and word-formation
+  components, for lemmatisation and morphological analysis of word
+  forms in terms of grammatical categories
+* `morph.{a,ca}`: transducer with inflection and word-formation
+  components, for the generation of morphologically segmented word
+  forms
+* `finite.{a,ca}`: transducer with an inflection component and a
   finite word-formation component, for testing purposes
-* `lib/dwdsmor-root.{a,ca}`: transducer with inflection and word-formation
-  components, for lexical analysis of word forms in terms of root lemmas (i.e.,
-  lemmas of ultimate word-formation bases), word-formation process,
-  word-formation means, and grammatical categories in term of the
-  Pattern-and-Restriction Theory of word formation (Nolda 2022)
-* `lib/dwdsmor-index.{a,ca}`: transducer with an inflection component only with
+* `root.{a,ca}`: transducer with inflection and word-formation
+  components, for lexical analysis of word forms in terms of root
+  lemmas (i.e., lemmas of ultimate word-formation bases),
+  word-formation process, word-formation means, and grammatical
+  categories in term of the Pattern-and-Restriction Theory of word
+  formation (Nolda 2022)
+* `index.{a,ca}`: transducer with an inflection component only with
   DWDS homographic lemma indices, for paradigm generation
 
 
@@ -344,19 +404,20 @@ In order to test basic transducer usage and for potential regressions, run
 
     pytest
 
-## Contact
+## License
 
-Feel free to contact [Andreas Nolda](mailto:[email protected]) for
-questions regarding the lexicon or the grammar and
-[Gregor Middell](mailto:gregor.middell@bbaw.de) for question related
-to the integration of DWDSmor into your corpus-annotation pipeline.
+As the original SMOR and SMORLemma grammars, the DWDSmor grammar and
+Python library are licensed under the GNU General Public License
+v2.0. The same applies to the open edition of the DWDSmor lexicon.
 
+For the DWDS edition based on the complete DWDS dictionary, all rights
+are reserved and individual license terms apply. If you are interested
+in the DWDS edition, please contact us.
 
-## License
+## Contact
 
-As the original SMOR and SMORLemma grammars, the DWDSmor grammar is
-licensed under the GNU General Public Licence v2.0. The same applies
-to the rest of this project.
+Feel free to contact [Andreas Nolda](mailto:[email protected]) for any
+question about this project.
 
 ## Credits
 
@@ -377,8 +438,7 @@ DWSDmor is based on the following software and datasets:
 
 * Berlin-Brandenburg Academy of Sciences and Humanities (BBAW) (ed.) (n.d.).
   DWDS – Digitales Wörterbuch der deutschen Sprache: Das Wortauskunftssystem zur
-  deutschen Sprache in Geschichte und Gegenwart.
-  https://www.dwds.de
+  deutschen Sprache in Geschichte und Gegenwart. [Online](https://www.dwds.de/)
 * Fitschen, Arne (2004). Ein computerlinguistisches Lexikon als komplexes
   System. Ph.D. thesis, Universität Stuttgart.
   [PDF](http://www.ims.uni-stuttgart.de/forschung/ressourcen/lexika/IMSLex/fitschendiss.pdf)