Upload folder using huggingface_hub

.dvc/.gitignore

@@ -0,0 +1,3 @@
+/config.local
+/tmp
+/cache

.dvc/config

@@ -0,0 +1,6 @@
+[cache]
+    dir = /data/dvc-cache/FintoAI-data-YKL
+    shared = group
+    type = symlink
+[core]
+    autostage = true

.dvcignore

@@ -0,0 +1,3 @@
+# Add patterns of files dvc should ignore, which could improve
+# the performance. Learn more at
+# https://dvc.org/doc/user-guide/dvcignore

.gitignore

@@ -0,0 +1,2 @@
+venv
+venv-installed

LICENSE

@@ -0,0 +1,121 @@
+Creative Commons Legal Code
+
+CC0 1.0 Universal
+
+    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
+    LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
+    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
+    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
+    REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
+    PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
+    THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
+    HEREUNDER.
+
+Statement of Purpose
+
+The laws of most jurisdictions throughout the world automatically confer
+exclusive Copyright and Related Rights (defined below) upon the creator
+and subsequent owner(s) (each and all, an "owner") of an original work of
+authorship and/or a database (each, a "Work").
+
+Certain owners wish to permanently relinquish those rights to a Work for
+the purpose of contributing to a commons of creative, cultural and
+scientific works ("Commons") that the public can reliably and without fear
+of later claims of infringement build upon, modify, incorporate in other
+works, reuse and redistribute as freely as possible in any form whatsoever
+and for any purposes, including without limitation commercial purposes.
+These owners may contribute to the Commons to promote the ideal of a free
+culture and the further production of creative, cultural and scientific
+works, or to gain reputation or greater distribution for their Work in
+part through the use and efforts of others.
+
+For these and/or other purposes and motivations, and without any
+expectation of additional consideration or compensation, the person
+associating CC0 with a Work (the "Affirmer"), to the extent that he or she
+is an owner of Copyright and Related Rights in the Work, voluntarily
+elects to apply CC0 to the Work and publicly distribute the Work under its
+terms, with knowledge of his or her Copyright and Related Rights in the
+Work and the meaning and intended legal effect of CC0 on those rights.
+
+1. Copyright and Related Rights. A Work made available under CC0 may be
+protected by copyright and related or neighboring rights ("Copyright and
+Related Rights"). Copyright and Related Rights include, but are not
+limited to, the following:
+
+  i. the right to reproduce, adapt, distribute, perform, display,
+     communicate, and translate a Work;
+ ii. moral rights retained by the original author(s) and/or performer(s);
+iii. publicity and privacy rights pertaining to a person's image or
+     likeness depicted in a Work;
+ iv. rights protecting against unfair competition in regards to a Work,
+     subject to the limitations in paragraph 4(a), below;
+  v. rights protecting the extraction, dissemination, use and reuse of data
+     in a Work;
+ vi. database rights (such as those arising under Directive 96/9/EC of the
+     European Parliament and of the Council of 11 March 1996 on the legal
+     protection of databases, and under any national implementation
+     thereof, including any amended or successor version of such
+     directive); and
+vii. other similar, equivalent or corresponding rights throughout the
+     world based on applicable law or treaty, and any national
+     implementations thereof.
+
+2. Waiver. To the greatest extent permitted by, but not in contravention
+of, applicable law, Affirmer hereby overtly, fully, permanently,
+irrevocably and unconditionally waives, abandons, and surrenders all of
+Affirmer's Copyright and Related Rights and associated claims and causes
+of action, whether now known or unknown (including existing as well as
+future claims and causes of action), in the Work (i) in all territories
+worldwide, (ii) for the maximum duration provided by applicable law or
+treaty (including future time extensions), (iii) in any current or future
+medium and for any number of copies, and (iv) for any purpose whatsoever,
+including without limitation commercial, advertising or promotional
+purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+member of the public at large and to the detriment of Affirmer's heirs and
+successors, fully intending that such Waiver shall not be subject to
+revocation, rescission, cancellation, termination, or any other legal or
+equitable action to disrupt the quiet enjoyment of the Work by the public
+as contemplated by Affirmer's express Statement of Purpose.
+
+3. Public License Fallback. Should any part of the Waiver for any reason
+be judged legally invalid or ineffective under applicable law, then the
+Waiver shall be preserved to the maximum extent permitted taking into
+account Affirmer's express Statement of Purpose. In addition, to the
+extent the Waiver is so judged Affirmer hereby grants to each affected
+person a royalty-free, non transferable, non sublicensable, non exclusive,
+irrevocable and unconditional license to exercise Affirmer's Copyright and
+Related Rights in the Work (i) in all territories worldwide, (ii) for the
+maximum duration provided by applicable law or treaty (including future
+time extensions), (iii) in any current or future medium and for any number
+of copies, and (iv) for any purpose whatsoever, including without
+limitation commercial, advertising or promotional purposes (the
+"License"). The License shall be deemed effective as of the date CC0 was
+applied by Affirmer to the Work. Should any part of the License for any
+reason be judged legally invalid or ineffective under applicable law, such
+partial invalidity or ineffectiveness shall not invalidate the remainder
+of the License, and in such case Affirmer hereby affirms that he or she
+will not (i) exercise any of his or her remaining Copyright and Related
+Rights in the Work or (ii) assert any associated claims and causes of
+action with respect to the Work, in either case contrary to Affirmer's
+express Statement of Purpose.
+
+4. Limitations and Disclaimers.
+
+ a. No trademark or patent rights held by Affirmer are waived, abandoned,
+    surrendered, licensed or otherwise affected by this document.
+ b. Affirmer offers the Work as-is and makes no representations or
+    warranties of any kind concerning the Work, express, implied,
+    statutory or otherwise, including without limitation warranties of
+    title, merchantability, fitness for a particular purpose, non
+    infringement, or the absence of latent or other defects, accuracy, or
+    the present or absence of errors, whether or not discoverable, all to
+    the greatest extent permissible under applicable law.
+ c. Affirmer disclaims responsibility for clearing rights of other persons
+    that may apply to the Work or any use thereof, including without
+    limitation any person's Copyright and Related Rights in the Work.
+    Further, Affirmer disclaims responsibility for obtaining any necessary
+    consents, permissions or other rights required for any use of the
+    Work.
+ d. Affirmer understands and acknowledges that Creative Commons is not a
+    party to this document and has no duty or obligation with respect to
+    this CC0 or use of the Work.

README.md

@@ -0,0 +1,17 @@
+# FintoAI-data-YKL
+Configurations for maintaining the Annif projects with YKL vocabulary used at [Finto AI service](ai.finto.fi/).
+
+The projects are trained and evaluated using a [DVC (Data Version Control) pipeline](https://dvc.org/doc/start/data-management/data-pipelines) defined in [dvc.yaml](/dvc.yaml).
+
+The pipeline takes care of 
+
+1. installing Annif in a venv,
+2. loading the vocabulary,
+3. training the projects,
+4. evaluating the projects.
+
+When the necessary vocabulary and training corpora are in place the pipeline can be run using the command
+
+    dvc repro
+    
+For more information about using DVC with Annif projects see the [DVC exercise of Annif tutorial](https://github.com/NatLibFi/Annif-tutorial/blob/master/exercises/OPT_dvc.md).

corpora/.gitignore

@@ -0,0 +1,2 @@
+/ykl-classes.tsv
+/ykl-skos.ttl

corpora/kirjaesittelyt2021/.gitignore

@@ -0,0 +1,9 @@
+/kvesit-ykl-fin-train.tsv
+/kvesit-ykl-fin-validate.tsv
+/kvesit-ykl-fin-test.tsv
+/kvesit-ykl-eng-train.tsv
+/kvesit-ykl-eng-validate.tsv
+/kvesit-ykl-eng-test.tsv
+/kvesit-ykl-swe-train.tsv
+/kvesit-ykl-swe-validate.tsv
+/kvesit-ykl-swe-test.tsv

corpora/kirjaesittelyt2021/kvesit-ykl-eng-test.tsv.dvc

@@ -0,0 +1,10 @@
+md5: d0048a06955ce66d56224834e8f6ef9c
+deps:
+- md5: bc7f924d84a20e9ab91e2600fd2415ec
+  size: 210866
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/kvesit-ykl-eng-test.tsv
+  hash: md5
+outs:
+- md5: bc7f924d84a20e9ab91e2600fd2415ec
+  size: 210866
+  path: kvesit-ykl-eng-test.tsv

corpora/kirjaesittelyt2021/kvesit-ykl-eng-train.tsv.dvc

@@ -0,0 +1,10 @@
+md5: fbe7fe15b03297c612dd0e436c9f63af
+deps:
+- md5: 87548d6f8f9a8185dc870e8112668c87
+  size: 1730632
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/kvesit-ykl-eng-train.tsv
+  hash: md5
+outs:
+- md5: 87548d6f8f9a8185dc870e8112668c87
+  size: 1730632
+  path: kvesit-ykl-eng-train.tsv

corpora/kirjaesittelyt2021/kvesit-ykl-eng-validate.tsv.dvc

@@ -0,0 +1,9 @@
+md5: da25cc54ab5e66d5e2ec5c21edc76796
+deps:
+- md5: c1706bb4102fa4799492fd653acc97a2
+  size: 218773
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/kvesit-ykl-eng-validate.tsv
+outs:
+- md5: c1706bb4102fa4799492fd653acc97a2
+  size: 218773
+  path: kvesit-ykl-eng-validate.tsv

corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv.dvc

@@ -0,0 +1,10 @@
+md5: e41640988a93564680ee91b589667028
+deps:
+- md5: 9f8d094ae71ec68c5d40dde46d991e1f
+  size: 4534782
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/kvesit-ykl-fin-test.tsv
+  hash: md5
+outs:
+- md5: 9f8d094ae71ec68c5d40dde46d991e1f
+  size: 4534782
+  path: kvesit-ykl-fin-test.tsv

corpora/kirjaesittelyt2021/kvesit-ykl-fin-train.tsv.dvc

@@ -0,0 +1,10 @@
+md5: d86b6c73a936179a8f4b069e5ac90322
+deps:
+- md5: cc52c6a5c21d799aad3fb73a889cd90e
+  size: 36684502
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/kvesit-ykl-fin-train.tsv
+  hash: md5
+outs:
+- md5: cc52c6a5c21d799aad3fb73a889cd90e
+  size: 36684502
+  path: kvesit-ykl-fin-train.tsv

corpora/kirjaesittelyt2021/kvesit-ykl-fin-validate.tsv.dvc

@@ -0,0 +1,9 @@
+md5: c11ba98458f2295383418b1fd8142813
+deps:
+- md5: 88d1c1d133953681474207186f0408e1
+  size: 4575927
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/kvesit-ykl-fin-validate.tsv
+outs:
+- md5: 88d1c1d133953681474207186f0408e1
+  size: 4575927
+  path: kvesit-ykl-fin-validate.tsv

corpora/kirjaesittelyt2021/kvesit-ykl-swe-test.tsv.dvc

@@ -0,0 +1,10 @@
+md5: 72edc7073e64d873fe62c73201c13965
+deps:
+- md5: a9177ace2f0d1dffa6a424fccdf04b37
+  size: 287821
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/kvesit-ykl-swe-test.tsv
+  hash: md5
+outs:
+- md5: a9177ace2f0d1dffa6a424fccdf04b37
+  size: 287821
+  path: kvesit-ykl-swe-test.tsv

corpora/kirjaesittelyt2021/kvesit-ykl-swe-train.tsv.dvc

@@ -0,0 +1,10 @@
+md5: b1f96f9bf1677ae056540572f3968311
+deps:
+- md5: fa74f039e20594e212d2429eb3e74c13
+  size: 2367569
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/kvesit-ykl-swe-train.tsv
+  hash: md5
+outs:
+- md5: fa74f039e20594e212d2429eb3e74c13
+  size: 2367569
+  path: kvesit-ykl-swe-train.tsv

corpora/kirjaesittelyt2021/kvesit-ykl-swe-validate.tsv.dvc

@@ -0,0 +1,9 @@
+md5: 896803dc4e4492ea12625b363ef21007
+deps:
+- md5: 6ae75a57ef7c1107f55978d759796c94
+  size: 294875
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/kvesit-ykl-swe-validate.tsv
+outs:
+- md5: 6ae75a57ef7c1107f55978d759796c94
+  size: 294875
+  path: kvesit-ykl-swe-validate.tsv

corpora/ykl-classes.tsv.dvc

@@ -0,0 +1,9 @@
+md5: c0b12e0a05399ad397f12e30d4bef364
+deps:
+- md5: b2f29070ccedefda72994b8db324e371
+  size: 187637
+  path: /data/Annif-corpora-restricted/kirjaesittelyt2021/ykl/ykl-classes.tsv
+outs:
+- md5: b2f29070ccedefda72994b8db324e371
+  size: 187637
+  path: ykl-classes.tsv

corpora/ykl-skos.ttl.dvc

@@ -0,0 +1,10 @@
+md5: 7f7e7284f20b95f4d8717a955778e726
+deps:
+- md5: fea11c863134c9e8379ea52e0f31e28a
+  size: 3901929
+  path: /data/Annif-corpora/vocab/ykl-skos.ttl
+  hash: md5
+outs:
+- md5: fea11c863134c9e8379ea52e0f31e28a
+  size: 3901929
+  path: ykl-skos.ttl

data/projects/.gitignore

@@ -0,0 +1,6 @@
+/ykl-omikuji-parabel-fi
+/ykl-omikuji-bonsai-fi
+/ykl-omikuji-bonsai-sv
+/ykl-omikuji-bonsai-en
+/ykl-omikuji-parabel-sv
+/ykl-omikuji-parabel-en

data/vocabs/.gitignore

@@ -0,0 +1,2 @@
+/ykl-fi
+/ykl

dvc.lock

@@ -0,0 +1,575 @@
+schema: '2.0'
+stages:
+  loadvoc-fi:
+    cmd: annif loadvoc ykl-omikuji-parabel-fi corpora/ykl-classes.tsv
+    deps:
+    - path: corpora/ykl-classes.tsv
+      md5: b2f29070ccedefda72994b8db324e371
+      size: 187637
+    - path: venv-installed
+      md5: 2e6baa8289dac9e06cb525999eb39a70
+      size: 42
+    outs:
+    - path: data/vocabs/ykl-fi
+      md5: 680732b8de9b5108536ccea40462734e.dir
+      size: 764449
+      nfiles: 3
+  train-omikuji-parabel-fi:
+    cmd: annif train ykl-omikuji-parabel-fi -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-fin-train.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-fin-train.tsv
+      md5: ffa2c09b76a8a5370dbe71540da8c0ff
+      size: 38164190
+    - path: data/vocabs/ykl
+      md5: 3b9896e0ea6b1a4fa9820f015831fd93.dir
+      size: 6503819
+      nfiles: 3
+    - path: venv-installed
+      md5: 2e6baa8289dac9e06cb525999eb39a70
+      size: 42
+    params:
+      projects.toml:
+        ykl-omikuji-parabel-fi:
+          name: YKL Omikuji Parabel Finnish
+          language: fi
+          backend: omikuji
+          analyzer: voikko(fi)
+          vocab: ykl
+          ngram: 1
+          min_df: 1
+    outs:
+    - path: data/projects/ykl-omikuji-parabel-fi
+      md5: e506a5b142c024d089a449b9ab99da7c.dir
+      size: 119198246
+      nfiles: 6
+  eval-omikuji-parabel-fi:
+    cmd: annif eval ykl-omikuji-parabel-fi -j 8 -m Precision@1 -m NDCG --metrics-file
+      reports/test-omikuji-parabel-fi.json corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+      md5: e4497d02e1c12d30de314f25ac5c5a8e
+      size: 3289456
+    - path: data/projects/ykl-omikuji-parabel-fi
+      md5: e506a5b142c024d089a449b9ab99da7c.dir
+      size: 119198246
+      nfiles: 6
+    - path: venv-installed
+      md5: 2e6baa8289dac9e06cb525999eb39a70
+      size: 42
+    outs:
+    - path: reports/test-omikuji-parabel-fi.json
+      md5: e0079ca11936a9627c9ee5b4a23e4c48
+      size: 100
+  install:
+    cmd: python3 -m venv venv && . venv/bin/activate && pip install -U pip wheel setuptools
+      && pip install -r requirements.txt && cp requirements.txt venv-installed
+    deps:
+    - path: requirements.txt
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+  train-omikuji-bonsai-fi:
+    cmd: annif train ykl-omikuji-bonsai-fi -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-fin-train.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-fin-train.tsv
+      md5: ffa2c09b76a8a5370dbe71540da8c0ff
+      size: 38164190
+    - path: data/vocabs/ykl
+      md5: 3b9896e0ea6b1a4fa9820f015831fd93.dir
+      size: 6503819
+      nfiles: 3
+    - path: venv-installed
+      md5: 2e6baa8289dac9e06cb525999eb39a70
+      size: 42
+    params:
+      projects.toml:
+        ykl-omikuji-bonsai-fi:
+          name: YKL Omikuji Bonsai Finnish
+          language: fi
+          backend: omikuji
+          cluster_balanced: 'False'
+          cluster_k: 100
+          max_depth: 3
+          analyzer: voikko(fi)
+          vocab: ykl
+          ngram: 1
+          min_df: 1
+    outs:
+    - path: data/projects/ykl-omikuji-bonsai-fi
+      md5: 1166661a5b68a9c155a57d40592d11eb.dir
+      size: 114643724
+      nfiles: 6
+  eval-omikuji-bonsai-fi:
+    cmd: annif eval ykl-omikuji-bonsai-fi -j 8 -m Precision@1 -m NDCG --metrics-file
+      reports/test-omikuji-bonsai-fi.json corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+      md5: e4497d02e1c12d30de314f25ac5c5a8e
+      size: 3289456
+    - path: data/projects/ykl-omikuji-bonsai-fi
+      md5: 1166661a5b68a9c155a57d40592d11eb.dir
+      size: 114643724
+      nfiles: 6
+    - path: venv-installed
+      md5: 2e6baa8289dac9e06cb525999eb39a70
+      size: 42
+    outs:
+    - path: reports/test-omikuji-bonsai-fi.json
+      md5: da9fafaa27f186fff29cb24b505c2fa2
+      size: 100
+  load-vocab:
+    cmd: annif load-vocab ykl corpora/ykl-skos.ttl
+    deps:
+    - path: corpora/ykl-skos.ttl
+      md5: fea11c863134c9e8379ea52e0f31e28a
+      size: 3901929
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: data/vocabs/ykl
+      hash: md5
+      md5: 083422d4d504723b2eb4fc4ee8805a99.dir
+      size: 6503179
+      nfiles: 3
+  train-omikuji-bonsai@0:
+    cmd: annif train ykl-omikuji-bonsai-fi -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-fin-train.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-fin-train.tsv
+      md5: cc52c6a5c21d799aad3fb73a889cd90e
+      size: 36684502
+    - path: data/vocabs/ykl
+      hash: md5
+      md5: 083422d4d504723b2eb4fc4ee8805a99.dir
+      size: 6503179
+      nfiles: 3
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    params:
+      projects.toml:
+        ykl-omikuji-bonsai-fi:
+          name: YKL Omikuji Bonsai Finnish
+          language: fi
+          backend: omikuji
+          cluster_balanced: 'False'
+          cluster_k: 100
+          max_depth: 3
+          analyzer: voikko(fi)
+          vocab: ykl
+          ngram: 1
+          min_df: 1
+          access: hidden
+    outs:
+    - path: data/projects/ykl-omikuji-bonsai-fi
+      hash: md5
+      md5: 9bdaf3071b9fe0d13f72e1b2d6873bfb.dir
+      size: 110266624
+      nfiles: 6
+  eval-omikuji-bonsai@0:
+    cmd: annif eval ykl-omikuji-bonsai-fi -j 8 -m Precision@1 -m NDCG --metrics-file
+      reports/test-omikuji-bonsai-fi.json --results-file reports/test-omikuji-bonsai-fi.csv
+      corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+      md5: 9f8d094ae71ec68c5d40dde46d991e1f
+      size: 4534782
+    - path: data/projects/ykl-omikuji-bonsai-fi
+      hash: md5
+      md5: 9bdaf3071b9fe0d13f72e1b2d6873bfb.dir
+      size: 110266624
+      nfiles: 6
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: reports/test-omikuji-bonsai-fi.csv
+      hash: md5
+      md5: 3c536124365595cea86fa741f9c355cb
+      size: 237892
+    - path: reports/test-omikuji-bonsai-fi.json
+      hash: md5
+      md5: 24f9c18652995a673cb765edf417b2c8
+      size: 100
+  train-omikuji-bonsai@1:
+    cmd: annif train ykl-omikuji-bonsai-sv -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-swe-train.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-swe-train.tsv
+      md5: fa74f039e20594e212d2429eb3e74c13
+      size: 2367569
+    - path: data/vocabs/ykl
+      hash: md5
+      md5: 083422d4d504723b2eb4fc4ee8805a99.dir
+      size: 6503179
+      nfiles: 3
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    params:
+      projects.toml:
+        ykl-omikuji-bonsai-sv:
+          name: YKL Omikuji Bonsai Swedish
+          language: sv
+          backend: omikuji
+          cluster_balanced: 'False'
+          cluster_k: 100
+          max_depth: 3
+          analyzer: snowball(swedish)
+          vocab: ykl
+          ngram: 1
+          min_df: 1
+          access: hidden
+    outs:
+    - path: data/projects/ykl-omikuji-bonsai-sv
+      hash: md5
+      md5: e589e2a60b5f7cdcc5e67c7443b4c87a.dir
+      size: 8507130
+      nfiles: 6
+  train-omikuji-bonsai@2:
+    cmd: annif train ykl-omikuji-bonsai-en -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-eng-train.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-eng-train.tsv
+      md5: 87548d6f8f9a8185dc870e8112668c87
+      size: 1730632
+    - path: data/vocabs/ykl
+      hash: md5
+      md5: 083422d4d504723b2eb4fc4ee8805a99.dir
+      size: 6503179
+      nfiles: 3
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    params:
+      projects.toml:
+        ykl-omikuji-bonsai-en:
+          name: YKL Omikuji Bonsai English
+          language: en
+          backend: omikuji
+          cluster_balanced: 'False'
+          cluster_k: 100
+          max_depth: 3
+          analyzer: snowball(english)
+          vocab: ykl
+          ngram: 1
+          min_df: 1
+          access: hidden
+    outs:
+    - path: data/projects/ykl-omikuji-bonsai-en
+      hash: md5
+      md5: 0bce7445a6e1df060f800ec5516959c1.dir
+      size: 5332560
+      nfiles: 6
+  eval-omikuji-bonsai@2:
+    cmd: annif eval ykl-omikuji-bonsai-en -j 8 -m Precision@1 -m NDCG --metrics-file
+      reports/test-omikuji-bonsai-en.json --results-file reports/test-omikuji-bonsai-en.csv
+      corpora/kirjaesittelyt2021/kvesit-ykl-eng-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-eng-test.tsv
+      md5: bc7f924d84a20e9ab91e2600fd2415ec
+      size: 210866
+    - path: data/projects/ykl-omikuji-bonsai-en
+      hash: md5
+      md5: 0bce7445a6e1df060f800ec5516959c1.dir
+      size: 5332560
+      nfiles: 6
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: reports/test-omikuji-bonsai-en.csv
+      hash: md5
+      md5: e4734c53c0225ad899a06926e8b7e0d1
+      size: 212494
+    - path: reports/test-omikuji-bonsai-en.json
+      hash: md5
+      md5: 6f0029dd304ac0fc94b7817aa9ea3462
+      size: 100
+  train-omikuji-parabel@0:
+    cmd: annif train ykl-omikuji-parabel-fi -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-fin-train.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-fin-train.tsv
+      md5: cc52c6a5c21d799aad3fb73a889cd90e
+      size: 36684502
+    - path: data/vocabs/ykl
+      hash: md5
+      md5: 083422d4d504723b2eb4fc4ee8805a99.dir
+      size: 6503179
+      nfiles: 3
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    params:
+      projects.toml:
+        ykl-omikuji-parabel-fi:
+          name: YKL Omikuji Parabel Finnish
+          language: fi
+          backend: omikuji
+          analyzer: voikko(fi)
+          vocab: ykl
+          ngram: 1
+          min_df: 1
+          access: hidden
+    outs:
+    - path: data/projects/ykl-omikuji-parabel-fi
+      hash: md5
+      md5: 44ab11d0b3e729a0dcd3b88fdadb7567.dir
+      size: 114032199
+      nfiles: 6
+  eval-omikuji-parabel@0:
+    cmd: annif eval ykl-omikuji-parabel-fi -j 8 -m Precision@1 -m NDCG --metrics-file
+      reports/test-omikuji-parabel-fi.json --results-file reports/test-omikuji-parabel-fi.csv
+      corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+      md5: 9f8d094ae71ec68c5d40dde46d991e1f
+      size: 4534782
+    - path: data/projects/ykl-omikuji-parabel-fi
+      hash: md5
+      md5: 44ab11d0b3e729a0dcd3b88fdadb7567.dir
+      size: 114032199
+      nfiles: 6
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: reports/test-omikuji-parabel-fi.csv
+      hash: md5
+      md5: 6ef001dff735a3b3a64082c02721736c
+      size: 236720
+    - path: reports/test-omikuji-parabel-fi.json
+      hash: md5
+      md5: 1a7877d85e63ee67dc0c77f8b5c25ba2
+      size: 97
+  train-omikuji-parabel@1:
+    cmd: annif train ykl-omikuji-parabel-sv -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-swe-train.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-swe-train.tsv
+      md5: fa74f039e20594e212d2429eb3e74c13
+      size: 2367569
+    - path: data/vocabs/ykl
+      hash: md5
+      md5: 083422d4d504723b2eb4fc4ee8805a99.dir
+      size: 6503179
+      nfiles: 3
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    params:
+      projects.toml:
+        ykl-omikuji-parabel-sv:
+          name: YKL Omikuji Parabel Swedish
+          language: sv
+          backend: omikuji
+          analyzer: snowball(swedish)
+          vocab: ykl
+          ngram: 1
+          min_df: 1
+          access: hidden
+    outs:
+    - path: data/projects/ykl-omikuji-parabel-sv
+      hash: md5
+      md5: ce70c5681d39ac52e3b2a9fc2efa7f4a.dir
+      size: 8635331
+      nfiles: 6
+  eval-omikuji-parabel@1:
+    cmd: annif eval ykl-omikuji-parabel-sv -j 8 -m Precision@1 -m NDCG --metrics-file
+      reports/test-omikuji-parabel-sv.json --results-file reports/test-omikuji-parabel-sv.csv
+      corpora/kirjaesittelyt2021/kvesit-ykl-swe-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-swe-test.tsv
+      md5: a9177ace2f0d1dffa6a424fccdf04b37
+      size: 287821
+    - path: data/projects/ykl-omikuji-parabel-sv
+      hash: md5
+      md5: ce70c5681d39ac52e3b2a9fc2efa7f4a.dir
+      size: 8635331
+      nfiles: 6
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: reports/test-omikuji-parabel-sv.csv
+      hash: md5
+      md5: a9b65c5400c421b4ce9f1f7e44dffccd
+      size: 214164
+    - path: reports/test-omikuji-parabel-sv.json
+      hash: md5
+      md5: 7f68b59b4db8a3a6116e8ec105913678
+      size: 87
+  train-omikuji-parabel@2:
+    cmd: annif train ykl-omikuji-parabel-en -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-eng-train.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-eng-train.tsv
+      md5: 87548d6f8f9a8185dc870e8112668c87
+      size: 1730632
+    - path: data/vocabs/ykl
+      hash: md5
+      md5: 083422d4d504723b2eb4fc4ee8805a99.dir
+      size: 6503179
+      nfiles: 3
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    params:
+      projects.toml:
+        ykl-omikuji-parabel-en:
+          name: YKL Omikuji Parabel English
+          language: en
+          backend: omikuji
+          analyzer: snowball(english)
+          vocab: ykl
+          ngram: 1
+          min_df: 1
+          access: hidden
+    outs:
+    - path: data/projects/ykl-omikuji-parabel-en
+      hash: md5
+      md5: f1d1a08423ae130c839f3db15bc9ddf3.dir
+      size: 5326770
+      nfiles: 6
+  eval-omikuji-parabel@2:
+    cmd: annif eval ykl-omikuji-parabel-en -j 8 -m Precision@1 -m NDCG --metrics-file
+      reports/test-omikuji-parabel-en.json --results-file reports/test-omikuji-parabel-en.csv
+      corpora/kirjaesittelyt2021/kvesit-ykl-eng-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-eng-test.tsv
+      md5: bc7f924d84a20e9ab91e2600fd2415ec
+      size: 210866
+    - path: data/projects/ykl-omikuji-parabel-en
+      hash: md5
+      md5: f1d1a08423ae130c839f3db15bc9ddf3.dir
+      size: 5326770
+      nfiles: 6
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: reports/test-omikuji-parabel-en.csv
+      hash: md5
+      md5: c856d122742783f93908ed93bf2357fc
+      size: 212459
+    - path: reports/test-omikuji-parabel-en.json
+      hash: md5
+      md5: 1a00cc0b691c5a04e71f5f29896af3a2
+      size: 100
+  eval-omikuji-bonsai@1:
+    cmd: annif eval ykl-omikuji-bonsai-sv -j 8 -m Precision@1 -m NDCG --metrics-file
+      reports/test-omikuji-bonsai-sv.json --results-file reports/test-omikuji-bonsai-sv.csv
+      corpora/kirjaesittelyt2021/kvesit-ykl-swe-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-swe-test.tsv
+      md5: a9177ace2f0d1dffa6a424fccdf04b37
+      size: 287821
+    - path: data/projects/ykl-omikuji-bonsai-sv
+      hash: md5
+      md5: e589e2a60b5f7cdcc5e67c7443b4c87a.dir
+      size: 8507130
+      nfiles: 6
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: reports/test-omikuji-bonsai-sv.csv
+      hash: md5
+      md5: 9a6cd19c12e6d7da7f0021735d81c62a
+      size: 214103
+    - path: reports/test-omikuji-bonsai-sv.json
+      hash: md5
+      md5: 95ea663acf56c7fbd27d43f755758610
+      size: 87
+  eval-ensemble@0:
+    cmd: annif eval ykl-fi -j 8 -m Precision@1 -m NDCG --metrics-file reports/test-fi.json
+      --results-file reports/test-fi.csv corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-fin-test.tsv
+      hash: md5
+      md5: 9f8d094ae71ec68c5d40dde46d991e1f
+      size: 4534782
+    - path: data/projects/ykl-fi
+      hash: md5
+      md5: d751713988987e9331980363e24189ce.dir
+      size: 0
+      nfiles: 0
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: reports/test-fi.csv
+      hash: md5
+      md5: 38c9f688aaf641cc166db2dc72da3ef1
+      size: 237209
+    - path: reports/test-fi.json
+      hash: md5
+      md5: 47cfa99502a3ab3f5a1852b8708cccf9
+      size: 100
+  eval-ensemble@1:
+    cmd: annif eval ykl-sv -j 8 -m Precision@1 -m NDCG --metrics-file reports/test-sv.json
+      --results-file reports/test-sv.csv corpora/kirjaesittelyt2021/kvesit-ykl-swe-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-swe-test.tsv
+      hash: md5
+      md5: a9177ace2f0d1dffa6a424fccdf04b37
+      size: 287821
+    - path: data/projects/ykl-sv
+      hash: md5
+      md5: d751713988987e9331980363e24189ce.dir
+      size: 0
+      nfiles: 0
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: reports/test-sv.csv
+      hash: md5
+      md5: 2fff6f3bc20fddd861bf040622e28166
+      size: 214271
+    - path: reports/test-sv.json
+      hash: md5
+      md5: 4f76e748ac704e7b47d74c2c459cacaf
+      size: 87
+  eval-ensemble@2:
+    cmd: annif eval ykl-en -j 8 -m Precision@1 -m NDCG --metrics-file reports/test-en.json
+      --results-file reports/test-en.csv corpora/kirjaesittelyt2021/kvesit-ykl-eng-test.tsv
+    deps:
+    - path: corpora/kirjaesittelyt2021/kvesit-ykl-eng-test.tsv
+      hash: md5
+      md5: bc7f924d84a20e9ab91e2600fd2415ec
+      size: 210866
+    - path: data/projects/ykl-en
+      hash: md5
+      md5: d751713988987e9331980363e24189ce.dir
+      size: 0
+      nfiles: 0
+    - path: venv-installed
+      hash: md5
+      md5: f85f21b68735b126c2241fbff83fd0ef
+      size: 41
+    outs:
+    - path: reports/test-en.csv
+      hash: md5
+      md5: eea5b22824d23cb90267eb969a613300
+      size: 212565
+    - path: reports/test-en.json
+      hash: md5
+      md5: 8d41a89395974efbf8ae06880f94d3c4
+      size: 100

dvc.yaml

@@ -0,0 +1,122 @@
+stages:
+  install:
+    cmd: python3 -m venv venv && . venv/bin/activate && pip install -U pip wheel setuptools && pip install -r requirements.txt && cp requirements.txt venv-installed
+    deps:
+    - requirements.txt
+    outs:
+    - venv-installed:
+        cache: false
+  load-vocab:
+    cmd: annif load-vocab ykl corpora/ykl-skos.ttl
+    deps:
+    - venv-installed
+    - corpora/ykl-skos.ttl
+    outs:
+    - data/vocabs/ykl
+  train-omikuji-parabel:
+    foreach:
+      - lang2: fi
+        lang3: fin
+      - lang2: sv
+        lang3: swe
+      - lang2: en
+        lang3: eng
+    do:
+      cmd: annif train ykl-omikuji-parabel-${item.lang2} -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-train.tsv
+      params:
+      - projects.toml:
+        - ykl-omikuji-parabel-${item.lang2}
+      deps:
+      - venv-installed
+      - corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-train.tsv
+      - data/vocabs/ykl
+      outs:
+      - data/projects/ykl-omikuji-parabel-${item.lang2}
+  eval-omikuji-parabel:
+    foreach:
+      - lang2: fi
+        lang3: fin
+      - lang2: sv
+        lang3: swe
+      - lang2: en
+        lang3: eng
+    do:
+      cmd: annif eval ykl-omikuji-parabel-${item.lang2} -j 8 -m Precision@1 -m NDCG
+        --metrics-file reports/test-omikuji-parabel-${item.lang2}.json
+        --results-file reports/test-omikuji-parabel-${item.lang2}.csv
+        corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-test.tsv
+      deps:
+      - venv-installed
+      - corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-test.tsv
+      - data/projects/ykl-omikuji-parabel-${item.lang2}
+      outs:
+      - reports/test-omikuji-parabel-${item.lang2}.csv:
+          cache: false
+      metrics:
+      - reports/test-omikuji-parabel-${item.lang2}.json:
+          cache: false
+  train-omikuji-bonsai:
+    foreach:
+      - lang2: fi
+        lang3: fin
+      - lang2: sv
+        lang3: swe
+      - lang2: en
+        lang3: eng
+    do:
+      cmd: annif train ykl-omikuji-bonsai-${item.lang2} -j 8 corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-train.tsv
+      params:
+      - projects.toml:
+        - ykl-omikuji-bonsai-${item.lang2}
+      deps:
+      - venv-installed
+      - corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-train.tsv
+      - data/vocabs/ykl
+      outs:
+      - data/projects/ykl-omikuji-bonsai-${item.lang2}
+  eval-omikuji-bonsai:
+    foreach:
+      - lang2: fi
+        lang3: fin
+      - lang2: sv
+        lang3: swe
+      - lang2: en
+        lang3: eng
+    do:
+      cmd: annif eval ykl-omikuji-bonsai-${item.lang2} -j 8 -m Precision@1 -m NDCG
+        --metrics-file reports/test-omikuji-bonsai-${item.lang2}.json
+        --results-file reports/test-omikuji-bonsai-${item.lang2}.csv
+        corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-test.tsv
+      deps:
+      - venv-installed
+      - corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-test.tsv
+      - data/projects/ykl-omikuji-bonsai-${item.lang2}
+      outs:
+      - reports/test-omikuji-bonsai-${item.lang2}.csv:
+          cache: false
+      metrics:
+      - reports/test-omikuji-bonsai-${item.lang2}.json:
+          cache: false
+  eval-ensemble:
+    foreach:
+      - lang2: fi
+        lang3: fin
+      - lang2: sv
+        lang3: swe
+      - lang2: en
+        lang3: eng
+    do:
+      cmd: annif eval ykl-${item.lang2} -j 8 -m Precision@1 -m NDCG
+        --metrics-file reports/test-${item.lang2}.json
+        --results-file reports/test-${item.lang2}.csv
+        corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-test.tsv
+      deps:
+      - venv-installed
+      - corpora/kirjaesittelyt2021/kvesit-ykl-${item.lang3}-test.tsv
+      - data/projects/ykl-${item.lang2}
+      outs:
+      - reports/test-${item.lang2}.csv:
+          cache: false
+      metrics:
+      - reports/test-${item.lang2}.json:
+          cache: false

projects.d/1-projects-ykl.toml

@@ -0,0 +1,89 @@
+[ykl-omikuji-parabel-fi]
+name="YKL Omikuji Parabel Finnish"
+language="fi"
+backend="omikuji"
+analyzer="voikko(fi)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-bonsai-fi]
+name="YKL Omikuji Bonsai Finnish"
+language="fi"
+backend="omikuji"
+cluster_balanced = "False"
+cluster_k = 100
+max_depth = 3
+analyzer="voikko(fi)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-parabel-sv]
+name="YKL Omikuji Parabel Swedish"
+language="sv"
+backend="omikuji"
+analyzer="snowball(swedish)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-bonsai-sv]
+name="YKL Omikuji Bonsai Swedish"
+language="sv"
+backend="omikuji"
+cluster_balanced = "False"
+cluster_k = 100
+max_depth = 3
+analyzer="snowball(swedish)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-parabel-en]
+name="YKL Omikuji Parabel English"
+language="en"
+backend="omikuji"
+analyzer="snowball(english)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-bonsai-en]
+name="YKL Omikuji Bonsai English"
+language="en"
+backend="omikuji"
+cluster_balanced = "False"
+cluster_k = 100
+max_depth = 3
+analyzer="snowball(english)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-fi]
+name="YKL suomi"
+language="fi"
+backend="ensemble"
+vocab="ykl"
+sources="ykl-omikuji-parabel-fi,ykl-omikuji-bonsai-fi"
+
+[ykl-sv]
+name="KAB svenska"
+language="sv"
+backend="ensemble"
+vocab="ykl"
+sources="ykl-omikuji-parabel-sv,ykl-omikuji-bonsai-sv"
+
+[ykl-en]
+name="PLC English"
+language="en"
+backend="ensemble"
+vocab="ykl"
+sources="ykl-omikuji-parabel-en,ykl-omikuji-bonsai-en"

projects.toml

@@ -0,0 +1,89 @@
+[ykl-omikuji-parabel-fi]
+name="YKL Omikuji Parabel Finnish"
+language="fi"
+backend="omikuji"
+analyzer="voikko(fi)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-bonsai-fi]
+name="YKL Omikuji Bonsai Finnish"
+language="fi"
+backend="omikuji"
+cluster_balanced = "False"
+cluster_k = 100
+max_depth = 3
+analyzer="voikko(fi)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-parabel-sv]
+name="YKL Omikuji Parabel Swedish"
+language="sv"
+backend="omikuji"
+analyzer="snowball(swedish)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-bonsai-sv]
+name="YKL Omikuji Bonsai Swedish"
+language="sv"
+backend="omikuji"
+cluster_balanced = "False"
+cluster_k = 100
+max_depth = 3
+analyzer="snowball(swedish)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-parabel-en]
+name="YKL Omikuji Parabel English"
+language="en"
+backend="omikuji"
+analyzer="snowball(english)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-omikuji-bonsai-en]
+name="YKL Omikuji Bonsai English"
+language="en"
+backend="omikuji"
+cluster_balanced = "False"
+cluster_k = 100
+max_depth = 3
+analyzer="snowball(english)"
+vocab="ykl"
+ngram=1
+min_df=1
+access="hidden"
+
+[ykl-fi]
+name="YKL suomi"
+language="fi"
+backend="ensemble"
+vocab="ykl"
+sources="ykl-omikuji-parabel-fi,ykl-omikuji-bonsai-fi"
+
+[ykl-sv]
+name="KAB svenska"
+language="sv"
+backend="ensemble"
+vocab="ykl"
+sources="ykl-omikuji-parabel-sv,ykl-omikuji-bonsai-sv"
+
+[ykl-en]
+name="PLC English"
+language="en"
+backend="ensemble"
+vocab="ykl"
+sources="ykl-omikuji-parabel-en,ykl-omikuji-bonsai-en"

reports/test-en.json

@@ -0,0 +1,5 @@
+{
+  "Precision@1": 0.49295774647887325,
+  "NDCG": 0.6053284406661987,
+  "Documents_evaluated": 213
+}

reports/test-fi.json

@@ -0,0 +1,5 @@
+{
+  "Precision@1": 0.5680110715697905,
+  "NDCG": 0.7042226791381836,
+  "Documents_evaluated": 5058
+}

reports/test-omikuji-bonsai-en.json

@@ -0,0 +1,5 @@
+{
+  "Precision@1": 0.49295774647887325,
+  "NDCG": 0.6082379221916199,
+  "Documents_evaluated": 213
+}

reports/test-omikuji-bonsai-fi.json

@@ -0,0 +1,5 @@
+{
+  "Precision@1": 0.5662317121391854,
+  "NDCG": 0.7037694454193115,
+  "Documents_evaluated": 5058
+}

reports/test-omikuji-bonsai-sv.json

@@ -0,0 +1,5 @@
+{
+  "Precision@1": 0.6375,
+  "NDCG": 0.7122230529785156,
+  "Documents_evaluated": 400
+}

reports/test-omikuji-parabel-en.json

@@ -0,0 +1,5 @@
+{
+  "Precision@1": 0.48826291079812206,
+  "NDCG": 0.5963867902755737,
+  "Documents_evaluated": 213
+}

reports/test-omikuji-parabel-fi.json

@@ -0,0 +1,5 @@
+{
+  "Precision@1": 0.56069592724397,
+  "NDCG": 0.691719114780426,
+  "Documents_evaluated": 5058
+}

reports/test-omikuji-parabel-sv.json

@@ -0,0 +1,5 @@
+{
+  "Precision@1": 0.6475,
+  "NDCG": 0.7179128527641296,
+  "Documents_evaluated": 400
+}

reports/test-sv.json

@@ -0,0 +1,5 @@
+{
+  "Precision@1": 0.6475,
+  "NDCG": 0.7206699848175049,
+  "Documents_evaluated": 400
+}

requirements.txt

@@ -0,0 +1 @@
+annif[fasttext,omikuji,nn,voikko]==1.0.*

sync-model-data-ocp.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+
+# Runs rsync to transfer model data from the current directory to an OpenShift volume
+# that is attached to a pod which is running Annif. The instance
+# {api-annif-org,ai-finto-fi, etc.} to transfer to is given as the argument.
+# You need to be logged to the cluster with the oc tool.
+
+set -e
+
+if [ $# -ne 1 ]
+  then
+    echo "Not enough arguments; argument 1: destination_instance"
+    exit 1
+fi
+
+pod=$(oc get pods -l app.kubernetes.io/instance=$1,app.kubernetes.io/name=annif -o name)
+
+if [[ $pod = *[[:space:]]* ]]
+  then
+    echo "Multiple pod exists; using first"
+    pod=(${pod//$'\n'/ })
+fi
+echo "Target is "$pod
+pod=${pod#pod/}
+if [ -z "${pod}" ]
+  then
+    echo "No target pod found"
+    exit 1
+fi
+
+rsync --rsh='oc rsh' -avrL --exclude="*train*" --exclude="*zip" --inplace projects.d $pod:/annif-projects
+rsync --rsh='oc rsh' -avrL --exclude="*train*" --exclude="*zip" --inplace data/{projects,vocabs} $pod:/annif-projects/data