File size: 2,075 Bytes
779ff0b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
"""Paths, hashing and file helpers every step shares.

Data, base weights and runs live outside the repository, under `$LAYA_TASKS_HOME` (default `~/.laya-tasks`), one
folder a task. None of them belong in a model repository's code folder, and keeping them apart makes that easy to
check.
"""

from __future__ import annotations

import hashlib
import json
import os
from pathlib import Path
from typing import Any, Dict, Iterable, List


def home() -> Path:
    """The root for every task's data and runs."""
    return Path(os.environ.get("LAYA_TASKS_HOME", "~/.laya-tasks")).expanduser()


def task_home(task: str) -> Path:
    return home() / task


def data_dir(task: str) -> Path:
    return task_home(task) / "data" / "v1"


def run_dir(task: str, run: str) -> Path:
    return task_home(task) / "runs" / run


def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for block in iter(lambda: f.read(1 << 20), b""):
            h.update(block)
    return h.hexdigest()


def write_jsonl(path: Path, items: Iterable[Dict[str, Any]]) -> None:
    """Write one JSON object a line, through a temporary file, so a crash never leaves half a file in place."""
    tmp = path.with_suffix(path.suffix + ".tmp")
    with open(tmp, "w", encoding="utf-8", newline="\n") as f:
        for item in items:
            f.write(json.dumps(item, ensure_ascii=False, sort_keys=True) + "\n")
    os.replace(tmp, path)


def read_jsonl(path: Path) -> List[Dict[str, Any]]:
    with open(path, encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]


def write_json(path: Path, value: Any) -> None:
    with open(path, "w", encoding="utf-8", newline="\n") as f:
        json.dump(value, f, indent=2, ensure_ascii=False)
        f.write("\n")


def read_json(path: Path) -> Any:
    with open(path, encoding="utf-8") as f:
        return json.load(f)


def normalise(text: str) -> str:
    """Text compared for copies between train and test: case and spacing ignored."""
    return " ".join(text.lower().split())