Upload 43 files

README.rst

@@ -0,0 +1,175 @@
+NLP : - TEXT TO KNOWLEDGE GRAPH
+we use the CamelParser to process it and extract the knowledge graph in JSON format.
+Then we use RULE-BASED Resoning to Extract Entities and Relations from the text and build a knowledge graph in JSON format.
+
+CamelParser
+=============
+
+.. image:: https://img.shields.io/pypi/l/camel-tools.svg
+   :target: https://opensource.org/licenses/MIT
+   :alt: MIT License
+
+Introduction
+------------
+
+CamelParser is an open-source Python-based Arabic dependency parser targeting two popular 
+Arabic dependency formalisms, the Columbia Arabic Treebank (CATiB), and Universal Dependencies (UD).
+
+The CamelParser pipeline handles the processing of raw text and produces tokenization, 
+part-of-speech and rich morphological features. For disambiguation, users can choose between 
+the BERT unfactored disambiguator, or a lighter Maximum Likelihood Estimation (MLE) disambiguator, 
+both of which are included in CAMeL Tools. For dependency parsing, we use the SuPar Biaffine Dependency Parser.
+
+
+Installation
+------------
+1. Clone this repo
+
+2. Set up a virtual environment using Python 3.11.13 (you can follow the tutorial `here <https://www.youtube.com/watch?si=g2zx-D0zdkL50knb&v=31WU0Dhw4sk&feature=youtu.be>`_).
+
+Currently, CamelParser does not work with later versions of python due to issues with some dependencies.
+
+3. Install the required packages:
+
+.. code-block:: bash
+
+    pip install -r requirements.txt
+
+4. Download dependency parsing models:
+
+.. code-block:: bash
+
+    python download_models.py
+
+Currently, two Arabic script models, CATiB and UD, will be downloaded from the CAMeL Lab's 
+`parser models collection <https://huggingface.co/collections/CAMeL-Lab/camelparser-654a3df21f70b3b5e72f95d9>`_ 
+on Hugging Face. More models will be added soon!
+
+Examples
+--------
+The CamelParser allows users to pass either a string or a file containing **one or more** sentences.
+Below are examples using the different *string* inputs that CamelParser accepts. 
+We pass each example as a string using -s. 
+However, we do recommend using the file method (-i) along with the path to the file
+when passing multiple sentences.
+
+You can also refer to the **sample_starting_point.py** to use the parser in your code, or for more advanced usage:
+
+.. code-block:: bash
+
+    text_to_conll_cli.py
+
+    handle_multiple_texts.py
+
+    handle_multiple_conll_files.py
+
+Passing text
+
+.. code-block:: bash
+    
+    python text_to_conll_cli.py -f text -s "جامعة نيويورك أبو ظبي تنشر أول أطلس لكوكب المريخ باللغة العربية."
+
+The verbose version of the above example (default values are shown)
+
+.. code-block:: bash
+
+    python text_to_conll_cli.py -f text -b r13 -d bert -m catib -s "جامعة نيويورك أبو ظبي تنشر أول أطلس لكوكب المريخ باللغة العربية."
+
+Passing preprocessed text (cleaned and whitespace tokenized)
+
+.. code-block:: bash
+    
+    python text_to_conll_cli.py -f preprocessed_text -s "جامعة نيويورك أبو ظبي تنشر أول أطلس لكوكب المريخ باللغة العربية ."
+
+*Note that the difference between the -f text and preprocessed_text parser input settings is*
+*that for text we use different utilities from CAMeL Tools to* 
+`normalize unicode <https://camel-tools.readthedocs.io/en/latest/api/utils/normalize.html?highlight=normalize_unicode#camel_tools.utils.normalize.normalize_unicode>`_, 
+`dediactritize <https://camel-tools.readthedocs.io/en/latest/api/utils/dediac.html?highlight=dediac_ar>`_, 
+*clean the text using* 
+`arclean <https://camel-tools.readthedocs.io/en/latest/api/utils/charmap.html?highlight=arclean#utility>`_, 
+*and perform* 
+`whitespace tokenization <https://camel-tools.readthedocs.io/en/latest/api/tokenizers/word.html?highlight=simple_word_tokenize#camel_tools.tokenizers.word.simple_word_tokenize>`_.
+
+tokenized is used when 1) the text has already been tokenized, and 2) only dependency relations are needed; 
+the POS tags and features will not be generated.
+
+.. code-block:: bash
+    
+    python text_to_conll_cli.py -f tokenized -s "جامعة نيويورك أبو ظبي تنشر أول أطلس ل+ كوكب المريخ ب+ اللغة العربية ."
+
+tokenized_tagged is used when the user has the tokens and POS tags. They should be passed as tuples.
+
+.. code-block:: bash
+    
+    python text_to_conll_cli.py -f tokenized_tagged -s "(جامعة, NOM) (نيويورك, PROP) (أبو, PROP) (ظبي, PROP) (تنشر, VRB) (أول, NOM) (أطلس, NOM) (ل+, PRT) (كوكب, NOM) (المريخ, PROP) (ب+, PRT) (اللغة, NOM) (العربية, NOM) (., PNX)"
+
+
+Using a custom model
+------------------
+You can use your own dependency parser models by
+
+1. placing the model in the models directory (this directory is created when you run download_models.py, but you can create it yourself)
+
+2. place your model in the models directory
+
+3. when running one of the scripts, add -m [model_name]. Just type the model name WITHOUT the path.
+
+Extending the code
+------------------
+
+You can also use different parts of the code to create your own pipeline. 
+The handle_multiple_texts.py is an example of that. It can be used to parse a directory of text files, 
+saving the resulting CoNLL-X files to a given output directory.
+
+Using another morphology database
+---------------------------------
+
+Curently, the CamelParser uses CAMeLTools' default morphology database, the morphology-db-msa-r13.
+
+For our paper, we used the calima-msa-s31 database. To use this database, 
+follow these steps (note that you need an account with the LDC):
+
+
+1. Install camel_tools v1.5.6 or later (you can check this using camel_data -v)
+
+2. Download the camel data for the BERT unfactored (MSA) model, as well as the morphology database:
+
+.. code-block:: bash
+
+    camel_data -i morphology-db-msa-s31 
+    camel_data -i disambig-bert-unfactored-msa
+
+3. Download the LDC2010L01 from the ldc downloads:
+    - go to https://catalog.ldc.upenn.edu/organization/downloads
+    - search for LDC2010L01.tgz and download it
+
+4. DO NOT EXTRACT LDC2010L01.tgz! We'll use the following command from camel tools to install the db:
+
+.. code-block:: bash
+
+    camel_data -p morphology-db-msa-s31 /path/to/LDC2010L01.tgz
+
+5. When running the main script, use -b and pass calima-msa-s31.
+
+Reproducing paper results
+-------------------------
+
+To reproduce the results in our paper *CamelParser2.0: A State-of-the-Art Dependency Parser for Arabic*, please use the code found
+in the *paper_version* branch.
+
+Citation
+--------
+
+If you find the CamelParser useful in your research, please cite
+
+.. code-block:: bibtex
+
+    @inproceedings{Elshabrawy:2023:camelparser,
+        title = "{CamelParser2.0: A State-of-the-Art Dependency Parser for Arabic}",
+        author = {Ahmed Elshabrawy and 
+    Muhammed AbuOdeh and
+    Go Inoue and
+    Nizar Habash} ,
+        booktitle = {Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023)},
+        year = "2023"
+    }

requirements.txt

@@ -0,0 +1,80 @@
+annotated-doc==0.0.5
+antlr4-python3-runtime==4.9.3
+anyio==4.14.2
+cachetools==5.3.1
+camel-kenlm==2025.4.8
+camel_tools==1.5.6
+certifi==2023.5.7
+charset-normalizer==3.1.0
+click==8.1.3
+colorama==0.4.6
+defusedxml==0.7.1
+dill==0.3.7
+docopt==0.6.2
+editdistance==0.6.2
+emoji==2.4.0
+exceptiongroup==1.1.3
+filelock==3.12.0
+fsspec==2023.5.0
+future==0.18.3
+h11==0.16.0
+hf-xet==1.6.0
+httpcore==1.0.9
+httpx==0.28.1
+huggingface-hub==0.15.1
+hydra-core==1.3.2
+idna==3.4
+iniconfig==2.0.0
+Jinja2==3.1.2
+joblib==1.2.0
+markdown-it-py==4.2.0
+MarkupSafe==2.1.3
+mdurl==0.1.2
+mpmath==1.3.0
+muddler==0.1.3
+multiprocess==0.70.15
+narwhals==2.24.0
+networkx==3.1
+nltk==3.8.1
+numpy==1.24.3
+omegaconf==2.3.0
+opt-einsum==3.3.0
+packaging==23.1
+pandas==2.0.2
+pathos==0.3.1
+platformdirs==4.11.1
+pluggy==1.2.0
+pox==0.3.3
+ppft==1.7.6.7
+protobuf==4.23.2
+Pygments==2.20.0
+pyrsistent==0.19.3
+pytest==7.4.0
+python-dateutil==2.8.2
+pytz==2023.3
+PyYAML==6.0
+regex==2023.6.3
+requests==2.31.0
+rich==15.0.0
+safetensors==0.8.0
+scikit-learn==1.2.2
+scipy==1.10.1
+shellingham==1.5.4
+six==1.16.0
+stanza==1.5.0
+supar @ git+https://github.com/yzhangcs/parser.git@17ec77dd05aaaa8160e3555134ff5f3b654e4991
+sympy==1.12
+tabulate==0.9.0
+termcolor==3.3.0
+threadpoolctl==3.1.0
+tokenizers==0.13.3
+tomli==2.0.1
+torch==2.0.1
+tqdm==4.65.0
+transformers==4.29.2
+typer==0.27.1
+typing_extensions==4.6.3
+tzdata==2023.3
+udapi==0.5.2
+udtools==0.2.8
+urllib3==2.0.2

src/classes.py

@@ -0,0 +1,70 @@
+from dataclasses import dataclass
+from typing import List, Union
+from dataclasses import astuple, dataclass
+import pandas as pd
+from camel_tools.disambig.bert import BERTUnfactoredDisambiguator
+from camel_tools.disambig.mle import MLEDisambiguator
+from camel_tools.utils.charmap import CharMapper
+
+def get_conll_tree_header_list():
+    return ["ID", "FORM", "LEMMA", "UPOS", "XPOS", "FEATS", "HEAD", "DEPREL", "DEPS", "MISC"]
+
+@dataclass
+class ConllParams:
+    file_path: str
+    parse_model_path: str
+    
+    def __iter__(self):
+        return iter(astuple(self))
+
+@dataclass
+class TextParams:
+    lines: List[str]
+    parse_model_path: str
+    arclean: CharMapper
+    disambiguator_param: Union[BERTUnfactoredDisambiguator, MLEDisambiguator, str]
+    clitic_feats_df: pd.DataFrame
+    tagset: str
+    morphology_db_type: str
+    
+    def __iter__(self):
+        return iter(astuple(self))
+
+@dataclass
+class PreprocessedTextParams:
+    lines: List[str]
+    parse_model_path: str
+    disambiguator: Union[BERTUnfactoredDisambiguator, MLEDisambiguator, str]
+    clitic_feats_df: pd.DataFrame
+    tagset: str
+    morphology_db_type: str
+    
+    def __iter__(self):
+        return iter(astuple(self))
+
+@dataclass
+class TokenizedParams:
+    lines: List[str]
+    parse_model_path: str
+
+@dataclass
+class TokenizedTaggedParams:
+    lines: List[str]
+    parse_model_path: str
+
+@dataclass
+class Token:
+    ID: int = -1
+    FORM: str = '_'
+    LEMMA: str = '_'
+    UPOS: str = '_'
+    XPOS: str = '_'
+    FEATS: str = '_'
+    HEAD: int = 0
+    DEPREL: str = '_'
+    DEPS: str = '_'
+    MISC: str = '_'
+    
+    def to_conll_row(self):
+        col_vals = [str(getattr(self, key)) for key in self.__dataclass_fields__.keys()]
+        return '\t'.join(col_vals)

src/conll_output.py

@@ -0,0 +1,50 @@
+
+
+from pathlib import Path
+import re
+from typing import List, Union
+
+from .classes import Token
+
+
+def print_to_conll(string_lines):
+    for line in string_lines:
+        print(line)
+
+def save_to_file(string_lines: List[str], file_path: Path):
+    with open(file_path, 'w') as f:
+        [f.write(f'{line}\n') for line in string_lines]
+
+def text_tuples_to_string(
+        text_tuples: List[List[tuple]], 
+        file_type,
+        annotations: Union[List[str], None]=None, 
+        sentences: Union[List[str], None]=None
+    ):
+    if sentences is not None and file_type != 'conll': 
+        # filter out empty lines
+        sentences = list(filter(lambda x : len(re.sub(r"\s+", "", x, flags=re.UNICODE)) > 0, sentences))
+    # get treeTokens
+    tokens = [[tup[1] for tup in sent] for sent in text_tuples]
+
+    string_lines: List[str] = []
+    for i, sentence_tuples in enumerate(text_tuples):
+        if file_type == 'conll': # dont add comments to preexisting conll files
+            pass
+        elif sentences:
+            string_lines.extend(
+                (
+                    f"# text = {sentences[i].strip()}",
+                    f"# treeTokens = {' '.join(tokens[i])}",
+                )
+            )
+        elif annotations:
+            string_lines.append(annotations[i])
+
+        for token_tuple in sentence_tuples:
+            token = Token(*token_tuple)
+            string_lines.append(token.to_conll_row())
+
+        string_lines.append('') # add empty line between trees
+    
+    return string_lines

src/data_preparation.py

@@ -0,0 +1,199 @@
+
+import os
+import re
+from typing import List, Union
+import pandas as pd
+from camel_tools.disambig.common import DisambiguatedWord
+
+from src.utils.conll_fixes import adjust_eof_newlines
+from .classes import ConllParams, TextParams, PreprocessedTextParams, TokenizedParams, TokenizedTaggedParams
+from .dependency_parser.biaff_parser import parse_conll, parse_text_tuples
+from .initialize_disambiguator.disambiguator_interface import get_disambiguator
+from .parse_disambiguation.disambiguation_analysis import to_sentence_analysis_list
+from .parse_disambiguation.feature_extraction import to_conll_fields_list
+from .utils.text_cleaner import clean_lines, clean_mad, split_lines_words
+from .logger import log
+
+
+FileTypeParams = Union[ConllParams, TextParams, PreprocessedTextParams, TokenizedParams, TokenizedTaggedParams]
+
+
+def get_feats_from_text_tuples(text_tuples: List[List[tuple]]) -> List[List[str]]:
+    """Extract the FEATS columns from the unparsed data.
+    FEATS will exist only for text and pre-processed text inputs.
+
+    Args:
+        text_tuples (List[List[tuple]]): unparsed data
+
+    Returns:
+        List[List[str]]: the FEATS column (or _ if it does not exist)
+    """
+    try:
+        return [[col_items[5] for col_items in tup_list] for tup_list in text_tuples]
+    except Exception as e:
+        print(e)
+        print('Not enough elements in tuple.')
+
+
+def add_feats(text_tuples: List[List[tuple]], text_feats: List[List[str]]) -> List[List[tuple]]:
+    """Add FEATS data to the text tuples.
+    The parent list (text_tuples) is a list of sentences.
+    Each sentence is a list of tuples.
+    Each tuple represents a token.
+
+    Args:
+        text_tuples (List[List[tuple]]): list of list of tuples
+        text_feats (List[List[str]]): list of list of FEATS
+
+    Returns:
+        List[List[tuple]]: text_tuples but with the FEATS column filled
+    """
+    text_tuples_with_feats = []
+    for sentence_tuples, sentence_feats in zip(text_tuples, text_feats):
+        
+        # get first 5 and last 4 items from parsed tuple using lists, and add features.
+        # Convert the list of fields to a tuple
+        merged_tuples = [
+            tuple(list(token_tuple[:5]) + [token_feats] + list(token_tuple[6:]))
+            for token_tuple, token_feats in zip(sentence_tuples, sentence_feats)
+        ]
+        text_tuples_with_feats.append(merged_tuples)
+    return text_tuples_with_feats
+
+def string_to_tuple_list(string_of_tuples: str) -> List[tuple[str, str]]:
+    """Take a string of space-separated tuples and convert it to a tuple list.
+    Example input: '(جامعة, NOM) (نيويورك, PROP)'
+    Example output: [(جامعة, NOM), (نيويورك, PROP)]
+
+    Args:
+        string_of_tuples (str): string of tuples
+
+    Returns:
+        List(tuple[str, str]): list of token-pos tuple pairs
+    """
+    sentence_tuples = []
+    
+    # split on space, and using positive lookbehind and lookahead
+    # to detect parentheses around the space
+    for tup in re.split(r'(?<=\)) (?=\()', string_of_tuples.strip()):
+        # tup = (جامعة, NOM)
+        tup_items = tup[1:-1] # removes parens
+        form = (','.join(tup_items.split(',')[:-1])).strip() # account for comma tokens
+        pos = (tup_items.split(',')[-1]).strip()
+        sentence_tuples.append((form, pos))
+    return sentence_tuples
+
+def get_tree_tokens(tok_pos_tuples):
+    sentences = []
+    for sentence_tuples in tok_pos_tuples:
+        sentence = ' '.join([tok_pos_tuple[0] for tok_pos_tuple in sentence_tuples])
+        sentences.append(sentence)
+    return sentences
+
+def handle_conll(file_type_params):
+    file_path, parse_model_path = file_type_params
+    # pass the path to the text file and the model path and name, and get the tuples
+    return parse_conll(file_path, parse_model=parse_model_path)
+
+@log
+def disambiguate_sentences(disambiguator, token_lines):
+    # moved to own function to add to logger
+    return disambiguator.disambiguate_sentences(token_lines)
+
+def handle_text_types(file_type_params, text_type: str):
+    if text_type == 'preprocessed_text':
+        lines, _, disambiguator_param, clitic_feats_df, tagset, morphology_db_type = file_type_params
+
+        token_lines = split_lines_words(lines)
+        token_lines = clean_mad(token_lines)
+    elif text_type == 'text':
+        lines, _, arclean, disambiguator_param, clitic_feats_df, tagset, morphology_db_type = file_type_params
+        # clean lines
+        token_lines = clean_lines(lines, arclean)
+    else:
+        assert False, f'Invalid type to process: {text_type}'
+
+    token_lines = [token_line for token_line in token_lines if token_line]
+    
+    # if str passed, we should create the disambiguator using disambiguator_param and morphology_db_type
+    if type(disambiguator_param) == str:
+        disambiguator = get_disambiguator(disambiguator_param, morphology_db_type)
+    else: # a disambiguator was passed
+        disambiguator = disambiguator_param
+    
+    # run the disambiguator on the sentence list to get an analysis for all sentences
+    disambiguated_sentences: List[List[DisambiguatedWord]] = disambiguate_sentences(disambiguator, token_lines)
+    # get a single analysis for each word (top or tok_match, match not implemented yet)
+    # sentence_analysis_list: List[List[dict]] = to_sentence_analysis_list(disambiguated_sentences, selection, selection_criteria)
+    sentence_analysis_list: List[List[dict]] = to_sentence_analysis_list(disambiguated_sentences, token_lines)
+
+    # extract the relevant items from each analysis into conll fields
+    return to_conll_fields_list(sentence_analysis_list, clitic_feats_df, tagset)
+    
+
+def handle_preprocessed_text(file_type_params):
+    return handle_text_types(file_type_params, 'preprocessed_text')
+
+def handle_text(file_type_params):
+    return handle_text_types(file_type_params, 'text')
+
+def handle_tokenized(file_type_params):
+    lines = file_type_params.lines
+    # construct tuples before sending them to the parser
+    return [[(0, tok, '_' ,'UNK', '_', '_', '_', '_', '_', '_') for tok in line.strip().split(' ')] for line in lines]
+
+def handle_tokenized_tagged(file_type_params):
+    lines = file_type_params.lines
+    # convert input tuple list into a tuple data structure
+    tok_pos_tuples_list = [string_to_tuple_list(line) for line in lines]
+    # since we did not start with sentences, we make sentences using the tokens (which we call tree tokens)
+    lines = get_tree_tokens(tok_pos_tuples_list)
+    # construct tuples before sending them to the parser
+    return [[(0, tup[0],'_' ,tup[1], '_', '_', '_', '_', '_', '_') for tup in tok_pos_tuples] for tok_pos_tuples in tok_pos_tuples_list]
+
+def get_file_type_params(lines, file_type, file_path, parse_model_path,
+    arclean, disambiguator_type, clitic_feats_df, tagset, morphology_db_type):
+    if file_type == 'conll':
+        return ConllParams(file_path, parse_model_path)
+    elif file_type == 'text':
+        return TextParams(lines, parse_model_path, arclean, disambiguator_type, clitic_feats_df, tagset, morphology_db_type)
+    elif file_type == 'preprocessed_text':
+        return PreprocessedTextParams(lines, parse_model_path, disambiguator_type, clitic_feats_df, tagset, morphology_db_type)
+    elif file_type == 'tokenized':
+        return TokenizedParams(lines, parse_model_path)
+    elif file_type == 'tokenized_tagged':
+        return TokenizedTaggedParams(lines, parse_model_path)
+
+def parse_text(file_type: str, file_type_params: FileTypeParams):
+    if file_type == 'conll':
+        # handle_conll(file_path, parse_model_path)
+        adjust_eof_newlines(file_type_params.file_path)
+        parsed_text_tuples = handle_conll(file_type_params)
+    else:
+        text_tuples: List[List[tuple]] = []
+        if file_type == 'text':
+            text_tuples = handle_text(file_type_params)
+        elif file_type == 'preprocessed_text':
+            text_tuples = handle_preprocessed_text(file_type_params)
+        elif file_type == 'tokenized':
+            text_tuples = handle_tokenized(file_type_params)
+        elif file_type == 'tokenized_tagged':
+            text_tuples = handle_tokenized_tagged(file_type_params)
+
+        # the text tuples created from the above processes is passed to the dependency parser
+        parsed_text_tuples = parse_text_tuples(text_tuples, parse_model=str(file_type_params.parse_model_path))
+        # for text/preprocessed_text, we want to extract the features to place in parsed_text_tuples
+        # TODO: check if this step can be skipped by placing features in a step above
+        text_feats: List[List[str]] = get_feats_from_text_tuples(text_tuples)
+        # place features in FEATS column
+        parsed_text_tuples = add_feats(parsed_text_tuples, text_feats)
+
+    return parsed_text_tuples
+
+def get_tagset(parse_model):
+    if parse_model == 'catib':
+        return 'catib6'
+    elif parse_model == 'ud':
+        return 'ud'
+    else:
+        raise f'{parse_model} model does not exist!'

src/dependency_parser/biaff_parser.py

@@ -0,0 +1,50 @@
+from typing import List, Union, Dict
+
+from supar import Parser
+from supar.utils import Dataset
+
+from camel_tools.utils.dediac import dediac_ar
+
+from ..logger import log
+
+"""
+conll object from parser
+iterate over sentences from this object
+iterate over the columns per sentence
+create token rows
+create conll sentence of token row tuples
+
+"""
+
+def parser_conll_to_conll_tuples(parser_conll: Dataset) -> List[List[tuple]]:
+    conll_sentences = []
+    for parser_sentence in parser_conll:
+        sentence_tree_token_tuples = []
+        for i in range(len(parser_sentence.values[0])):
+            token_tuple_row = tuple(column[i] for column in parser_sentence.values)
+            sentence_tree_token_tuples.append(token_tuple_row)
+        conll_sentences.append(sentence_tree_token_tuples)
+    return conll_sentences
+
+def filter_tatweel(form):
+    if form.replace("_", "").replace("\u0640","").replace("\u005F", "") == "":
+        return form
+    return form.replace("_", "").replace("\u0640","").replace("\u005F", "")
+
+@log
+def parse(conll_path_or_parsed_tuples: Union[List[List[tuple]], str], parse_model:str) -> List[List[tuple]]:
+    parser = Parser.load(parse_model)
+    return parser.predict(conll_path_or_parsed_tuples, verbose=False, tree=True, proj=True)
+
+
+def parse_text_tuples(sentence_tuples: List[List[tuple]], parse_model) -> List[List[tuple]]:
+    sentence_tuples = [[val[1:4] for val in sent] for sent in sentence_tuples]
+    form_lemma_pos_tuple = [[(filter_tatweel(dediac_ar(val[0])), filter_tatweel(dediac_ar(val[1])), val[2]) for val in sent] for sent in sentence_tuples]
+    conll = parse(form_lemma_pos_tuple, parse_model=parse_model)
+    return parser_conll_to_conll_tuples(conll)
+
+def parse_conll(conll_path: str, parse_model) -> List[List[tuple]]:
+    conll = parse(conll_path, parse_model=parse_model)
+    for i, sent in enumerate(conll):
+        conll[i].values[1] = [filter_tatweel(form) for form in sent.values[1]]
+    return parser_conll_to_conll_tuples(conll)

src/initialize_disambiguator/bert_disambiguator.py

@@ -0,0 +1,11 @@
+
+
+# class BertDisambiguatorAdapter:
+#     pass
+
+from camel_tools.disambig.bert import BERTUnfactoredDisambiguator
+
+def create_bert_disambiguator(analyzer):
+    model = BERTUnfactoredDisambiguator.pretrained("msa", top=1000, pretrained_cache=False)
+    model._analyzer = analyzer
+    return model

src/initialize_disambiguator/disambiguator_interface.py

@@ -0,0 +1,28 @@
+from typing import Union
+from camel_tools.morphology.database import MorphologyDB
+from camel_tools.morphology.analyzer import Analyzer
+from camel_tools.disambig.bert import BERTUnfactoredDisambiguator
+
+from ..logger import log
+from .bert_disambiguator import create_bert_disambiguator
+from .mle_disambiguator import MLEDisambiguatorAdapter
+
+def set_up_analyzer(morphology_db: str) -> Analyzer:
+    # used to initialize an Analyzer with ADD_PROP backoff 
+    # db = MorphologyDB.builtin_db('calima-msa-s31')
+    db_type = None if morphology_db == 'r13' else morphology_db
+    db = MorphologyDB.builtin_db(db_name=db_type)
+    return Analyzer(db=db, backoff='ADD_PROP', cache_size=100000)
+
+@log
+def get_disambiguator(model_name: str, morphology_db: str) -> Union[MLEDisambiguatorAdapter, BERTUnfactoredDisambiguator]:
+    analyzer = set_up_analyzer(morphology_db)
+    
+    if model_name == 'mle':
+        model = MLEDisambiguatorAdapter(analyzer)
+    elif model_name == 'bert':
+        model = create_bert_disambiguator(analyzer)
+    else:
+        raise ValueError('Invalid model')
+    
+    return model

src/initialize_disambiguator/mle_disambiguator.py

@@ -0,0 +1,17 @@
+from typing import List
+from camel_tools.disambig.mle import MLEDisambiguator
+from camel_tools.disambig.common import DisambiguatedWord
+from camel_tools.morphology.analyzer import Analyzer
+
+class MLEDisambiguatorAdapter():
+    def __init__(self, analyzer: Analyzer):
+        self.disambiguator = MLEDisambiguator(analyzer=analyzer)
+    
+    # def pretrained(self, analyzer):
+    #     self.disambiguator = self.disambiguator
+    
+    def disambiguate(self, sentence: List[str]) -> List[DisambiguatedWord]:
+        return self.disambiguator.disambiguate(sentence)
+    
+    def disambiguate_sentences(self, lines: List[List[str]]) -> List[List[DisambiguatedWord]]:
+        return [self.disambiguator.disambiguate(line) for line in lines]

src/logger.py

@@ -0,0 +1,46 @@
+
+import functools
+import logging
+import os
+import time
+from datetime import datetime
+
+logging.basicConfig(level = logging.DEBUG)
+logger = logging.getLogger()
+
+log_path = f'{os.getcwd()}/logs.txt'
+    
+with open(log_path, 'a') as f:
+    f.write(f'[{datetime.now().strftime("%d/%b/%Y %H:%M:%S")}]\n')
+
+def map_function_to_phrase(function_name):
+    """Replace logged function name with a proper phrase.
+    """
+    if function_name == "parse":
+        return "parsing duration"
+    elif function_name == "get_disambiguator":
+        return "disambiguator setup duration"
+    elif function_name == "disambiguate_sentences":
+        return "disambiguating sentences"
+    elif function_name == "main":
+        return "total execution time"
+    else:
+        return function_name
+
+def log(func):
+    @functools.wraps(func)
+    def wrapper(*args, **kwargs):
+        try:
+
+            start_time = time.time()
+            result = func(*args, **kwargs)
+            end_time = time.time()
+            
+            with open(log_path, 'a') as f:
+                f.write(f'{map_function_to_phrase(func.__name__)}: {round(end_time - start_time, 2)}s\n')
+            return result
+        except Exception as e:
+            logger.exception(f"Exception raised in {func.__name__}. exception: {str(e)}")
+            raise e
+
+    return wrapper

src/parse_disambiguation/disambiguation_analysis.py

@@ -0,0 +1,69 @@
+import sys
+from typing import List
+from camel_tools.disambig.common import DisambiguatedWord
+from camel_tools.utils.dediac import dediac_ar
+from ..classes import Token
+
+def get_tok_match_analysis(disambig_word, token):
+    for i, scored_analysis in enumerate(disambig_word.analyses):
+        if dediac_ar(scored_analysis.analysis['diac']) == dediac_ar(token):
+            if i != 0:
+                # print(f"#NOT_FIRST Input token: {token}, analysis: {get_first_analysis(disambig_word)['diac']}", file=sys.stderr)
+                print(f"#NOT_FIRST {token}", file=sys.stderr)
+
+            return scored_analysis.analysis
+    
+    # print(f"#NOAN token {token}, analysis: {get_first_analysis(disambig_word)['diac']}", file=sys.stderr)
+    print(f"#NOAN {token}", file=sys.stderr)
+    
+    return get_first_analysis(disambig_word) # no token match, so just return first
+
+
+def get_analysis_by_criteria(disambig_word: DisambiguatedWord, selection_criteria: dict) -> dict:
+    # runs assertions to ensure the data is good
+    is_analysis(disambig_word)
+    
+    for scored_analysis in disambig_word.analyses:
+        analysis = scored_analysis.analysis
+        if all(analysis[key] == selection_criteria[key] for key in selection_criteria):
+            return analysis
+    
+    # if the criteria is not found, return the first analysis
+    print(f"No analysis found for the given criteria on {disambig_word.word}, will return the first analysis.")
+    return get_first_analysis(disambig_word)
+
+def is_analysis(disambig_word: DisambiguatedWord) -> dict:
+    # this should not be empty
+    assert disambig_word, "Disambiguated word not found!"
+    
+    # this could be empty if there is no backoff, and this code assumes backoff
+    assert disambig_word.analyses, "No analyses found, are you using backoff?"
+    assert disambig_word.analyses[0], "No analyses found, are you using backoff?"
+
+def get_first_analysis(disambig_word: DisambiguatedWord):
+    # runs assertions to ensure the data is good
+    is_analysis(disambig_word)
+    
+    return disambig_word.analyses[0].analysis
+
+
+def get_sentence_analysis(disambiguated_sentence: List[DisambiguatedWord], token_line, selection, selection_criteria) -> List[dict]:
+    # sourcery skip: switch
+    # get an analysis based on the selection criteria
+    # top means take first
+    # tok_match takes the first analysis that matches diac, otherwise takes first
+    # match means take the first analysis that matches the selection criteria.
+    # selection criteria: {'criteria1': 'value1', 'criteria2': 'value2'...}
+    # return None if criteria not fulfilled.
+    # TODO: check if selection criteria contains valid keys
+    if selection == 'top':
+        return [get_first_analysis(disambig_word) for disambig_word in disambiguated_sentence]
+    elif selection == 'tok_match':
+        return [get_tok_match_analysis(disambig_word, token) for disambig_word, token in zip(disambiguated_sentence, token_line)]
+    elif selection == 'match':
+        return [get_analysis_by_criteria(disambig_word, selection_criteria) for disambig_word in disambiguated_sentence]
+    else:
+        raise ValueError(f"the selection {selection} is not valid!")
+
+def to_sentence_analysis_list(disambiguated_sentences: List[List[DisambiguatedWord]], token_lines, selection: str='tok_match', selection_criteria: dict=None) -> List[List[Token]]:
+    return [get_sentence_analysis(disambiguated_sentence, token_line, selection, selection_criteria) for disambiguated_sentence, token_line in zip(disambiguated_sentences, token_lines)]

src/parse_disambiguation/feature_extraction.py

@@ -0,0 +1,212 @@
+"""Given a list of list of analyses (one analysis per token),
+get features based on a given criteria.
+
+If no criteria is given, return atbtok and catib6
+"""
+
+import re
+from typing import List
+import json
+from camel_tools.utils.dediac import dediac_ar
+from camel_tools.utils.charmap import CharMapper
+from camel_tools.utils.transliterate import Transliterator
+import pandas as pd
+
+FEATURES_LIST = ["pos", "prc3", "prc2", "prc1", "prc0", "enc0", "asp", "vox", "mod", "gen", "num", "stt", "cas", "per", "rat"]
+
+def feats_dict_to_string(feats_dict):
+    # prc3=na|prc2=na|prc1=na|prc0=na|per=na|asp=na|vox=na|mod=na|gen=na|num=na|stt=na|cas=na|enc0=na|rat=na
+    feats_str = json.dumps(feats_dict)
+    return feats_str.replace('", "', "|").replace('": "', "=").replace('{"', '').replace('"}', '')
+
+def build_clitic_feats_dict(clitic_feat_list):
+    assert len(clitic_feat_list) != 0, f'invalid clitic, {clitic_feat_list}'
+    assert len(clitic_feat_list) == 1, f'multiple clitics matched when only one should, {clitic_feat_list}'
+    clitic_feat_list[0]['deciding_feat']
+    final_clitic_feats = {k: v for k,v in clitic_feat_list[0].items() if k in FEATURES_LIST}
+    final_clitic_feats['token_type'] = clitic_feat_list[0]['deciding_feat'].split(':')[0]
+    return final_clitic_feats
+
+def get_clitic_feats(token, clitic_order, clitic_feats, stem_feats):
+    mapper = CharMapper.builtin_mapper('ar2bw')
+    transliterator = Transliterator(mapper)
+    token = transliterator.transliterate(dediac_ar(token))
+    filtered_clitics = clitic_feats[(clitic_feats.clitic == token) & (clitic_feats.deciding_feat.str.startswith(clitic_order))]
+    clitic_list = [f'{k}:{v}' for k, v in stem_feats.items() if k.startswith(clitic_order) and v not in ['0', 'na']]
+
+    for feat_check in clitic_list:
+        clitic_feat_list = filtered_clitics[filtered_clitics.deciding_feat == feat_check].to_dict('records')
+        if clitic_feat_list:
+            return build_clitic_feats_dict(clitic_feat_list)
+    assert False, f"clitic '{token}' does not exist in clitics list. Stem features: {stem_feats}"
+
+def get_stem_feats(word_analysis):
+    return {feat: word_analysis[feat] for feat in FEATURES_LIST}
+
+def get_clitic_order(token):
+    if token.endswith('+'):
+        clitic_order = 'prc'
+    elif token.startswith('+'):
+        clitic_order = 'enc'
+    return clitic_order
+
+def is_clitic(token):
+    return (token.startswith('+') or token.endswith('+')) and not re.match(r'^\++$', token)
+
+def empty_clitic_feats_from_baseword(stem_feats):
+    clitic_type_list = ["prc3", "prc2", "prc1", "prc0", "enc0"]
+    for clitic_type in clitic_type_list:
+        if stem_feats[clitic_type] not in ['0', 'na', 'Al_det']:
+            stem_feats[clitic_type] = '0'
+    return stem_feats
+    
+
+def r13_fixes(token, stem_feats):
+    # handling edge cases where li comes from variations of wa li>n (i.e. li>nhu, li>nanY)
+    li_feats = {'pos': 'conj_sub', 'prc3': '0', 'prc2': 'wa_conj', 'prc1': '0', 'prc0': 'na', 'enc0': '3ms_pron', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'gen': 'na', 'num': 'na', 'stt': 'na', 'cas': 'na', 'per': 'na', 'rat': 'na'}
+    if {k for k, _ in stem_feats.items() ^ li_feats.items()} == {'enc0'}:
+        stem_feats['prc1'] = 'li_conj'
+    
+    # handling edge cases where li comes from variations of li>n (i.e. li>nhu, li>nanY)
+    li_feats = {'pos': 'conj_sub', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': 'na', 'enc0': '0', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'gen': 'na', 'num': 'na', 'stt': 'na', 'cas': 'na', 'per': 'na', 'rat': 'na'}
+    if {k for k, _ in stem_feats.items() ^ li_feats.items()} == {'enc0'}:
+        stem_feats['prc1'] = 'li_conj'
+        
+    # handling edge cases where li comes from li>n
+    if token == 'لِ+' and stem_feats == {'pos': 'conj_sub', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': 'na', 'enc0': '0', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'gen': 'na', 'num': 'na', 'stt': 'na', 'cas': 'na', 'per': 'na', 'rat': 'na'}:
+            stem_feats['prc1'] = 'li_conj'
+    
+    # handling edge cases where li comes from wa li>n
+    if token == 'لِ+' and stem_feats == {'pos': 'conj_sub', 'prc3': '0', 'prc2': 'wa_conj', 'prc1': '0', 'prc0': 'na', 'enc0': '3ms_pron', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'gen': 'na', 'num': 'na', 'stt': 'na', 'cas': 'na', 'per': 'na', 'rat': 'na'}:
+            stem_feats['prc1'] = 'li_conj'
+    
+    # handling edge cases where mA comes from qlmA
+    if token == '+ما' and stem_feats == {'pos': 'conj', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': 'na', 'enc0': '0', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'gen': 'na', 'num': 'na', 'stt': 'na', 'cas': 'na', 'per': 'na', 'rat': 'n'}:
+            stem_feats['enc0'] = 'mA_sub'
+    
+    # handling an edge case where li comes from likY
+    # added li_conj to clitic_feats.csv
+    if token == 'لِ+' and stem_feats == {'pos': 'conj', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': 'na', 'enc0': '0', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'gen': 'na', 'num': 'na', 'stt': 'na', 'cas': 'na', 'per': 'na', 'rat': 'na'}:
+        stem_feats['prc1'] = 'li_conj'
+
+def add_remaining_features(tokens_df, stem_feats, clitic_feats):
+    existing_clitics = ['prc0']
+
+    clitic_feats_list = []
+
+    for _, row in tokens_df.iterrows():
+        token = row['token']
+        if not is_clitic(token):
+            baseword_feats_dict = empty_clitic_feats_from_baseword(dict(stem_feats))
+            baseword_feats_dict['token_type'] = 'baseword'
+            clitic_feats_list.append(baseword_feats_dict)
+        else:
+            clitic_order = get_clitic_order(token)
+            # handling an edge case where lA is negative
+            if token == 'لِ+' and \
+                (stem_feats == {'pos': 'conj_sub', 'prc3': '0', 'prc2': '0', 'prc1': '0', 'prc0': 'na', 'enc0': 'lA_neg', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'gen': 'na', 'num': 'na', 'stt': 'na', 'cas': 'na', 'per': 'na', 'rat': 'na'}
+                or stem_feats == {'pos': 'conj_sub', 'prc3': '0', 'prc2': 'fa_conj', 'prc1': '0', 'prc0': 'na', 'enc0': 'lA_neg', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'gen': 'na', 'num': 'na', 'stt': 'na', 'cas': 'na', 'per': 'na', 'rat': 'na'} 
+                or stem_feats == {'pos': 'conj_sub', 'prc3': '0', 'prc2': 'wa_part', 'prc1': '0', 'prc0': 'na', 'enc0': 'lA_neg', 'asp': 'na', 'vox': 'na', 'mod': 'na', 'gen': 'na', 'num': 'na', 'stt': 'na', 'cas': 'na', 'per': 'na', 'rat': 'na'} 
+                ):
+                    stem_feats['prc1'] = 'li_prep'
+            
+            r13_fixes(token, stem_feats)
+            
+            clitic_feats_list.append(get_clitic_feats(token.replace('+', ''), clitic_order, clitic_feats, stem_feats))
+
+    feats_df = pd.DataFrame(clitic_feats_list)
+    assert tokens_df.shape[0] == feats_df.shape[0], f'token-feature mismatch!,\ntokens: \n{tokens_df},\n\n features: \n{feats_df}'
+    
+    return pd.concat([tokens_df, feats_df], axis=1)
+
+def get_lemmas(lemma, tokens):
+    lemmas = []
+    for token in tokens:
+        if '+' in token:
+            lemmas.append(token)
+        else:
+            lemmas.append(lemma)
+    return lemmas
+
+def get_main_features_df(word_analysis):
+    # if there are no clitics
+    if '+' not in word_analysis['catib6']:
+        tokens = [word_analysis['atbtok']]
+        catib6 = [word_analysis['catib6']]
+        ud = [word_analysis['ud']]
+        lemmas = [word_analysis['lex']]
+    else:    
+        tokens = word_analysis['atbtok'].split('_')
+        catib6 = word_analysis['catib6'].split('+')
+        ud = word_analysis['ud'].split('+')
+        lemmas = get_lemmas(word_analysis['lex'], tokens)
+    
+    if len(catib6) < len(tokens):
+        print(tokens)
+        print(catib6)
+        catib6.append("NOM")
+        ud.append("NOUN")
+        return pd.DataFrame({'token': tokens, 'catib6': catib6, 'ud': ud, 'lemma': lemmas})
+    elif len(tokens) < len(catib6):
+        return pd.DataFrame({'token': tokens, 'catib6': catib6[:len(tokens)], 'ud': ud[:len(tokens)], 'lemma': lemmas})
+
+    return pd.DataFrame({'token': tokens, 'catib6': catib6, 'ud': ud, 'lemma': lemmas})
+
+def get_word_features_df(word_analysis, clitic_feats):
+    """if a word is composed of multiple tokens, return them all.
+    otherwise, just return the word and catib6 tag
+
+    Args:
+        word_analysis (dict): analysis generated from a cameltools disambiguator
+
+    Returns:
+        List[tuple]: a list of one or more tokens
+    """
+    # gets forms, tokens, feats, ...
+    main_feats = get_main_features_df(word_analysis)
+    stem_feats = get_stem_feats(word_analysis)
+    return add_remaining_features(main_feats, stem_feats, clitic_feats)
+
+def join_feats(word_feats_df, tagset):
+    word_features = {
+        'tokens': list(word_feats_df['token']),
+        'pos_tags': list(word_feats_df[tagset]),
+        'lemmas': list(word_feats_df['lemma']),
+    }
+    
+    word_feats_df.drop(['token', tagset, 'lemma'], axis=1, inplace=True)
+    feats = word_feats_df.to_dict('records')
+    word_features['feats'] = [feats_dict_to_string(row) for row in feats]
+
+    return word_features
+
+def update_sentence_features(sentence_features, word_features):
+    sentence_features['tokens'] += word_features['tokens']
+    sentence_features['lemmas'] += word_features['lemmas']
+    sentence_features['pos_tags'] += word_features['pos_tags']
+    sentence_features['feats'] += word_features['feats']
+    
+    return sentence_features
+
+def build_token_list(sentence_features):
+    return [
+        (idx, dediac_ar(token), lemma, pos_tag, '_', feats, '_', '_', '_', '_')
+        for idx, (token, lemma, pos_tag, feats) in enumerate(
+            zip(sentence_features['tokens'], sentence_features['lemmas'], sentence_features['pos_tags'], sentence_features['feats'])
+        , 1)
+    ]
+
+def to_conll_fields_list(sentence_analysis_list: List[List[dict]], clitic_feats, tagset):
+    sentence_features_list = []
+    
+    for sentence_analysis in sentence_analysis_list:
+        sentence_features = {'tokens': [], 'lemmas': [], 'pos_tags': [], 'feats': []}
+        for word_analysis in sentence_analysis:
+            word_features_df = get_word_features_df(word_analysis, clitic_feats)
+            
+            word_features = join_feats(word_features_df, tagset)
+            sentence_features = update_sentence_features(sentence_features, word_features)
+        token_list = build_token_list(sentence_features)
+        sentence_features_list.append(token_list)
+    
+    return sentence_features_list

src/utils/conll_fixes.py

@@ -0,0 +1,33 @@
+"""The parsing code requires that conll files have two newlines at the end. 
+The functions below ensure that this is the case before parsing"""
+
+def read_n_to_last_line(filename, n = 1):
+    """Returns the nth before last line of a file (n=1 gives last line)
+    
+    from https://stackoverflow.com/questions/46258499/how-to-read-the-last-line-of-a-file-in-python"""
+    num_newlines = 0
+    with open(filename, 'rb') as f:
+        try:
+            for i in range(1, n+1):
+                f.seek(-i, 2)    
+                    # while num_newlines < n:
+                    #     f.seek(-2, os.SEEK_CUR)
+                    #     if f.read(1) == b'\n':
+                    #         num_newlines += 1
+        except OSError:
+            f.seek(0)
+        last_line = f.readline().decode()
+    return last_line
+
+def add_newlines(file_path, second_last_line, last_line):
+    with open(file_path, 'a') as f:
+        if second_last_line != '\n':
+            f.write('\n')
+        if last_line != '\n':
+            f.write('\n')
+
+def adjust_eof_newlines(file_path):
+    second_last_line = read_n_to_last_line(file_path, 2)
+    last_line = read_n_to_last_line(file_path, 1)
+    if not (second_last_line == last_line == '\n'):
+        add_newlines(file_path, second_last_line, last_line)

src/utils/model_downloader.py

@@ -0,0 +1,33 @@
+""" Script used to set up different parts of the text_to_conll_cli.py"""
+
+import os
+from pathlib import Path
+from huggingface_hub import hf_hub_download
+
+def download_default_models(model_path: Path) -> None:
+    # check if models folder exists
+    if not os.path.exists(model_path):
+        os.mkdir(model_path)
+
+    # check if default models exist
+    if not os.path.exists(model_path / "CAMeLBERT-CATiB-biaffine.model"):
+        print('downloading catib model')
+        hf_hub_download(repo_id="CAMeL-Lab/camelbert-catib-parser", filename="CAMeLBERT-CATiB-biaffine.model", local_dir=model_path)
+    if not os.path.exists(model_path / "CAMeLBERT-UD-biaffine.model"):
+        print('downloading ud model')
+        hf_hub_download(repo_id="CAMeL-Lab/camelbert-ud-parser", filename="CAMeLBERT-UD-biaffine.model", local_dir=model_path)
+    print('Default models downloaded.')
+
+def get_model_name(parse_model: str, model_path: Path) -> str:
+    if parse_model == "catib":
+        if os.path.exists(model_path / "CAMeLBERT-CATiB-biaffine.model"):
+            return "CAMeLBERT-CATiB-biaffine.model"
+        else:
+            assert False, "CATiB default model not found. Did you run 'python download_models'?"
+    elif parse_model == "ud":
+        if os.path.exists(model_path / "CAMeLBERT-UD-biaffine.model"):
+            return "CAMeLBERT-UD-biaffine.model"
+        else:
+            assert False, "UD default model not found. Did you run 'python download_models'?"
+    else:
+        return parse_model

src/utils/text_cleaner.py

@@ -0,0 +1,34 @@
+from camel_tools.tokenizers.word import simple_word_tokenize
+from camel_tools.utils.dediac import dediac_ar
+from camel_tools.utils.normalize import normalize_unicode
+from camel_tools.utils.charmap import CharMapper
+
+def clean_line(line, arclean):
+    return simple_word_tokenize(arclean(dediac_ar(normalize_unicode(line.strip()))))
+
+def split_lines_words(lines):
+    return [line.strip().split() for line in lines]
+
+def clean_mad(lines):
+    new_lines = []
+    for line in lines:
+        new_line = []
+        for word in line:
+            # replacing underscore with mad character
+            if word == 'ـ':
+                new_line.append('_')
+            else:
+                new_line.append(word)
+        new_lines.append(new_line)
+    return new_lines
+
+def clean_lines(lines, arclean):
+    return [clean_line(line, arclean) for line in lines]
+
+if __name__ == '__main__':
+    lines = []
+    with open('data/sample_text.txt', 'r') as f:
+        lines = f.readlines()
+    
+    arclean = CharMapper.builtin_mapper("arclean")
+    new_lines = clean_lines(lines, arclean)

tests/test_feats.tsv

@@ -0,0 +1,3 @@
+token	catib6	ud	lemma	prc3	prc2	prc1	prc0	per	asp	vox	mod	gen	num	stt	cas	enc0	rat
+بِ+	PRT	ADP	بِ+	0	0	0	na	na	na	na	na	na	na	na	na	0	na
+اِسْمِ	NOM	NOUN	ٱِسْم	0	0	bi_prep	0	na	na	na	na	m	s	c	g	0	i

tests/test_feature_extraction.py

@@ -0,0 +1,55 @@
+import pytest
+from pandas import read_csv
+
+from src.parse_disambiguation.feature_extraction import get_word_features_df, join_feats
+
+@pytest.fixture
+def word_analysis():
+    return {'diac': 'بِسْمِ', 'lex': 'ٱِسْم', 'caphi': 'b_i_s_m_i', 
+        'gloss': 'in/by_+_(the)_Name_of_+_[def.gen.]', 
+        'bw': 'بِ/PREP+ٱِسْم/NOUN+ِ/CASE_DEF_GEN', 'pos': 'noun', 
+        'catib6': 'PRT+NOM', 'ud': 'ADP+NOUN', 'root': 'س.م.#', 
+        'pattern': 'بِ1ْ2ِ', 'prc3': '0', 'prc2': '0', 'prc1': 
+        'bi_prep', 'prc0': '0', 'per': 'na', 'asp': 'na', 
+        'vox': 'na', 'mod': 'na', 'form_gen': 'm', 'gen': 'm', 
+        'form_num': 's', 'num': 's', 'stt': 'c', 'cas': 'g', 
+        'enc0': '0', 'rat': 'i', 'source': 'lex', 'd1seg': 'بِسْمِ', 
+        'd2seg': 'بِ+_سْمِ', 'd3seg': 'بِ+_سْمِ', 'atbseg': 'بِ+_سْمِ', 
+        'd1tok': 'بِسْمِ', 'd2tok': 'بِ+_اِسْمِ', 'd3tok': 'بِ+_اِسْمِ', 
+        'atbtok': 'بِ+_اِسْمِ', 'bwtok': 'بِ+_ٱِسْم_+ِ', 'pos_logprob': -0.4344233, 
+        'lex_logprob': -3.156274, 'pos_lex_logprob': -3.156274, 
+        'stem': 'بِسْمِ', 'stemgloss': 'in/by_+_(the)_Name_of_+_[def.gen.]', 'stemcat': 'FW-Wa'
+    }
+
+@pytest.fixture
+def clitic_feats():
+    return read_csv('data/clitic_feats.csv')
+
+@pytest.fixture
+def word_feats():
+    df = read_csv('tests/test_feats.tsv', sep='\t')
+    return df.astype(str).astype(object)
+    
+
+def test_get_word_features_df(word_analysis, clitic_feats, word_feats):
+    tagset = "catib6"
+    
+    df = get_word_features_df(word_analysis, clitic_feats)
+    df = df.astype(str).astype(object)
+    assert word_feats.equals(df)
+
+def test_join_feats_catib6(word_feats):
+    word_feats = join_feats(word_feats, 'catib6')
+    
+    assert word_feats['tokens'] == ['بِ+', 'اِسْمِ']
+    assert word_feats['pos_tags'] == ['PRT', 'NOM']
+    assert word_feats['lemmas'] == ['بِ+', 'ٱِسْم']
+    assert word_feats['feats'] == ['ud=ADP|prc3=0|prc2=0|prc1=0|prc0=na|per=na|asp=na|vox=na|mod=na|gen=na|num=na|stt=na|cas=na|enc0=0|rat=na', 'ud=NOUN|prc3=0|prc2=0|prc1=bi_prep|prc0=0|per=na|asp=na|vox=na|mod=na|gen=m|num=s|stt=c|cas=g|enc0=0|rat=i']
+
+def test_join_feats_ud(word_feats):
+    word_feats = join_feats(word_feats, 'ud')
+    
+    assert word_feats['tokens'] == ['بِ+', 'اِسْمِ']
+    assert word_feats['pos_tags'] == ['ADP', 'NOUN']
+    assert word_feats['lemmas'] == ['بِ+', 'ٱِسْم']
+    assert word_feats['feats'] == ['catib6=PRT|prc3=0|prc2=0|prc1=0|prc0=na|per=na|asp=na|vox=na|mod=na|gen=na|num=na|stt=na|cas=na|enc0=0|rat=na', 'catib6=NOM|prc3=0|prc2=0|prc1=bi_prep|prc0=0|per=na|asp=na|vox=na|mod=na|gen=m|num=s|stt=c|cas=g|enc0=0|rat=i']

text_to_conll_cli.py

@@ -0,0 +1,110 @@
+"""
+Disambiguator and Conll builder CLI.
+
+Usage:
+    text_to_conll_cli (-i <input> | --input=<input> | -s <string> | --string=<string>)
+        (-f <file_type> | --file_type=<file_type>)
+        [-b <morphology_db_type> | --morphology_db_type=<morphology_db_type>]
+        [-d <disambiguator> | --disambiguator=<disambiguator>]
+        [-m <model> | --model=<model>]
+    text_to_conll_cli (-h | --help)
+
+Options:
+    -i <input> --input=<input>
+        A text file or conll file.
+    -s <string> --string=<string>
+        A string to parse.
+    -f <file_type> --file_type=<file_type>
+        The type of file passed. Could be 
+            conll: conll
+            text: raw text
+            preprocessed_text: whitespace tokenized text (text will not be cleaned)
+            tokenized_tagged: text is already tokenized and POS tagged, in tuple form
+            tokenized: text is already tokenized, only parse tokenized input; don't disambiguate to add POS tags or features
+    -b <morphology_db_type> --morphology_db_type=<morphology_db_type>
+        The morphology database to use; will use camel_tools built-in by default [default: r13]
+    -d <disambiguator> --disambiguator=<disambiguator>
+        The disambiguation technique used to tokenize the text lines, either 'mle' or 'bert' [default: bert]
+    -m <model> --model=<model>
+        The name BERT model used to parse (to be placed in the model directory) [default: catib]
+    -h --help
+        Show this screen.
+"""
+
+from src.logger import log
+from pathlib import Path
+from camel_tools.utils.charmap import CharMapper
+from src.conll_output import print_to_conll, text_tuples_to_string
+from src.data_preparation import get_file_type_params, get_tagset, parse_text
+from src.utils.model_downloader import get_model_name
+from docopt import docopt
+from transformers.utils import logging
+from pandas import read_csv
+
+arguments = docopt(__doc__)
+
+logging.set_verbosity_error()
+
+def get_file_type(file_type):
+    if file_type in ['conll', 'text', 'preprocessed_text', 'tokenized_tagged', 'tokenized']:
+        return file_type 
+    assert False, 'Unknown file type'
+
+@log
+def main():
+    root_dir = Path(__file__).parent
+    model_path = root_dir/"models"
+    
+    # camel_tools import used to clean text
+    arclean = CharMapper.builtin_mapper("arclean")
+
+    #
+    ### Get clitic features
+    #
+    clitic_feats_df = read_csv(root_dir / 'data/clitic_feats.csv')
+    clitic_feats_df = clitic_feats_df.astype(str).astype(object) # so ints read are treated as string objects
+    
+
+    #
+    ### cli user input ###
+    #
+    file_path = arguments['--input']
+    string_text = arguments['--string']
+    file_type = get_file_type(arguments['--file_type'])
+    morphology_db_type = arguments['--morphology_db_type']
+    disambiguator_type = arguments['--disambiguator']
+    parse_model = arguments['--model']
+
+
+    #
+    ### Set up parsing model 
+    # (download defaults models, and get correct model name from the models directory)
+    #
+    model_name = get_model_name(parse_model, model_path=model_path)
+
+    # 
+    ### get tagset (depends on model)
+    #
+    tagset = get_tagset(parse_model)
+    
+    
+    #
+    ### main code ###
+    #
+    lines = []
+    if string_text is not None:
+        lines = [string_text]
+    elif file_path is not None:
+        with open(file_path, 'r') as f:
+            lines = [line for line in f.readlines() if line.strip()]
+
+
+    file_type_params = get_file_type_params(lines, file_type, file_path, model_path/model_name,
+        arclean, disambiguator_type, clitic_feats_df, tagset, morphology_db_type)
+    parsed_text_tuples = parse_text(file_type, file_type_params)
+
+    string_lines = text_tuples_to_string(parsed_text_tuples, file_type, sentences=lines)
+    print_to_conll(string_lines)
+
+if __name__ == '__main__':
+    main()