Duplicate from AnReu/math_albert

Co-authored-by: Anja <AnReu@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,33 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,44 @@
+---
+language: 
+  - en
+tags:
+- mathematics
+- math-aware
+datasets:
+- MathematicalStackExchange
+---
+
+# Math-aware ALBERT
+
+This repository contains our best *base* model for ARQMath 3. It was initialised from **ALBERT-base-v2** and further pre-trained on **Math StackExchange** in three different stages. We also added more **LaTeX tokens** to the tokenizer to enable a better tokenization of mathematical formulas. This model is not yet fine-tuned on a specific task. If you are looking for the **fine-tuned model**, please refer to this page: [AnReu/albert-for-arqmath-3](https://huggingface.co/AnReu/albert-for-arqmath-3)
+
+If you are looking for a **Math-pre-trained BERT** model: Check out our [AnReu/math_pretrained_bert](https://huggingface.co/AnReu/math_pretrained_bert) which is trained in the same way as this model here.
+
+# Training Details
+
+The model was instantiated from ALBERT-base-v2 weights and further pre-trained in three stages using different data for the sentence order prediction. During all three stages, the mask language modelling task was trained simultaneously. In addition, we added around 500 LaTeX tokens to the tokenizer to better cope with mathematical formulas. 
+
+The image illustrates the three pre-training stages: First, we train on mathematical formulas only. The SOP classifier predicts which segment contains the left hand side of the formula and which one contains the right hand side. This way we model inter-formula-coherence. The second stages models formula-sentence-coherence, i.e., whether the formula comes first in the original document or whether the natural language part comes first. Finally, we add the inter-sentence-coherence stage that is default for ALBERT. In this stage, sentences were split by a sentence separator.
+
+![Image](https://huggingface.co/AnReu/math_albert/resolve/main/Screenshot%202022-09-02%20at%2018.06.04.png)
+
+For further details, please read our paper: http://ceur-ws.org/Vol-3180/paper-07.pdf. 
+
+
+
+# Usage
+
+You can use this model to further fine-tune it on any math-aware task you have in mind, e.g., classification, question-answering, etc. . Please note, that the model in this repository is only pre-trained and not fine-tuned. If you are looking for the fine-tuned model, please refer to this page: [AnReu/albert-for-arqmath-3](https://huggingface.co/AnReu/albert-for-arqmath-3)
+
+
+# Citation
+
+If you find this model useful, consider citing our paper:
+```
+@article{reusch2022transformer,
+  title={Transformer-Encoder and Decoder Models for Questions on Math},
+  author={Reusch, Anja and Thiele, Maik and Lehner, Wolfgang},
+  year={2022},
+  organization={CLEF}
+}
+```

added_tokens.json

@@ -0,0 +1 @@
+{"\\Vert": 30047, "\\mid": 30266, "\\sinh": 30379, "\\angle": 30054, "\\Gamma": 30012, "\\coth": 30109, "\\bowtie": 30089, "\\tfrac": 30420, "\\injlim": 30201, "\\sqrt": 30387, "\\signum": 30375, "}": 30494, "\\lor": 30254, "\\xrightarrow": 30483, "\\bmod": 30087, "\\gnapprox": 30174, "\\Leftrightarrow": 30018, "\\measuredangle": 30264, "\\supsetneq": 30411, "\\max": 30263, "\\alpha": 30052, "\\vee": 30476, "\\dotsc": 30136, "\\varepsilon": 30454, "\\tbinom": 30417, "\\Longleftarrow": 30020, "\\lambda": 30209, "\\Uparrow": 30042, "\\Lsh": 30023, "\\gtrless": 30183, "\\lbrace": 30212, "\\overrightarrow": 30319, "\\qquad": 30343, "\\preccurlyeq": 30332, "\\varpropto": 30463, "\\neq": 30280, "\\varpi": 30461, "\\land": 30210, "\\end": 30149, "\\varsubsetneqq": 30467, "\\bigcup": 30076, "\\multimap": 30271, "\\bigwedge": 30085, "\\not": 30295, "\\mapsto": 30259, "\\Sigma": 30037, "\\cot": 30108, "\\rVert": 30346, "\\supseteqq": 30410, "\\downdownarrows": 30143, "\\curvearrowleft": 30117, "\\ltimes": 30256, "\\preceq": 30333, "\\nwarrow": 30306, "\\eth": 30158, "align": 30500, "\\Leftarrow": 30017, "\\Longleftrightarrow": 30021, "\\bigotimes": 30079, "\\rightarrowtail": 30356, "\\sqsubset": 30388, "\\Beta": 30003, "\\looparrowright": 30253, "\\varGamma": 30446, "\\aleph": 30051, "\\nleftarrow": 30288, "\\varlimsup": 30458, "\\Kappa": 30015, "\\precapprox": 30331, "\\ln": 30242, "\\equiv": 30156, "\\iiint": 30193, "\\Delta": 30007, "\\bigodot": 30077, "\\vartriangleleft": 30472, "\\succapprox": 30399, "\\triangleq": 30431, "\\nsucceq": 30303, "\\lfloor": 30234, "\\Omicron": 30027, "\\nparallel": 30296, "\\dots": 30134, "\\rbrack": 30349, ">": 30490, "\\otimes": 30315, "\\mathcal": 30261, "\\mod": 30268, "\\ddddot": 30121, "\\jmath": 30205, "\\varsupsetneqq": 30469, "\\dddot": 30122, "\\varliminf": 30457, "vmatrix": 30515, "\\between": 30072, "\\bigvee": 30084, "\\Updownarrow": 30043, "\\searrow": 30369, "\\gg": 30171, "\\overline": 30318, "\\omega": 30310, "\\imath": 30195, "\\mathrm": 30262, "\\nRightarrow": 30274, "\\in": 30198, "\\gcd": 30165, "\\precnapprox": 30334, "\\smallint": 30381, "gather": 30508, "_": 30495, "\\eta": 30157, "\\Longrightarrow": 30022, "\\exp": 30160, "\\Eta": 30011, "\\underline": 30434, "\\leftharpoondown": 30221, "\\theta": 30421, "\\supsetneqq": 30412, "\\sqsupset": 30390, "\\propto": 30341, "\\looparrowleft": 30252, "\\impliedby": 30196, "\\leftrightarrows": 30225, "\\to": 30425, "\\hookleftarrow": 30187, "\\Rsh": 30036, "\\ominus": 30312, "\\above": 30050, "\\varDelta": 30445, "\\iiiint": 30192, "\\bigtriangledown": 30081, "\\succsim": 30405, "\\gtrapprox": 30179, "±": 30496, "\\lg": 30235, "\\ge": 30166, "\\inf": 30199, "\\arccos": 30057, "\\zeta": 30484, "\\triangledown": 30428, "\\brack": 30091, "\\backsim": 30066, "\\neg": 30279, "\\triangleright": 30432, "\\mp": 30269, "\\oint": 30309, "equation": 30507, "\\smallsmile": 30383, "\\nLeftrightarrow": 30273, "\\upharpoonright": 30441, "\\circlearrowleft": 30101, "\\VarOmega": 30046, "gathered": 30509, "\\chi": 30097, "matrix": 30510, "\\det": 30127, "\\nrightarrow": 30299, "\\backepsilon": 30065, "\\ngtr": 30286, "\\Cap": 30004, "\\leftharpoonup": 30222, "\\Upsilon": 30044, "\\geqq": 30168, "\\sim": 30376, "\\succneqq": 30403, "\\vec": 30475, "multline": 30511, "\\ngeqslant": 30285, "\\prec": 30330, "\\rightharpoonup": 30358, "\\varsubsetneq": 30466, "\\subseteqq": 30395, "\\ggg": 30172, "\\limsup": 30239, "\\nsubseteq": 30301, "\\sup": 30407, "\\implies": 30197, "\\Iota": 30013, "\\varTheta": 30451, "\\succcurlyeq": 30400, "\\nearrow": 30278, "\\begin": 30073, "\\bigsqcup": 30080, "\\qvar": 30345, "\\curlyeqprec": 30113, "\\vert": 30478, "bmatrix": 30504, "\\eqslantless": 30155, "\\Mu": 30024, "\\deg": 30125, "\\geqslant": 30169, "\\bigtriangleup": 30082, "\\rightleftarrows": 30359, "\\nabla": 30275, "\\underset": 30435, "\\rangle": 30347, "\\empty": 30147, "\\gtreqless": 30181, "\\projlim": 30340, "^": 30492, "\\quad": 30344, "\\nleqslant": 30292, "\\sin": 30378, "\\eqslantgtr": 30154, "\\xleftarrow": 30482, "\\mho": 30265, "\\subseteq": 30394, "\\gneqq": 30176, "\\tan": 30414, "\\curlyeqsucc": 30114, "\\beth": 30071, "\\sum": 30406, "\\dotsi": 30137, "\\Lleftarrow": 30019, "{": 30493, "\\backsimeq": 30067, "\\cdots": 30095, "\\prime": 30338, "\\cap": 30093, "\\text": 30418, "\\thicksim": 30423, "\\eqsim": 30153, "\\sqsubseteq": 30389, "\\rightsquigarrow": 30362, "\\rtimes": 30365, "\\psi": 30342, "\\Phi": 30028, "\\longleftrightarrow": 30249, "\\gvertneqq": 30185, "\\gets": 30170, "\\Downarrow": 30009, "\\omicron": 30311, "\\trianglerighteq": 30433, "\\leq": 30228, "\\sign": 30374, "\\hom": 30186, "\\rightrightarrows": 30361, "\\arg": 30060, "array": 30503, "\\rgroup": 30352, "\\vartriangle": 30471, "\\sqcup": 30386, "\\cosh": 30107, "\\lt": 30255, "\\VarLambda": 30045, "\\varprojlim": 30462, "\\forall": 30161, "\\rceil": 30350, "\\gt": 30178, "<": 30491, "\\cong": 30104, "\\root": 30364, "\\trianglelefteq": 30430, "\\geq": 30167, "\\lmoustache": 30241, "\\updownarrow": 30439, "\\lessapprox": 30231, "\\succnapprox": 30402, "\\Chi": 30005, "\\top": 30426, "\\supseteq": 30409, "\\ngeq": 30283, "\\leqslant": 30230, "\\odot": 30307, "\\ngeqq": 30284, "\\vdots": 30474, "\\of": 30308, "\\gtreqqless": 30182, "\\supset": 30408, "\\tanh": 30415, "\\Join": 30014, "\\precneqq": 30335, "\\ldots": 30215, "\\Epsilon": 30010, "\\sigma": 30373, "\\leftleftarrows": 30223, "\\sec": 30370, "\\Pi": 30029, "\\Lambda": 30016, "∞": 30497, "\\dot=": 30141, "\\delta": 30126, "\\ddot": 30123, "\\subsetneqq": 30397, "\\cos": 30106, "\\Cup": 30006, "eqnarray": 30506, "\\overleftarrow": 30317, "\\arctan": 30059, "\\over": 30316, ":=": 30489, "\\leftarrowtail": 30220, "\\perp": 30324, "\\kappa": 30206, "\\enspace": 30150, "\\tau": 30416, "\\Subset": 30038, "\\bot": 30088, "\\subset": 30393, "\\oplus": 30313, "\\setminus": 30371, "\\arrowvert": 30061, "\\gneq": 30175, "\\bigoplus": 30078, "\\downharpoonleft": 30144, "\\lceil": 30213, "\\thickapprox": 30422, "\\buildrel": 30092, "\\vartriangleright": 30473, "\\varnothing": 30459, "\\eqcirc": 30152, "\\curlywedge": 30116, "\\gtrsim": 30184, "\\intop": 30203, "\\ker": 30207, "\\ni": 30287, "\\varsigma": 30465, "\\dotsm": 30138, "\\varsupsetneq": 30468, "\\nLeftarrow": 30272, "\\nsim": 30300, "\\veebar": 30477, "\\scriptstyle": 30368, "\\rbrace": 30348, "\\choose": 30098, "\\log": 30247, "\\exists": 30159, "\\nsucc": 30302, "\\curvearrowright": 30118, "\\nless": 30293, "\\gnsim": 30177, "\\smallsetminus": 30382, "\\Re": 30032, "\\int": 30202, "\\leftrightarrow": 30224, "\\varXi": 30453, "\\dotsb": 30135, "\\owns": 30321, "\\iddots": 30189, "\\curlyvee": 30115, "\\lneqq": 30245, "\\sphericalangle": 30384, "\\varinjlim": 30455, "\\circeq": 30100, "\\upuparrows": 30444, "\\mu": 30270, "\\left": 30218, "\\lbrack": 30214, "\\unlhd": 30436, "\\biguplus": 30083, "\\sgn": 30372, "\\rho": 30353, "\\varPhi": 30447, "\\liminf": 30238, "\\Rho": 30033, "\\lVert": 30208, "\\varPi": 30448, "\\nexists": 30282, "\\lim": 30237, "\\iota": 30204, "\\iff": 30191, "\\Theta": 30041, "\\iint": 30194, "\\rmoustache": 30363, "\\dotplus": 30133, "\\And": 30001, "\\Tau": 30040, "\\succeq": 30401, "\\^": 30486, "\\dbinom": 30120, "\\csc": 30111, "\\cr": 30110, "\\varUpsilon": 30452, "\\right": 30354, "\\npreceq": 30298, "\\Rightarrow": 30034, "\\\\": 30488, "\\downharpoonright": 30145, "\\rightarrow": 30355, "\\arcsin": 30058, "\\succnsim": 30404, "\\Supset": 30039, "\\stackrel": 30392, "\\parallel": 30322, "\\prod": 30339, "\\nsupseteq": 30304, "\\bracevert": 30090, "\\operatorname": 30314, "\\scriptscriptstyle": 30367, "\\longrightarrow": 30251, "\\bigcirc": 30075, "\\dim": 30129, "\\lneq": 30244, "\\lvert": 30257, "\\infty": 30200, "\\ast": 30062, "\\unrhd": 30437, "\\frown": 30163, "\\Omega": 30026, "\\triangleleft": 30429, "\\nleftrightarrow": 30289, "\\longleftarrow": 30248, "\\emptyset": 30148, "\\nu": 30305, "\\lesssim": 30233, "\\doteq": 30140, "\\bigcap": 30074, "\\gggtr": 30173, "\\upharpoonleft": 30440, "\\colon": 30103, "\\frac": 30162, "Bmatrix": 30498, "\\%": 30485, "\\upsilon": 30443, "\\pi": 30326, "\\varSigma": 30450, "smallmatrix": 30513, "\\displaystyle": 30130, "\\subsetneq": 30396, "\\sqcap": 30385, "\\pm": 30327, "\\overset": 30320, "\\Xi": 30048, "\\epsilon": 30151, "\\leftrightsquigarrow": 30227, "\\simeq": 30377, "\\wedge": 30479, "\\approxeq": 30056, "\\Rrightarrow": 30035, "\\precsim": 30337, "\\vartheta": 30470, "\\precnsim": 30336, "\\ddots": 30124, "\\ncong": 30276, "\\dotso": 30139, "\\nleq": 30290, "\\wr": 30480, "\\uparrow": 30438, "\\xi": 30481, "\\rightharpoondown": 30357, "\\cup": 30112, "\\longmapsto": 30250, "\\lnapprox": 30243, "\\varPsi": 30449, "\\gtrdot": 30180, "\\asymp": 30063, "\\cfrac": 30096, "\\Nu": 30025, "\\ell": 30146, "\\triangle": 30427, "\\leqq": 30229, "\\downarrow": 30142, "\\swarrow": 30413, "\\min": 30267, "\\approx": 30055, "alignat": 30501, "\\lessdot": 30232, "\\varkappa": 30456, "alignedat": 30502, "pmatrix": 30512, "\\atop": 30064, "\\Alpha": 30000, "\\leadsto": 30217, "\\ne": 30277, "\\amalg": 30053, "\\ll": 30240, "\\leftrightharpoons": 30226, "\\circ": 30099, "\\divideontimes": 30132, "\\textstyle": 30419, "\\Pr": 30030, "\\dashv": 30119, "\\pod": 30329, "\\Zeta": 30049, "\\rfloor": 30351, "\\div": 30131, "\\Arrowvert": 30002, "\\sqsupseteq": 30391, "\\uplus": 30442, "\\le": 30216, "\\lvertneqq": 30258, "\\mathbb": 30260, "cases": 30505, "\\succ": 30398, "\\Doteq": 30008, "\\lgroup": 30236, "\\gamma": 30164, "\\cdot": 30094, "\\varphi": 30460, "subarray": 30514, "\\smallfrown": 30380, "\\phi": 30325, "\\circlearrowright": 30102, "\\rightleftharpoons": 30360, "\\beta": 30070, "\\leftarrow": 30219, "\\Psi": 30031, "\\langle": 30211, "\\varrho": 30464, "\\times": 30424, "\\pmod": 30328, "\\rvert": 30366, "\\backslash": 30068, "\\nleqq": 30291, "Vmatrix": 30499, "\\coprod": 30105, "\\partial": 30323, "\\binom": 30086, "\\idotsint": 30190, "\\newline": 30281, "\\nprec": 30297, "\\hookrightarrow": 30188, "\\lnot": 30246, "\\dfrac": 30128, "\\barwedge": 30069, "\\nmid": 30294, "\\": 30487}

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "albert-base-v2",
+  "architectures": [
+    "AlbertModel"
+  ],
+  "attention_probs_dropout_prob": 0,
+  "bos_token_id": 2,
+  "classifier_dropout_prob": 0.1,
+  "down_scale_factor": 1,
+  "embedding_size": 128,
+  "eos_token_id": 3,
+  "gap_size": 0,
+  "hidden_act": "gelu_new",
+  "hidden_dropout_prob": 0,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "inner_group_num": 1,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "albert",
+  "net_structure_type": 0,
+  "num_attention_heads": 12,
+  "num_hidden_groups": 1,
+  "num_hidden_layers": 12,
+  "num_memory_blocks": 0,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.6.1",
+  "type_vocab_size": 2,
+  "vocab_size": 30516
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03c88776fa528d0b92d7d83734db35b63def53e399883ab861ba6d3768a0344b
+size 47529600

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6ef3aa1d575f97050c86eabea16de6da34b0a5e6dd576b6465cd84486581957
+size 47538562

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "[CLS]", "eos_token": "[SEP]", "unk_token": "<unk>", "sep_token": "[SEP]", "pad_token": "<pad>", "cls_token": "[CLS]", "mask_token": {"content": "[MASK]", "single_word": false, "lstrip": true, "rstrip": false, "normalized": false}}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fefb02b667a6c5c2fe27602d28e5fb3428f66ab89c7d6f388e7c8d44a02d0336
+size 760289

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "remove_space": true, "keep_accents": false, "bos_token": "[CLS]", "eos_token": "[SEP]", "unk_token": "<unk>", "sep_token": "[SEP]", "pad_token": "<pad>", "cls_token": "[CLS]", "mask_token": {"content": "[MASK]", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "albert-base-v2"}