Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1,51 @@
 *.7z filter=lfs diff=lfs merge=lfs -text
 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
 *.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
 *.ftz filter=lfs diff=lfs merge=lfs -text
 *.gz filter=lfs diff=lfs merge=lfs -text
 *.h5 filter=lfs diff=lfs merge=lfs -text
 *.joblib filter=lfs diff=lfs merge=lfs -text
 *.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
 *.model filter=lfs diff=lfs merge=lfs -text
 *.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
 *.onnx filter=lfs diff=lfs merge=lfs -text
 *.ot filter=lfs diff=lfs merge=lfs -text
 *.parquet filter=lfs diff=lfs merge=lfs -text
 *.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
 *.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
 saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
 *.tflite filter=lfs diff=lfs merge=lfs -text
 *.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
 *.xz filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*.tfevents* filter=lfs diff=lfs merge=lfs -text
+*.db* filter=lfs diff=lfs merge=lfs -text
+*.ark* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.gguf* filter=lfs diff=lfs merge=lfs -text
+*.ggml filter=lfs diff=lfs merge=lfs -text
+*.llamafile* filter=lfs diff=lfs merge=lfs -text
+*.pt2 filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -textconf/Figures/UniKP.jpg filter=lfs diff=lfs merge=lfs -text
+conf/Figures/UniKP.png filter=lfs diff=lfs merge=lfs -text
+conf/datasets/Generated_pH_unified_smiles_636.xlsx filter=lfs diff=lfs merge=lfs -text
+conf/datasets/Kcat_combination_0918_wildtype_mutant.json filter=lfs diff=lfs merge=lfs -text
+conf/datasets/kcat_km_samples.xlsx filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,282 @@
+---
+license: gpl-3.0
+language:
+- en
+- zh
+tags:
+- OneScience
+- life-science
+- enzyme
+- enzyme-kinetics
+- UniKP
+frameworks: PyTorch
+---
+
+<p align="center">
+  <strong><span style="font-size: 30px;">UniKP</span></strong>
+</p>
+
+# Model Introduction
+
+UniKP is a unified framework for predicting enzyme kinetic parameters based on pretrained language models. Given a protein sequence and substrate structure, it can predict the enzyme turnover number $k_{cat}$, Michaelis constant $K_m$, and catalytic efficiency $k_{cat}/K_m$. UniKP uses a protein language model to extract enzyme sequence representations and combines them with substrate representations generated by a molecular language model to predict kinetic parameters.
+
+Paper:
+
+> **UniKP: a unified framework for the prediction of enzyme kinetic parameters**  
+> https://doi.org/10.1038/s41467-023-44113-1
+
+# Model Description
+
+UniKP takes the enzyme protein sequence and substrate SMILES as two separate inputs. On the protein side, ProtT5-XL-UniRef50 is used to extract sequence representations, while on the substrate side, a SMILES Transformer is used to generate molecular representations. The two feature representations are concatenated and then fed into trained regression models to predict $k_{cat}$, $K_m$, and $k_{cat}/K_m$, respectively.
+
+# Use Cases
+
+| Scenario | Description |
+| --- | --- |
+| Enzyme turnover number prediction | Predict $k_{cat}$ from a protein sequence and substrate SMILES |
+| Michaelis constant prediction | Predict $K_m$ for an enzyme-substrate system |
+| Catalytic efficiency prediction | Predict $k_{cat}/K_m$ |
+| Enzyme discovery and screening | Predict and rank kinetic parameters for candidate enzyme sequences |
+| Enzyme directed evolution | Compare predicted kinetic parameters of wild-type and mutant candidates |
+| Environmental factor analysis | Use EF-UniKP-related implementations to investigate the effects of temperature and pH on $k_{cat}$ |
+
+# Usage
+
+## 1. Using OneCode
+
+You can use the OneCode online environment for intelligent one-click AI4S programming:
+
+[Try intelligent one-click AI4S programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)
+
+## 2. Manual Installation and Usage
+
+**Hardware Requirements**
+
+- ProtT5-XL-UniRef50 is relatively large, so GPU/DCU acceleration is recommended for protein representation extraction.
+- A CPU can be used for a small number of short sequences, but it will be significantly slower than an accelerator.
+- For batch prediction or long-sequence tasks, it is recommended to reduce the batch size and adjust it according to the available device memory.
+
+### Environment Setup
+
+#### DCU Environment
+
+```bash
+# Activate DTK and CONDA first
+conda create -n onescience311 python=3.11 -y
+conda activate onescience311
+
+pip install onescience[bio] \
+  -i http://mirrors.onescience.ai:3141/pypi/simple/ \
+  --trusted-host mirrors.onescience.ai
+```
+
+#### Environment Notes
+- If you encounter missing dependencies or version incompatibilities during execution, refer to the dependency versions specified in `requirements.txt` and install or adjust the corresponding packages as needed.
+
+### Model Weights
+
+Complete UniKP inference requires three types of model resources:
+
+1. ProtT5-XL-UniRef50;
+2. UniKP regression models for $k_{cat}$, $K_m$, and $k_{cat}/K_m$;
+3. The vocabulary and pretrained weights of the SMILES Transformer.
+
+#### 1) ProtT5-XL-UniRef50
+
+ProtT5-XL-UniRef50 needs to be downloaded separately:
+
+```text
+https://zenodo.org/records/4644188
+```
+
+It is recommended to place it as follows:
+
+```text
+UniKP/
+└── weight/
+    └── prot_t5_xl_uniref50/
+```
+
+The current code will first try to use:
+
+```python
+T5Tokenizer.from_pretrained("weight/prot_t5_xl_uniref50")
+T5EncoderModel.from_pretrained("weight/prot_t5_xl_uniref50")
+```
+
+If `weight/prot_t5_xl_uniref50` does not exist, the code will fall back to `"prot_t5_xl_uniref50"`. You can also adjust `scripts/project_paths.py` according to the actual local path.
+
+#### 2) UniKP Regression Models
+
+The official README provides download links for the models corresponding to the three tasks:
+
+```text
+https://huggingface.co/HanselYu/UniKP/tree/main
+```
+
+Typical files include:
+
+```text
+UniKP for kcat.pkl
+UniKP for Km.pkl
+UniKP for kcat_Km.pkl
+```
+
+- These files are included in `weight/UniKP_model`.
+
+## 3. Quick Start
+
+### Download the Model Package
+
+```bash
+hf download OneScience-Group/UniKP --local-dir ./UniKP
+cd UniKP-main
+```
+
+- UniKP additionally depends on **ProtT5-XL-UniRef50** and the three types of **UniKP regression model weights**. Please prepare the required models according to the "Model Weights" section before proceeding.
+- The SMILES Transformer-related code is located in `model/`. Before running, make sure that `weight/vocab.pkl` and `weight/trfm_12_23000.pkl` exist.
+
+### Quick Verification
+
+Check whether ProtT5 can be loaded locally:
+
+```bash
+python - <<'PY'
+from transformers import T5Tokenizer, T5EncoderModel
+path = "./weight/prot_t5_xl_uniref50"
+T5Tokenizer.from_pretrained(path, do_lower_case=False)
+T5EncoderModel.from_pretrained(path)
+print("ProtT5 load OK")
+PY
+```
+
+Check the regression model and SMILES Transformer resources:
+
+```bash
+ls -lh weight/UniKP_model/
+ls -lh weight/vocab.pkl weight/trfm_12_23000.pkl
+```
+If all of the above checks pass, you can continue with the single-sample inference script:
+
+```bash
+python scripts/demo_kcat.py
+```
+
+# Example Data
+
+The core inputs for UniKP inference are:
+
+```text
+Protein amino acid sequence
++
+Substrate SMILES
+```
+
+Example:
+
+| Input | Example |
+| --- | --- |
+| Protein sequence | `MSELMKLSAV...MAQR` |
+| Substrate SMILES | `CC(O)O` |
+
+The corresponding output can be:
+
+```text
+kcat
+Km
+kcat / Km
+```
+
+# Inference Examples
+
+## Single-Sample kcat Prediction
+
+Run the following command from the UniKP root directory:
+
+```bash
+python scripts/demo_kcat.py
+```
+
+This script reads an example protein sequence and substrate SMILES, extracts their representations using ProtT5-XL-UniRef50 and the SMILES Transformer, respectively, concatenates the representations, and then loads `weight/UniKP_model/UniKP for kcat.pkl` to perform kcat prediction.
+
+The prediction result is printed to the terminal and saved as:
+
+```text
+UniKP_kcat_prediction.xlsx
+```
+
+By default, `demo_kcat.py` loads the kcat regression model:
+
+```python
+with open("weight/UniKP_model/UniKP for kcat.pkl", "rb") as f:
+    model = pickle.load(f)
+```
+
+To predict **Km**, replace the model path above with:
+
+```python
+with open("weight/UniKP_model/UniKP for Km.pkl", "rb") as f:
+    model = pickle.load(f)
+```
+
+To predict **kcat/Km**, replace it with:
+
+```python
+with open("weight/UniKP_model/UniKP for kcat_Km.pkl", "rb") as f:
+    model = pickle.load(f)
+```
+
+Except for the regression model, the protein representation extraction, SMILES representation extraction, feature concatenation, and prediction workflow remain unchanged. The model outputs are in `log10` space, and the script converts them back to the actual kinetic parameter values using `10 ** x`.
+
+## Batch Prediction
+
+The repository provides separate batch prediction scripts for kcat, Km, and kcat/Km:
+
+```bash
+python scripts/UniKP_kcat.py
+python scripts/UniKP_Km.py
+python scripts/UniKP_kcat_Km.py
+```
+
+The three scripts read the corresponding task data files and load the matching UniKP regression models to perform batch prediction on multiple protein sequences and substrate entries.
+
+Before running, make sure that the data files for the corresponding task are located in `conf/datasets/`.
+
+
+# Output Description
+
+UniKP outputs the predicted values of the corresponding enzyme kinetic parameters.
+
+| Parameter | Unit Used in the Official Example |
+| --- | --- |
+| $k_{cat}$ | s⁻¹ |
+| $K_m$ | mM |
+| $k_{cat}/K_m$ | s⁻¹·mM⁻¹ |
+
+The internal predictions of the UniKP regression models are in `log10` space, so the output of `model.predict()` cannot be treated directly as the actual kinetic parameter values. Use:
+
+```python
+pred = model.predict(fused_vector)
+pred_real = [10 ** x for x in pred]
+```
+
+The official example saves the results as:
+
+```text
+Kinetic_parameters_predicted_label.xlsx
+```
+
+# OneScience Official Information
+
+| Platform | OneScience Main Repository | Skills Repository |
+| --- | --- | --- |
+| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
+| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |
+
+
+# Citation and License
+
+- Original UniKP paper: [UniKP: a unified framework for the prediction of enzyme kinetic parameters](https://doi.org/10.1038/s41467-023-44113-1).
+- UniKP is licensed under the GNU General Public License version 3 (GPL-3.0).
+- UniKP inference depends on third-party models and code such as ProtT5-XL-UniRef50 and the SMILES Transformer. Their respective licenses and terms of use must also be followed when using, modifying, or redistributing these resources.
+- For research use, it is recommended to cite the original UniKP paper. If ProtT5 or the SMILES Transformer is used to generate representations, the corresponding projects should also be cited as required.

conf/datasets/Generated_pH_unified_smiles_636.xlsx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3aaee418b09a2a7ff952f015d4af544121f376bd5a5ab2486cdfe8faafc011b
+size 100403

conf/datasets/Kcat_combination_0918_wildtype_mutant.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13643b0b36374f8d3f64d8b014882cf1b3b58946eeaae2b9dcd59e8b2c2d6719
+size 12132719

conf/datasets/Km_test_11722.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:10f565a6b369e374d15c872ce732630de80a06e89dac17256e62b1dfaa75a790
+size 3295007

conf/datasets/kcat_km_samples.xlsx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a6488b25a0bb696ad656cc28e3a7bde18545d08f9fce6027dad2f389340ed53
+size 5904797

config.json

@@ -0,0 +1,18 @@
+{
+  "directories": {
+    "conf": "conf",
+    "model": "model",
+    "scripts": "scripts",
+    "weight": "weight"
+  },
+  "paths": {
+    "datasets": "conf/datasets",
+    "figures": "conf/Figures",
+    "model_modules": "model",
+    "script_bootstrap": "scripts/_bootstrap.py",
+    "vocab": "weight/vocab.pkl",
+    "smiles_transformer_weight": "weight/trfm_12_23000.pkl",
+    "unikp_models": "weight/UniKP_model",
+    "prot_t5_model": "weight/prot_t5_xl_uniref50"
+  }
+}

model/__init__.py

@@ -0,0 +1 @@
+"""Model and representation utilities used by UniKP scripts."""

model/build_vocab.py

@@ -0,0 +1,173 @@
+import argparse
+import pickle
+from collections import Counter
+
+from project_paths import VOCAB_PATH
+
+
+class TorchVocab(object):
+    """
+    :property freqs: collections.Counter, コーパス中の単語の出現頻度を保持するオブジェクト
+    :property stoi: collections.defaultdict, string → id の対応を示す辞書
+    :property itos: collections.defaultdict, id → string の対応を示す辞書
+    """
+    def __init__(self, counter, max_size=None, min_freq=1, specials=['<pad>', '<oov>'],
+                 vectors=None, unk_init=None, vectors_cache=None):
+        """
+        :param counter: collections.Counter, データ中に含まれる単語の頻度を計測するためのcounter
+        :param max_size: int, vocabularyの最大のサイズ. Noneの場合は最大値なし. defaultはNone
+        :param min_freq: int, vocabulary中の単語の最低出現頻度. この数以下の出現回数の単語はvocabularyに加えられない.
+        :param specials: list of str, vocabularyにあらかじめ登録するtoken
+        :param vectors: list of vectors, 事前学習済みのベクトル. ex)Vocab.load_vectors
+        """
+        self.freqs = counter
+        counter = counter.copy()
+        min_freq = max(min_freq, 1)
+
+        self.itos = list(specials)
+        # special tokensの出現頻度はvocabulary作成の際にカウントされない
+        for tok in specials:
+            del counter[tok]
+
+        max_size = None if max_size is None else max_size + len(self.itos)
+
+        # まず頻度でソートし、次に文字順で並び替える
+        words_and_frequencies = sorted(counter.items(), key=lambda tup: tup[0])
+        words_and_frequencies.sort(key=lambda tup: tup[1], reverse=True)
+        
+        # 出現頻度がmin_freq未満のものはvocabに加えない
+        for word, freq in words_and_frequencies:
+            if freq < min_freq or len(self.itos) == max_size:
+                break
+            self.itos.append(word)
+
+        # dictのk,vをいれかえてstoiを作成する
+        self.stoi = {tok: i for i, tok in enumerate(self.itos)}
+
+        self.vectors = None
+        if vectors is not None:
+            self.load_vectors(vectors, unk_init=unk_init, cache=vectors_cache)
+        else:
+            assert unk_init is None and vectors_cache is None
+
+    def __eq__(self, other):
+        if self.freqs != other.freqs:
+            return False
+        if self.stoi != other.stoi:
+            return False
+        if self.itos != other.itos:
+            return False
+        if self.vectors != other.vectors:
+            return False
+        return True
+
+    def __len__(self):
+        return len(self.itos)
+
+    def vocab_rerank(self):
+        self.stoi = {word: i for i, word in enumerate(self.itos)}
+
+    def extend(self, v, sort=False):
+        words = sorted(v.itos) if sort else v.itos
+        for w in words:
+            if w not in self.stoi:
+                self.itos.append(w)
+                self.stoi[w] = len(self.itos) - 1
+
+
+class Vocab(TorchVocab):
+    def __init__(self, counter, max_size=None, min_freq=1):
+        self.pad_index = 0
+        self.unk_index = 1
+        self.eos_index = 2
+        self.sos_index = 3
+        self.mask_index = 4
+        super().__init__(counter, specials=["<pad>", "<unk>", "<eos>", "<sos>", "<mask>"], max_size=max_size, min_freq=min_freq)
+
+    # override用
+    def to_seq(self, sentece, seq_len, with_eos=False, with_sos=False) -> list:
+        pass
+
+    # override用
+    def from_seq(self, seq, join=False, with_pad=False):
+        pass
+
+    @staticmethod
+    def load_vocab(vocab_path: str) -> 'Vocab':
+        with open(vocab_path, "rb") as f:
+            return pickle.load(f)
+
+    def save_vocab(self, vocab_path):
+        with open(vocab_path, "wb") as f:
+            pickle.dump(self, f)
+
+
+# テキストファイルからvocabを作成する
+class WordVocab(Vocab):
+    def __init__(self, texts, max_size=None, min_freq=1):
+        print("Building Vocab")
+        counter = Counter()
+        for line in texts:
+            if isinstance(line, list):
+                words = line
+            else:
+                words = line.replace("\n", "").replace("\t", "").split()
+
+            for word in words:
+                counter[word] += 1
+        super().__init__(counter, max_size=max_size, min_freq=min_freq)
+
+    def to_seq(self, sentence, seq_len=None, with_eos=False, with_sos=False, with_len=False):
+        if isinstance(sentence, str):
+            sentence = sentence.split()
+
+        seq = [self.stoi.get(word, self.unk_index) for word in sentence]
+
+        if with_eos:
+            seq += [self.eos_index]  # this would be index 1
+        if with_sos:
+            seq = [self.sos_index] + seq
+
+        origin_seq_len = len(seq)
+
+        if seq_len is None:
+            pass
+        elif len(seq) <= seq_len:
+            seq += [self.pad_index for _ in range(seq_len - len(seq))]
+        else:
+            seq = seq[:seq_len]
+
+        return (seq, origin_seq_len) if with_len else seq
+
+    def from_seq(self, seq, join=False, with_pad=False):
+        words = [self.itos[idx]
+                 if idx < len(self.itos)
+                 else "<%d>" % idx
+                 for idx in seq
+                 if not with_pad or idx != self.pad_index]
+
+        return " ".join(words) if join else words
+
+    @staticmethod
+    def load_vocab(vocab_path: str) -> 'WordVocab':
+        with open(vocab_path, "rb") as f:
+            return pickle.load(f)
+
+
+def main():
+    parser = argparse.ArgumentParser(description='Build a vocabulary pickle')
+    parser.add_argument('--corpus_path', '-c', type=str, default='data/chembl24_corpus.txt', help='path to th ecorpus')
+    parser.add_argument('--out_path', '-o', type=str, default=str(VOCAB_PATH), help='output file')
+    parser.add_argument('--min_freq', '-m', type=int, default=500, help='minimum frequency for vocabulary')
+    parser.add_argument('--vocab_size', '-v', type=int, default=None, help='max vocabulary size')
+    parser.add_argument('--encoding', '-e', type=str, default='utf-8', help='encoding of corpus')
+    args = parser.parse_args()
+
+    with open(args.corpus_path, "r", encoding=args.encoding) as f:
+        vocab = WordVocab(f, max_size=args.vocab_size, min_freq=args.min_freq)
+
+    print("VOCAB SIZE:", len(vocab))
+    vocab.save_vocab(args.out_path)
+
+if __name__=='__main__':
+    main()

model/dataset.py

@@ -0,0 +1,55 @@
+import random
+import pandas as pd
+import torch
+from torch.utils.data import Dataset, DataLoader
+
+from .enumerator import SmilesEnumerator
+from .utils import split
+
+PAD = 0
+MAX_LEN = 220
+
+class Randomizer(object):
+
+    def __init__(self):
+        self.sme = SmilesEnumerator()
+    
+    def __call__(self, sm):
+        sm_r = self.sme.randomize_smiles(sm) # Random transoform
+        if sm_r is None:
+            sm_spaced = split(sm) # Spacing
+        else:
+            sm_spaced = split(sm_r) # Spacing
+        sm_split = sm_spaced.split()
+        if len(sm_split)<=MAX_LEN - 2:
+            return sm_split # List
+        else:
+            return split(sm).split()
+
+    def random_transform(self, sm):
+        '''
+        function: Random transformation for SMILES. It may take some time.
+        input: A SMILES
+        output: A randomized SMILES
+        '''
+        return self.sme.randomize_smiles(sm)
+
+class Seq2seqDataset(Dataset):
+
+    def __init__(self, smiles, vocab, seq_len=220, transform=Randomizer()):
+        self.smiles = smiles
+        self.vocab = vocab
+        self.seq_len = seq_len
+        self.transform = transform
+
+    def __len__(self):
+        return len(self.smiles)
+
+    def __getitem__(self, item):
+        sm = self.smiles[item]
+        sm = self.transform(sm) # List
+        content = [self.vocab.stoi.get(token, self.vocab.unk_index) for token in sm]
+        X = [self.vocab.sos_index] + content + [self.vocab.eos_index]
+        padding = [self.vocab.pad_index]*(self.seq_len - len(X))
+        X.extend(padding)
+        return torch.tensor(X)

model/enumerator.py

@@ -0,0 +1,223 @@
+#Experimental Class for Smiles Enumeration, Iterator and SmilesIterator adapted from Keras 1.2.2
+from rdkit import Chem
+import numpy as np
+import threading
+
+class Iterator(object):
+    """Abstract base class for data iterators.
+
+    # Arguments
+        n: Integer, total number of samples in the dataset to loop over.
+        batch_size: Integer, size of a batch.
+        shuffle: Boolean, whether to shuffle the data between epochs.
+        seed: Random seeding for data shuffling.
+    """
+
+    def __init__(self, n, batch_size, shuffle, seed):
+        self.n = n
+        self.batch_size = batch_size
+        self.shuffle = shuffle
+        self.batch_index = 0
+        self.total_batches_seen = 0
+        self.lock = threading.Lock()
+        self.index_generator = self._flow_index(n, batch_size, shuffle, seed)
+        if n < batch_size:
+            raise ValueError('Input data length is shorter than batch_size\nAdjust batch_size')
+
+    def reset(self):
+        self.batch_index = 0
+
+    def _flow_index(self, n, batch_size=32, shuffle=False, seed=None):
+        # Ensure self.batch_index is 0.
+        self.reset()
+        while 1:
+            if seed is not None:
+                np.random.seed(seed + self.total_batches_seen)
+            if self.batch_index == 0:
+                index_array = np.arange(n)
+                if shuffle:
+                    index_array = np.random.permutation(n)
+
+            current_index = (self.batch_index * batch_size) % n
+            if n > current_index + batch_size:
+                current_batch_size = batch_size
+                self.batch_index += 1
+            else:
+                current_batch_size = n - current_index
+                self.batch_index = 0
+            self.total_batches_seen += 1
+            yield (index_array[current_index: current_index + current_batch_size],
+                   current_index, current_batch_size)
+
+    def __iter__(self):
+        # Needed if we want to do something like:
+        # for x, y in data_gen.flow(...):
+        return self
+
+    def __next__(self, *args, **kwargs):
+        return self.next(*args, **kwargs)
+
+
+
+
+class SmilesIterator(Iterator):
+    """Iterator yielding data from a SMILES array.
+
+    # Arguments
+        x: Numpy array of SMILES input data.
+        y: Numpy array of targets data.
+        smiles_data_generator: Instance of `SmilesEnumerator`
+            to use for random SMILES generation.
+        batch_size: Integer, size of a batch.
+        shuffle: Boolean, whether to shuffle the data between epochs.
+        seed: Random seed for data shuffling.
+        dtype: dtype to use for returned batch. Set to keras.backend.floatx if using Keras
+    """
+
+    def __init__(self, x, y, smiles_data_generator,
+                 batch_size=32, shuffle=False, seed=None,
+                 dtype=np.float32
+                 ):
+        if y is not None and len(x) != len(y):
+            raise ValueError('X (images tensor) and y (labels) '
+                             'should have the same length. '
+                             'Found: X.shape = %s, y.shape = %s' %
+                             (np.asarray(x).shape, np.asarray(y).shape))
+
+        self.x = np.asarray(x)
+
+        if y is not None:
+            self.y = np.asarray(y)
+        else:
+            self.y = None
+        self.smiles_data_generator = smiles_data_generator
+        self.dtype = dtype
+        super(SmilesIterator, self).__init__(x.shape[0], batch_size, shuffle, seed)
+
+    def next(self):
+        """For python 2.x.
+
+        # Returns
+            The next batch.
+        """
+        # Keeps under lock only the mechanism which advances
+        # the indexing of each batch.
+        with self.lock:
+            index_array, current_index, current_batch_size = next(self.index_generator)
+        # The transformation of images is not under thread lock
+        # so it can be done in parallel
+        batch_x = np.zeros(tuple([current_batch_size] + [ self.smiles_data_generator.pad, self.smiles_data_generator._charlen]), dtype=self.dtype)
+        for i, j in enumerate(index_array):
+            smiles = self.x[j:j+1]
+            x = self.smiles_data_generator.transform(smiles)
+            batch_x[i] = x
+
+        if self.y is None:
+            return batch_x
+        batch_y = self.y[index_array]
+        return batch_x, batch_y
+
+
+class SmilesEnumerator(object):
+    """SMILES Enumerator, vectorizer and devectorizer
+    
+    #Arguments
+        charset: string containing the characters for the vectorization
+          can also be generated via the .fit() method
+        pad: Length of the vectorization
+        leftpad: Add spaces to the left of the SMILES
+        isomericSmiles: Generate SMILES containing information about stereogenic centers
+        enum: Enumerate the SMILES during transform
+        canonical: use canonical SMILES during transform (overrides enum)
+    """
+    def __init__(self, charset = '@C)(=cOn1S2/H[N]\\', pad=120, leftpad=True, isomericSmiles=True, enum=True, canonical=False):
+        self._charset = None
+        self.charset = charset
+        self.pad = pad
+        self.leftpad = leftpad
+        self.isomericSmiles = isomericSmiles
+        self.enumerate = enum
+        self.canonical = canonical
+
+    @property
+    def charset(self):
+        return self._charset
+        
+    @charset.setter
+    def charset(self, charset):
+        self._charset = charset
+        self._charlen = len(charset)
+        self._char_to_int = dict((c,i) for i,c in enumerate(charset))
+        self._int_to_char = dict((i,c) for i,c in enumerate(charset))
+        
+    def fit(self, smiles, extra_chars=[], extra_pad = 5):
+        """Performs extraction of the charset and length of a SMILES datasets and sets self.pad and self.charset
+        
+        #Arguments
+            smiles: Numpy array or Pandas series containing smiles as strings
+            extra_chars: List of extra chars to add to the charset (e.g. "\\\\" when "/" is present)
+            extra_pad: Extra padding to add before or after the SMILES vectorization
+        """
+        charset = set("".join(list(smiles)))
+        #print(charset)
+        self.charset = "".join(charset.union(set(extra_chars)))
+        #print(self.charset)
+        self.pad = max([len(smile) for smile in smiles]) + extra_pad
+        
+    def randomize_smiles(self, smiles):
+        """Perform a randomization of a SMILES string
+        must be RDKit sanitizable"""
+        m = Chem.MolFromSmiles(smiles)
+        if m is None:
+            return None # Invalid SMILES
+        ans = list(range(m.GetNumAtoms()))
+        np.random.shuffle(ans)
+        nm = Chem.RenumberAtoms(m,ans)
+        return Chem.MolToSmiles(nm, canonical=self.canonical, isomericSmiles=self.isomericSmiles)
+
+    def transform(self, smiles):
+        """Perform an enumeration (randomization) and vectorization of a Numpy array of smiles strings
+        #Arguments
+            smiles: Numpy array or Pandas series containing smiles as strings
+        """
+        one_hot =  np.zeros((smiles.shape[0], self.pad, self._charlen),dtype=np.int8)
+        
+        if self.leftpad:
+            #print(smiles)
+            for i,ss in enumerate(smiles):
+                if self.enumerate: 
+                    ss = self.randomize_smiles(ss)
+                l = len(ss)
+                #print("???", ss)
+                diff = self.pad - l
+                for j,c in enumerate(ss):
+                    one_hot[i,j+diff,self._char_to_int[c]] = 1
+            return one_hot
+        else:
+            for i,ss in enumerate(smiles):
+                if self.enumerate: 
+                    ss = self.randomize_smiles(ss)
+                for j,c in enumerate(ss):
+                    one_hot[i,j,self._char_to_int[c]] = 1
+            return one_hot
+
+      
+    def reverse_transform(self, vect):
+        """ Performs a conversion of a vectorized SMILES to a smiles strings
+        charset must be the same as used for vectorization.
+        #Arguments
+            vect: Numpy array of vectorized SMILES.
+        """       
+        smiles = []
+        for v in vect:
+            #mask v 
+            v=v[v.sum(axis=1)==1]
+            #Find one hot encoded index with argmax, translate to char and join to string
+            smile = "".join(self._int_to_char[i] for i in v.argmax(axis=1))
+            smiles.append(smile)
+        return np.array(smiles)
+
+
+
+
+        

model/pretrain_rnn.py

@@ -0,0 +1,233 @@
+import argparse
+import random
+import numpy as np
+from tqdm import tqdm
+from torch.utils.data import DataLoader
+import os
+import math
+import argparse
+import torch
+from torch import nn
+from torch import optim
+from torch.autograd import Variable
+from torch.nn.utils import clip_grad_norm_
+from torch.nn import functional as F
+from .build_vocab import WordVocab
+from .dataset import Seq2seqDataset
+from project_paths import VOCAB_PATH
+
+PAD = 0
+UNK = 1
+EOS = 2
+SOS = 3
+MASK = 4
+
+class Encoder(nn.Module):
+    def __init__(self, input_size, embed_size, hidden_size,
+                 n_layers=1, dropout=0.5):
+        super(Encoder, self).__init__()
+        self.input_size = input_size
+        self.hidden_size = hidden_size
+        self.embed_size = embed_size
+        self.embed = nn.Embedding(input_size, embed_size)
+        self.gru = nn.GRU(embed_size, hidden_size, n_layers,
+                          dropout=dropout, bidirectional=True)
+
+    def forward(self, src, hidden=None):
+        # src: (T,B)
+        embedded = self.embed(src)# (T,B,H)
+        outputs, hidden = self.gru(embedded, hidden) # (T,B,2H), (2L,B,H) 
+        # sum bidirectional outputs
+        outputs = (outputs[:, :, :self.hidden_size] +
+                   outputs[:, :, self.hidden_size:])
+        return outputs, hidden # (T,B,H), (2L,B,H)
+
+
+class Attention(nn.Module):
+    def __init__(self, hidden_size):
+        super(Attention, self).__init__()
+        self.hidden_size = hidden_size
+        self.attn = nn.Linear(self.hidden_size * 2, hidden_size)
+        self.v = nn.Parameter(torch.rand(hidden_size))
+        stdv = 1. / math.sqrt(self.v.size(0))
+        self.v.data.uniform_(-stdv, stdv)
+
+    def forward(self, hidden, encoder_outputs):
+        timestep = encoder_outputs.size(0)
+        h = hidden.repeat(timestep, 1, 1).transpose(0, 1)
+        encoder_outputs = encoder_outputs.transpose(0, 1)  # [B*T*H]
+        attn_energies = self.score(h, encoder_outputs)
+        return F.relu(attn_energies).unsqueeze(1)
+
+    def score(self, hidden, encoder_outputs):
+        # [B*T*2H]->[B*T*H]
+        energy = F.softmax(self.attn(torch.cat([hidden, encoder_outputs], 2)))
+        energy = energy.transpose(1, 2)  # [B*H*T]
+        v = self.v.repeat(encoder_outputs.size(0), 1).unsqueeze(1)  # [B*1*H]
+        energy = torch.bmm(v, energy)  # [B*1*T]
+        return energy.squeeze(1)  # [B*T]
+
+
+class Decoder(nn.Module):
+    def __init__(self, embed_size, hidden_size, output_size,
+                 n_layers=1, dropout=0.2):
+        super(Decoder, self).__init__()
+        self.embed_size = embed_size
+        self.hidden_size = hidden_size
+        self.output_size = output_size
+        self.n_layers = n_layers
+
+        self.embed = nn.Embedding(output_size, embed_size)
+        self.dropout = nn.Dropout(dropout, inplace=True)
+        self.attention = Attention(hidden_size)
+        self.gru = nn.GRU(hidden_size + embed_size, hidden_size,
+                          n_layers, dropout=dropout)
+        self.out = nn.Linear(hidden_size * 2, output_size)
+
+    def forward(self, input, last_hidden, encoder_outputs):
+        # Get the embedding of the current input word (last output word)
+        embedded = self.embed(input).unsqueeze(0)  # (1,B,N)
+        embedded = self.dropout(embedded)
+        # Calculate attention weights and apply to encoder outputs
+        attn_weights = self.attention(last_hidden[-1], encoder_outputs)
+        context = attn_weights.bmm(encoder_outputs.transpose(0, 1))  # (B,1,N)
+        context = context.transpose(0, 1)  # (1,B,N)
+        # Combine embedded input word and attended context, run through RNN
+        rnn_input = torch.cat([embedded, context], 2)
+        output, hidden = self.gru(rnn_input, last_hidden)
+        output = output.squeeze(0)  # (1,B,N) -> (B,N)
+        context = context.squeeze(0)
+        output = self.out(torch.cat([output, context], 1))
+        output = F.log_softmax(output, dim=1) # log???
+        return output, hidden, attn_weights
+
+
+class RNNSeq2Seq(nn.Module):
+    def __init__(self, in_size, hidden_size, out_size, n_layers):
+        super(RNNSeq2Seq, self).__init__()
+        self.encoder = Encoder(in_size, hidden_size, hidden_size, n_layers)
+        self.decoder = Decoder(hidden_size, hidden_size, out_size, n_layers)
+
+    def forward(self, src, trg, teacher_forcing_ratio=0.5): # (T,B)
+        batch_size = src.size(1)
+        max_len = trg.size(0)
+        vocab_size = self.decoder.output_size
+        outputs = Variable(torch.zeros(max_len, batch_size, vocab_size)).cuda() # (T,B,V)
+        encoder_output, hidden = self.encoder(src) # (T,B,H), (2L,B,H)
+        hidden = hidden[:self.decoder.n_layers] # (L,B,H)
+        output = Variable(trg.data[0, :])  # sos
+        for t in range(1, max_len):
+            output, hidden, attn_weights = self.decoder(output, hidden, encoder_output) # (B,V), (L,B,H)
+            outputs[t] = output
+            is_teacher = random.random() < teacher_forcing_ratio
+            top1 = output.data.max(dim=1)[1] # (B)
+            output = Variable(trg.data[t] if is_teacher else top1).cuda()
+        return outputs # (T,B,V)
+
+    def _encode(self, src):
+        # src: (T,B)
+        embedded = self.encoder.embed(src)# (T,B,H)
+        _, hidden = self.encoder.gru(embedded, None) # (T,B,2H), (2L,B,H)
+        hidden = hidden.detach().numpy() 
+        return np.hstack(hidden[2:]) #(B,4H)
+        
+    def encode(self, src):
+        # src: (T,B)
+        batch_size = src.shape[1]
+        if batch_size<=100:
+            return self._encode(src)
+        else: # Batch is too large to load
+            print('There are {:d} molecules. It will take a little time.'.format(batch_size))
+            st,ed = 0,100
+            out = self._encode(src[:,st:ed]) # (B,4H)
+            while ed<batch_size:
+                st += 100
+                ed += 100
+                out = np.concatenate([out, self._encode(src[:,st:ed])], axis=0)
+            return out
+        
+def parse_arguments():
+    parser = argparse.ArgumentParser(description='Hyperparams')
+    parser.add_argument('--n_epoch', '-e', type=int, default=20, help='number of epochs')
+    parser.add_argument('--vocab', '-v', type=str, default=str(VOCAB_PATH), help='vocabulary (.pkl)')
+    parser.add_argument('--train_data', type=str, default='data/chembl24_bert_train.csv', help='train corpus (.csv)')
+    parser.add_argument('--test_data', type=str, default='data/chembl24_bert_test.csv', help='test corpus (.csv)')
+    parser.add_argument('--out-dir', '-o', type=str, default='../result', help='output directory')
+    parser.add_argument('--name', '-n', type=str, default='ST', help='model name')
+    parser.add_argument('--seq_len', type=int, default=220, help='maximum length of the paired seqence')
+    parser.add_argument('--batch_size', '-b', type=int, default=16, help='batch size')
+    parser.add_argument('--n_worker', '-w', type=int, default=16, help='number of workers')
+    parser.add_argument('--hidden', type=int, default=256, help='length of hidden vector')
+    parser.add_argument('--n_layer', '-l', type=int, default=8, help='number of layers')
+    parser.add_argument('--n_head', type=int, default=8, help='number of attention heads')
+    parser.add_argument('--dropout', '-d', type=float, default=0.1, help='dropout rate')
+    parser.add_argument('--lr', type=float, default=1e-4, help='Adam learning rate')
+    parser.add_argument('--lr-decay', type=int, default=50000, help='lr decay step size')
+    parser.add_argument('--log-freq', type=int, default=100, help='log frequency')
+    parser.add_argument('--gpu', metavar='N', type=int, nargs='+', help='list of GPU IDs to use')
+    parser.add_argument('--checkpoint', '-c', type=str, default=None, help='Parameter to load')
+    parser.add_argument('-grad_clip', type=float, default=10.0, help='in case of gradient explosion')
+    return parser.parse_args()
+
+def evaluate(model, val_loader, vocab):
+    model.eval()
+    total_loss = 0
+    for b, data in enumerate(val_loader):
+        sm1, sm2 = torch.t(data[0].cuda()), torch.t(data[1].cuda()) # (T,B)
+        with torch.no_grad():
+            output = model(sm1, sm2, teacher_forcing_ratio=0.0) # (T,B,V)
+        loss = F.nll_loss(output[1:].view(-1, len(vocab)),
+                               sm2[1:].contiguous().view(-1),
+                               ignore_index=PAD)
+        total_loss += loss.item()
+    return total_loss / len(val_loader)
+
+def main():
+    args = parse_arguments()
+    hidden_size = 256
+    embed_size = 256
+    assert torch.cuda.is_available()
+
+    vocab = WordVocab.load_vocab(args.vocab)
+    print("[!] Instantiating models...")
+    encoder = Encoder(len(vocab), embed_size, hidden_size, n_layers=3, dropout=0.5)
+    decoder = Decoder(embed_size, hidden_size, len(vocab), n_layers=3, dropout=0.5)
+    model = RNNSeq2Seq(encoder, decoder).cuda()
+    optimizer = optim.Adam(model.parameters(), lr=args.lr)
+    train_dataset = Seq2seqDataset(args.train_data, vocab)
+    train_loader = DataLoader(train_dataset, batch_size=args.batch_size, shuffle=True, num_workers=args.n_worker)
+    val_dataset = Seq2seqDataset(args.test_data, vocab, is_train=False)
+    val_loader = DataLoader(val_dataset, batch_size=args.batch_size, shuffle=False, num_workers=args.n_worker)
+    print(model)
+    print('Total parameters:', sum(p.numel() for p in model.parameters()))
+
+    best_loss = None
+    for e in range(1, args.n_epoch):
+        for b,data in tqdm(enumerate(train_loader)):
+            model.train()
+            sm1, sm2 = torch.t(data[0].cuda()), torch.t(data[1].cuda()) # (T,B)
+            optimizer.zero_grad()
+            output = model(sm1, sm2, teacher_forcing_ratio=1.0) # (T,B,V)
+            loss = F.nll_loss(output[1:].view(-1, len(vocab)),
+                    sm2[1:].contiguous().view(-1), ignore_index=PAD)
+            loss.backward()
+            clip_grad_norm_(model.parameters(), args.grad_clip)
+            optimizer.step()
+            if b%100==0:
+                print('Train {:3d}: iter {:5d} | loss {:.3f} | ppl {:.3f}'.format(e, b, loss.item(), math.exp(loss.item())))
+            if b%1000==0:
+                loss = evaluate(model, val_loader, vocab)
+                print('Val {:3d}: iter {:5d} | loss {:.3f} | ppl {:.3f}'.format(e, b, loss, math.exp(loss)))
+                # Save the model if the validation loss is the best we've seen so far.
+                
+                print("[!] saving model...")
+                if not os.path.isdir(".save"):
+                    os.makedirs(".save")
+                torch.save(model.state_dict(), './.save/rnnenum_%d_%d.pkl' % (e,b))
+                best_loss = loss
+
+if __name__ == "__main__":
+    try:
+        main()
+    except KeyboardInterrupt as e:
+        print("[STOP]", e)

model/pretrain_trfm.py

@@ -0,0 +1,174 @@
+import argparse
+import math
+import os
+
+import numpy as np
+import pandas as pd
+import torch
+from torch import nn
+from torch import optim
+from torch.autograd import Variable
+from torch.nn import functional as F
+from torch.utils.data import DataLoader
+from tqdm import tqdm
+
+from .build_vocab import WordVocab
+from .dataset import Seq2seqDataset
+from project_paths import VOCAB_PATH
+
+PAD = 0
+UNK = 1
+EOS = 2
+SOS = 3
+MASK = 4
+
+class PositionalEncoding(nn.Module):
+    "Implement the PE function. No batch support?"
+    def __init__(self, d_model, dropout, max_len=5000):
+        super(PositionalEncoding, self).__init__()
+        self.dropout = nn.Dropout(p=dropout)
+        
+        # Compute the positional encodings once in log space.
+        pe = torch.zeros(max_len, d_model) # (T,H)
+        position = torch.arange(0., max_len).unsqueeze(1)
+        div_term = torch.exp(torch.arange(0., d_model, 2) * -(math.log(10000.0) / d_model))
+        pe[:, 0::2] = torch.sin(position * div_term)
+        pe[:, 1::2] = torch.cos(position * div_term)
+        pe = pe.unsqueeze(0)
+        self.register_buffer('pe', pe)
+        
+    def forward(self, x):
+        x = x + Variable(self.pe[:, :x.size(1)], 
+                         requires_grad=False)
+        return self.dropout(x)
+
+class TrfmSeq2seq(nn.Module):
+    def __init__(self, in_size, hidden_size, out_size, n_layers, dropout=0.1):
+        super(TrfmSeq2seq, self).__init__()
+        self.in_size = in_size
+        self.hidden_size = hidden_size
+        self.embed = nn.Embedding(in_size, hidden_size)
+        self.pe = PositionalEncoding(hidden_size, dropout)
+        self.trfm = nn.Transformer(d_model=hidden_size, nhead=4, 
+        num_encoder_layers=n_layers, num_decoder_layers=n_layers, dim_feedforward=hidden_size)
+        self.out = nn.Linear(hidden_size, out_size)
+
+    def forward(self, src):
+        # src: (T,B)
+        embedded = self.embed(src)  # (T,B,H)
+        embedded = self.pe(embedded) # (T,B,H)
+        hidden = self.trfm(embedded, embedded) # (T,B,H)
+        out = self.out(hidden) # (T,B,V)
+        out = F.log_softmax(out, dim=2) # (T,B,V)
+        return out # (T,B,V)
+
+    def _encode(self, src):
+        # src: (T,B)
+        embedded = self.embed(src)  # (T,B,H)
+        embedded = self.pe(embedded) # (T,B,H)
+        output = embedded
+        for i in range(self.trfm.encoder.num_layers - 1):
+            output = self.trfm.encoder.layers[i](output, None)  # (T,B,H)
+        penul = output.detach().numpy()
+        output = self.trfm.encoder.layers[-1](output, None)  # (T,B,H)
+        if self.trfm.encoder.norm:
+            output = self.trfm.encoder.norm(output) # (T,B,H)
+        output = output.detach().numpy()
+        # mean, max, first*2
+        return np.hstack([np.mean(output, axis=0), np.max(output, axis=0), output[0,:,:], penul[0,:,:] ]) # (B,4H)
+    
+    def encode(self, src):
+        # src: (T,B)
+        batch_size = src.shape[1]
+        if batch_size<=100:
+            return self._encode(src)
+        else: # Batch is too large to load
+            print('There are {:d} molecules. It will take a little time.'.format(batch_size))
+            st,ed = 0,100
+            out = self._encode(src[:,st:ed]) # (B,4H)
+            while ed<batch_size:
+                st += 100
+                ed += 100
+                out = np.concatenate([out, self._encode(src[:,st:ed])], axis=0)
+            return out
+
+def parse_arguments():
+    parser = argparse.ArgumentParser(description='Hyperparams')
+    parser.add_argument('--n_epoch', '-e', type=int, default=5, help='number of epochs')
+    parser.add_argument('--vocab', '-v', type=str, default=str(VOCAB_PATH), help='vocabulary (.pkl)')
+    parser.add_argument('--data', '-d', type=str, default='data/chembl_25.csv', help='train corpus (.csv)')
+    parser.add_argument('--out-dir', '-o', type=str, default='../result', help='output directory')
+    parser.add_argument('--name', '-n', type=str, default='ST', help='model name')
+    parser.add_argument('--seq_len', type=int, default=220, help='maximum length of the paired seqence')
+    parser.add_argument('--batch_size', '-b', type=int, default=8, help='batch size')
+    parser.add_argument('--n_worker', '-w', type=int, default=16, help='number of workers')
+    parser.add_argument('--hidden', type=int, default=256, help='length of hidden vector')
+    parser.add_argument('--n_layer', '-l', type=int, default=4, help='number of layers')
+    parser.add_argument('--n_head', type=int, default=4, help='number of attention heads')
+    parser.add_argument('--lr', type=float, default=1e-4, help='Adam learning rate')
+    parser.add_argument('--gpu', metavar='N', type=int, nargs='+', help='list of GPU IDs to use')
+    return parser.parse_args()
+
+
+def evaluate(model, test_loader, vocab):
+    model.eval()
+    total_loss = 0
+    for b, sm in enumerate(test_loader):
+        sm = torch.t(sm.cuda()) # (T,B)
+        with torch.no_grad():
+            output = model(sm) # (T,B,V)
+        loss = F.nll_loss(output.view(-1, len(vocab)),
+                               sm.contiguous().view(-1),
+                               ignore_index=PAD)
+        total_loss += loss.item()
+    return total_loss / len(test_loader)
+
+def main():
+    args = parse_arguments()
+    assert torch.cuda.is_available()
+
+    print('Loading dataset...')
+    vocab = WordVocab.load_vocab(args.vocab)
+    dataset = Seq2seqDataset(pd.read_csv(args.data)['canonical_smiles'].values, vocab)
+    test_size = 10000
+    train, test = torch.utils.data.random_split(dataset, [len(dataset)-test_size, test_size])
+    train_loader = DataLoader(train, batch_size=args.batch_size, shuffle=True, num_workers=args.n_worker)
+    test_loader = DataLoader(test, batch_size=args.batch_size, shuffle=False, num_workers=args.n_worker)
+    print('Train size:', len(train))
+    print('Test size:', len(test))
+    del dataset, train, test
+
+    model = TrfmSeq2seq(len(vocab), args.hidden, len(vocab), args.n_layer).cuda()
+    optimizer = optim.Adam(model.parameters(), lr=args.lr)
+    print(model)
+    print('Total parameters:', sum(p.numel() for p in model.parameters()))
+
+    best_loss = None
+    for e in range(1, args.n_epoch):
+        for b, sm in tqdm(enumerate(train_loader)):
+            sm = torch.t(sm.cuda()) # (T,B)
+            optimizer.zero_grad()
+            output = model(sm) # (T,B,V)
+            loss = F.nll_loss(output.view(-1, len(vocab)),
+                    sm.contiguous().view(-1), ignore_index=PAD)
+            loss.backward()
+            optimizer.step()
+            if b%1000==0:
+                print('Train {:3d}: iter {:5d} | loss {:.3f} | ppl {:.3f}'.format(e, b, loss.item(), math.exp(loss.item())))
+            if b%10000==0:
+                loss = evaluate(model, test_loader, vocab)
+                print('Val {:3d}: iter {:5d} | loss {:.3f} | ppl {:.3f}'.format(e, b, loss, math.exp(loss)))
+                # Save the model if the validation loss is the best we've seen so far.
+                if not best_loss or loss < best_loss:
+                    print("[!] saving model...")
+                    if not os.path.isdir(".save"):
+                        os.makedirs(".save")
+                    torch.save(model.state_dict(), './.save/trfm_new_%d_%d.pkl' % (e,b))
+                    best_loss = loss
+
+
+if __name__ == "__main__":
+    try:
+        main()
+    except KeyboardInterrupt as e:
+        print("[STOP]", e)

model/sample.py

@@ -0,0 +1,7 @@
+from .enumerator import SmilesEnumerator
+
+sme = SmilesEnumerator()
+smiles = ''
+for i in range(10):
+
+    print(sme.randomize_smiles("CN1C(=NS(=O)(=O)c2ccc(Cl)cc2)C(=NN=P(c3ccccc3)(c4ccccc4)c5ccccc5)c6ccccc16COc1ccc(Cl)cc1c2cc([nH]n2)C(=O)Nc3ccc(OC)nc3"))

model/utils.py

@@ -0,0 +1,194 @@
+import torch
+import math
+import torch.nn as nn
+from rdkit import Chem
+from rdkit import rdBase
+rdBase.DisableLog('rdApp.*')
+
+# Split SMILES into words
+def split(sm):
+    '''
+    function: Split SMILES into words. Care for Cl, Br, Si, Se, Na etc.
+    input: A SMILES
+    output: A string with space between words
+    '''
+    arr = []
+    i = 0
+    while i < len(sm)-1:
+        if not sm[i] in ['%', 'C', 'B', 'S', 'N', 'R', 'X', 'L', 'A', 'M', \
+                        'T', 'Z', 's', 't', 'H', '+', '-', 'K', 'F']:
+            arr.append(sm[i])
+            i += 1
+        elif sm[i]=='%':
+            arr.append(sm[i:i+3])
+            i += 3
+        elif sm[i]=='C' and sm[i+1]=='l':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='C' and sm[i+1]=='a':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='C' and sm[i+1]=='u':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='B' and sm[i+1]=='r':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='B' and sm[i+1]=='e':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='B' and sm[i+1]=='a':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='B' and sm[i+1]=='i':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='S' and sm[i+1]=='i':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='S' and sm[i+1]=='e':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='S' and sm[i+1]=='r':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='N' and sm[i+1]=='a':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='N' and sm[i+1]=='i':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='R' and sm[i+1]=='b':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='R' and sm[i+1]=='a':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='X' and sm[i+1]=='e':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='L' and sm[i+1]=='i':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='A' and sm[i+1]=='l':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='A' and sm[i+1]=='s':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='A' and sm[i+1]=='g':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='A' and sm[i+1]=='u':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='M' and sm[i+1]=='g':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='M' and sm[i+1]=='n':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='T' and sm[i+1]=='e':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='Z' and sm[i+1]=='n':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='s' and sm[i+1]=='i':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='s' and sm[i+1]=='e':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='t' and sm[i+1]=='e':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='H' and sm[i+1]=='e':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='+' and sm[i+1]=='2':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='+' and sm[i+1]=='3':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='+' and sm[i+1]=='4':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='-' and sm[i+1]=='2':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='-' and sm[i+1]=='3':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='-' and sm[i+1]=='4':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='K' and sm[i+1]=='r':
+            arr.append(sm[i:i+2])
+            i += 2
+        elif sm[i]=='F' and sm[i+1]=='e':
+            arr.append(sm[i:i+2])
+            i += 2
+        else:
+            arr.append(sm[i])
+            i += 1
+    if i == len(sm)-1:
+        arr.append(sm[i])
+    return ' '.join(arr) 
+
+# 活性化関数
+class GELU(nn.Module):
+    def forward(self, x):
+        return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+
+# 位置情報を考慮したFFN
+class PositionwiseFeedForward(nn.Module):
+    def __init__(self, d_model, d_ff, dropout=0.1):
+        super(PositionwiseFeedForward, self).__init__()
+        self.w_1 = nn.Linear(d_model, d_ff)
+        self.w_2 = nn.Linear(d_ff, d_model)
+        self.dropout = nn.Dropout(dropout)
+        self.activation = GELU()
+
+    def forward(self, x):
+        return self.w_2(self.dropout(self.activation(self.w_1(x))))
+    
+# 正規化層
+class LayerNorm(nn.Module):
+    def __init__(self, features, eps=1e-6):
+        super(LayerNorm, self).__init__()
+        self.a_2 = nn.Parameter(torch.ones(features))
+        self.b_2 = nn.Parameter(torch.zeros(features))
+        self.eps = eps
+
+    def forward(self, x):
+        mean = x.mean(-1, keepdim=True)
+        std = x.std(-1, keepdim=True)
+        return self.a_2 * (x - mean) / (std + self.eps) + self.b_2
+
+
+class SublayerConnection(nn.Module):
+    def __init__(self, size, dropout):
+        super(SublayerConnection, self).__init__()
+        self.norm = LayerNorm(size)
+        self.dropout = nn.Dropout(dropout)
+
+    def forward(self, x, sublayer):
+        return x + self.dropout(sublayer(self.norm(x)))
+
+# Sample SMILES from probablistic distribution
+def sample(msms):
+    ret = []
+    for msm in msms:
+        ret.append(torch.multinomial(msm.exp(), 1).squeeze())
+    return torch.stack(ret)
+
+def validity(smiles):
+    loss = 0
+    for sm in smiles:
+        mol = Chem.MolFromSmiles(sm)
+        if mol is None:
+            loss += 1
+    return 1-loss/len(smiles)
+

requirements.txt

@@ -0,0 +1,15 @@
+# This file is part of the OneScience IgGM project.
+https://download.sourcefind.cn:65024/directlink/4/pytorch/DAS1.7/torch-2.5.1+das.opt1.dtk25042-cp311-cp311-manylinux_2_28_x86_64.whl
+#https://download.sourcefind.cn:65024/file/4/triton/DAS1.7/triton-3.1.0+das.opt1.dtk25042-cp311-cp311-manylinux_2_28_x86_64.whl
+#https://download.sourcefind.cn:65024/directlink/9/onesicence/dtk-25.04.2/openmm-8.3.1-cp311-cp311-linux_x86_64.whl
+https://download.sourcefind.cn:65024/directlink/4/fastpt/DAS1.7/fastpt-2.1.1+das.dtk25042-cp311-cp311-linux_x86_64.whl
+https://download.sourcefind.cn:65024/directlink/4/vision/DAS1.7/torchvision-0.20.1+das.opt1.dtk25042-cp311-cp311-manylinux_2_28_x86_64.whl
+
+numpy==1.26.3
+pandas
+tqdm
+rdkit
+transformers
+SentencePiece
+scikit-learn
+openpyxl

scripts/Preprocessing.py

@@ -0,0 +1,148 @@
+import pandas as pd
+import numpy as np
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from scipy.stats import pearsonr
+
+
+res = np.array(pd.read_excel('1_all_samples_metrics.xlsx', sheet_name='Sheet1')).T
+sequence = res[1]
+smiles = res[2]
+Type = res[6]
+Value = res[7]
+Predict_Label = res[8]
+Training_test = res[9]
+print(sequence[0], smiles[0], Type[0], Value[0], Predict_Label[0], Training_test[0])
+
+
+def Whole_dataset():
+    # Calculate the whole dataset
+    Pcc = pearsonr(Value, Predict_Label)[0]
+    RMSE = np.sqrt(mean_squared_error(Value, Predict_Label))
+    MAE = mean_absolute_error(Value, Predict_Label)
+    r2 = r2_score(Value, Predict_Label)
+    print('***The whole set***')
+    print('Pcc:', Pcc, 'RMSE:', RMSE, 'MAE:', MAE, 'r2:', r2)
+
+
+def test_dataset():
+    # Calculate the test dataset
+    Value_test = []
+    Predict_Label_test = []
+    for i in range(len(Training_test)):
+        if Training_test[i] == 1:
+            Value_test.append(Value[i])
+            Predict_Label_test.append(Predict_Label[i])
+    Value_test = np.array(Value_test)
+    Predict_Label_test = np.array(Predict_Label_test)
+    Pcc_test = pearsonr(Value_test, Predict_Label_test)[0]
+    RMSE_test = np.sqrt(mean_squared_error(Value_test, Predict_Label_test))
+    MAE_test = mean_absolute_error(Value_test, Predict_Label_test)
+    r2_test = r2_score(Value_test, Predict_Label_test)
+    print('***Test set***')
+    print('Pcc:', Pcc_test, 'RMSE:', RMSE_test, 'MAE:', MAE_test, 'r2:', r2_test)
+
+
+def Wildtype_all_dataset():
+    # Calculate the Wildtype/Mutant dataset
+    Value_wildtype = []
+    Predict_Label_wildtype = []
+    for i in range(len(Type)):
+        if Type[i] == 'wildtype':
+            Value_wildtype.append(Value[i])
+            Predict_Label_wildtype.append(Predict_Label[i])
+    Value_wildtype = np.array(Value_wildtype)
+    Predict_Label_wildtype = np.array(Predict_Label_wildtype)
+    Pcc_test = pearsonr(Value_wildtype, Predict_Label_wildtype)[0]
+    RMSE_test = np.sqrt(mean_squared_error(Value_wildtype, Predict_Label_wildtype))
+    MAE_test = mean_absolute_error(Value_wildtype, Predict_Label_wildtype)
+    r2_test = r2_score(Value_wildtype, Predict_Label_wildtype)
+    print('***The whole wildtype set***')
+    print('Pcc:', Pcc_test, 'RMSE:', RMSE_test, 'MAE:', MAE_test, 'r2:', r2_test)
+
+
+def Wildtype_test_dataset():
+    # Calculate the Wildtype/Mutant dataset
+    Value_wildtype = []
+    Predict_Label_wildtype = []
+    for i in range(len(Type)):
+        if Type[i] == 'wildtype' and Training_test[i] == 1:
+            Value_wildtype.append(Value[i])
+            Predict_Label_wildtype.append(Predict_Label[i])
+    Value_wildtype = np.array(Value_wildtype)
+    Predict_Label_wildtype = np.array(Predict_Label_wildtype)
+    Pcc_test = pearsonr(Value_wildtype, Predict_Label_wildtype)[0]
+    RMSE_test = np.sqrt(mean_squared_error(Value_wildtype, Predict_Label_wildtype))
+    MAE_test = mean_absolute_error(Value_wildtype, Predict_Label_wildtype)
+    r2_test = r2_score(Value_wildtype, Predict_Label_wildtype)
+    print('***The test wildtype set***')
+    print('Pcc:', Pcc_test, 'RMSE:', RMSE_test, 'MAE:', MAE_test, 'r2:', r2_test)
+
+
+def Mutant_all_dataset():
+    # Calculate the Wildtype/Mutant dataset
+    Value_wildtype = []
+    Predict_Label_wildtype = []
+    for i in range(len(Type)):
+        if Type[i] != 'wildtype':
+            Value_wildtype.append(Value[i])
+            Predict_Label_wildtype.append(Predict_Label[i])
+    Value_wildtype = np.array(Value_wildtype)
+    Predict_Label_wildtype = np.array(Predict_Label_wildtype)
+    Pcc_test = pearsonr(Value_wildtype, Predict_Label_wildtype)[0]
+    RMSE_test = np.sqrt(mean_squared_error(Value_wildtype, Predict_Label_wildtype))
+    MAE_test = mean_absolute_error(Value_wildtype, Predict_Label_wildtype)
+    r2_test = r2_score(Value_wildtype, Predict_Label_wildtype)
+    print('***The whole mutant set***')
+    print('Pcc:', Pcc_test, 'RMSE:', RMSE_test, 'MAE:', MAE_test, 'r2:', r2_test)
+
+
+def Mutant_test_dataset():
+    # Calculate the Wildtype/Mutant dataset
+    Value_wildtype = []
+    Predict_Label_wildtype = []
+    for i in range(len(Type)):
+        if Type[i] != 'wildtype' and Training_test[i] == 1:
+            Value_wildtype.append(Value[i])
+            Predict_Label_wildtype.append(Predict_Label[i])
+    Value_wildtype = np.array(Value_wildtype)
+    Predict_Label_wildtype = np.array(Predict_Label_wildtype)
+    Pcc_test = pearsonr(Value_wildtype, Predict_Label_wildtype)[0]
+    RMSE_test = np.sqrt(mean_squared_error(Value_wildtype, Predict_Label_wildtype))
+    MAE_test = mean_absolute_error(Value_wildtype, Predict_Label_wildtype)
+    r2_test = r2_score(Value_wildtype, Predict_Label_wildtype)
+    print('***The test mutant set***')
+    print('Pcc:', Pcc_test, 'RMSE:', RMSE_test, 'MAE:', MAE_test, 'r2:', r2_test)
+
+
+def New_substrate_enzyme_dataset():
+    # Calculate the test New_substrate_enzyme dataset
+    Trainingset_seq_smiles = []
+    for i in range(len(Training_test)):
+        if Training_test[i] == 0:
+            Trainingset_seq_smiles.append(sequence[i])
+            Trainingset_seq_smiles.append(smiles[i])
+    Value_test = []
+    Predict_Label_test = []
+    for i in range(len(Training_test)):
+        if Training_test[i] == 1 and (sequence[i] not in Trainingset_seq_smiles or smiles[i] not in Trainingset_seq_smiles):
+            Value_test.append(Value[i])
+            Predict_Label_test.append(Predict_Label[i])
+    Value_test = np.array(Value_test)
+    Predict_Label_test = np.array(Predict_Label_test)
+    Pcc_test = pearsonr(Value_test, Predict_Label_test)[0]
+    RMSE_test = np.sqrt(mean_squared_error(Value_test, Predict_Label_test))
+    MAE_test = mean_absolute_error(Value_test, Predict_Label_test)
+    r2_test = r2_score(Value_test, Predict_Label_test)
+    print('***The Test new_substrate_enzyme dataset***')
+    print('Pcc:', Pcc_test, 'RMSE:', RMSE_test, 'MAE:', MAE_test, 'r2:', r2_test)
+
+
+if __name__ == '__main__':
+    # Whole_dataset()
+    # test_dataset()
+    # Wildtype_all_dataset()
+    # Wildtype_test_dataset()
+    # Mutant_all_dataset()
+    # Mutant_test_dataset()
+    New_substrate_enzyme_dataset()

scripts/UniKP_CBW.py

@@ -0,0 +1,231 @@
+import _bootstrap
+from collections import Counter
+from scipy.ndimage import convolve1d
+from scipy.ndimage import gaussian_filter1d
+from scipy.signal.windows import triang
+import matplotlib.pyplot as plt
+
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def get_lds_kernel_window(kernel, ks, sigma):
+    assert kernel in ['gaussian', 'triang', 'laplace']
+    half_ks = (ks - 1) // 2
+    if kernel == 'gaussian':
+        base_kernel = [0.] * half_ks + [1.] + [0.] * half_ks
+        kernel_window = gaussian_filter1d(base_kernel, sigma=sigma) / max(gaussian_filter1d(base_kernel, sigma=sigma))
+    elif kernel == 'triang':
+        kernel_window = triang(ks)
+    else:
+        laplace = lambda x: np.exp(-abs(x) / sigma) / (2. * sigma)
+        kernel_window = list(map(laplace, np.arange(-half_ks, half_ks + 1))) / max(map(laplace, np.arange(-half_ks, half_ks + 1)))
+    return kernel_window
+
+
+def Smooth_Label(Label_new):
+    labels = Label_new
+    for i in range(len(labels)):
+        labels[i] = labels[i] - min(labels)
+    bin_index_per_label = [int(label*10) for label in labels]
+    # print(bin_index_per_label)
+    Nb = max(bin_index_per_label) + 1
+    print(Nb)
+    num_samples_of_bins = dict(Counter(bin_index_per_label))
+    print(num_samples_of_bins)
+    emp_label_dist = [num_samples_of_bins.get(i, 0) for i in range(Nb)]
+    print(emp_label_dist, len(emp_label_dist))
+    eff_label_dist = []
+    beta = 0.9
+    for i in range(len(emp_label_dist)):
+        eff_label_dist.append((1-math.pow(beta, emp_label_dist[i])) / (1-beta))
+    print(eff_label_dist)
+    eff_num_per_label = [eff_label_dist[bin_idx] for bin_idx in bin_index_per_label]
+    weights = [np.float32(1 / x) for x in eff_num_per_label]
+    weights = np.array(weights)
+    print(weights)
+    print(len(weights))
+    return weights
+
+
+def Kcat_predict(Ifeature, Label, weights):
+    for i in range(3, 10):
+        kf = KFold(n_splits=5, shuffle=True)
+        All_pre_label = []
+        All_real_label = []
+        for train_index, test_index in kf.split(Ifeature, Label):
+            Train_data, Train_label = Ifeature[train_index], Label[train_index]
+            Test_data, Test_label = Ifeature[test_index], Label[test_index]
+            model = ExtraTreesRegressor()
+            model.fit(Train_data, Train_label, sample_weight=weights[train_index])
+            Pre_label = model.predict(Test_data)
+            All_pre_label.extend(Pre_label)
+            All_real_label.extend(Test_label)
+        res = pd.DataFrame({'Value': All_real_label, 'Predict_Label': All_pre_label})
+        res.to_excel('CBW/'+str(i)+'_0.9_Re_weighting_Kcat_5_cv'+'.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    with open(KCAT_DATASET_PATH, 'r') as file:
+        datasets = json.load(file)
+    # print(len(datasets))
+    # datasets = datasets[:50]
+    sequence = [data['Sequence'] for data in datasets]
+    Smiles = [data['Smiles'] for data in datasets]
+    Label = [float(data['Value']) for data in datasets]
+    ECNumber = [data['ECNumber'] for data in datasets]
+    Organism = [data['Organism'] for data in datasets]
+    Substrate = [data['Substrate'] for data in datasets]
+    Type = [data['Type'] for data in datasets]
+    for i in range(len(Label)):
+        if Label[i] == 0:
+            Label[i] = -10000000000
+        else:
+            Label[i] = math.log(Label[i], 10)
+    Label = np.array(Label)
+    print(max(Label), min(Label))
+    # Feature Extractor
+    # smiles_input = smiles_to_vec(Smiles)
+    # sequence_input = Seq_to_vec(sequence)
+    # feature = np.concatenate((smiles_input, sequence_input), axis=1)
+    with open("PreKcat_new/features_17010_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    # feature = feature[:50]
+    # Input dataset
+    feature_new = []
+    Label_new = []
+    sequence_new = []
+    Smiles_new = []
+    ECNumber_new = []
+    Organism_new = []
+    Substrate_new = []
+    Type_new = []
+    for i in range(len(Label)):
+        if -10000000000 < Label[i] and '.' not in Smiles[i]:
+            feature_new.append(feature[i])
+            Label_new.append(Label[i])
+            sequence_new.append(sequence[i])
+            Smiles_new.append(Smiles[i])
+            ECNumber_new.append(ECNumber[i])
+            Organism_new.append(Organism[i])
+            Substrate_new.append(Substrate[i])
+            Type_new.append(Type[i])
+    print(len(Label_new), min(Label_new), max(Label_new))
+    feature_new = np.array(feature_new)
+    Label_new = np.array(Label_new)
+    sl_label = [Label_new[i] for i in range(len(Label_new))]
+    weights = Smooth_Label(sl_label)
+    # weights = np.ones([len(Label_new)], dtype=float)
+    # for i in range(len(weights)):
+    #     if Label_new[i] > 5:
+    #         weights[i] = 2
+    # sum_weights = np.sum(weights)
+    # for i in range(len(weights)):
+    #     weights[i] /= sum_weights
+    # Modelling
+    Kcat_predict(feature_new, Label_new, weights)

scripts/UniKP_CSW.py

@@ -0,0 +1,226 @@
+import _bootstrap
+from collections import Counter
+from scipy.ndimage import convolve1d
+from scipy.ndimage import gaussian_filter1d
+from scipy.signal.windows import triang
+import matplotlib.pyplot as plt
+
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def get_lds_kernel_window(kernel, ks, sigma):
+    assert kernel in ['gaussian', 'triang', 'laplace']
+    half_ks = (ks - 1) // 2
+    if kernel == 'gaussian':
+        base_kernel = [0.] * half_ks + [1.] + [0.] * half_ks
+        kernel_window = gaussian_filter1d(base_kernel, sigma=sigma) / max(gaussian_filter1d(base_kernel, sigma=sigma))
+    elif kernel == 'triang':
+        kernel_window = triang(ks)
+    else:
+        laplace = lambda x: np.exp(-abs(x) / sigma) / (2. * sigma)
+        kernel_window = list(map(laplace, np.arange(-half_ks, half_ks + 1))) / max(map(laplace, np.arange(-half_ks, half_ks + 1)))
+    return kernel_window
+
+
+def Smooth_Label(Label_new):
+    labels = Label_new
+    for i in range(len(labels)):
+        labels[i] = labels[i] - min(labels)
+    bin_index_per_label = [int(label*10) for label in labels]
+    # print(bin_index_per_label)
+    Nb = max(bin_index_per_label) + 1
+    print(Nb)
+    num_samples_of_bins = dict(Counter(bin_index_per_label))
+    print(num_samples_of_bins)
+    emp_label_dist = [num_samples_of_bins.get(i, 0) for i in range(Nb)]
+    print(emp_label_dist, len(emp_label_dist))
+    eff_label_dist = []
+    eff_label_dist = emp_label_dist
+    eff_num_per_label = [eff_label_dist[bin_idx] for bin_idx in bin_index_per_label]
+    weights = [math.sqrt(np.float32(1 / x)) for x in eff_num_per_label]
+    weights = np.array(weights)
+    print(weights)
+    print(len(weights))
+    return weights
+
+
+def Kcat_predict(Ifeature, Label, weights):
+    kf = KFold(n_splits=5, shuffle=True)
+    All_pre_label = []
+    All_real_label = []
+    for train_index, test_index in kf.split(Ifeature, Label):
+        Train_data, Train_label = Ifeature[train_index], Label[train_index]
+        Test_data, Test_label = Ifeature[test_index], Label[test_index]
+        model = ExtraTreesRegressor()
+        model.fit(Train_data, Train_label, sample_weight=weights[train_index])
+        Pre_label = model.predict(Test_data)
+        All_pre_label.extend(Pre_label)
+        All_real_label.extend(Test_label)
+    res = pd.DataFrame({'Value': All_real_label, 'Predict_Label': All_pre_label})
+    res.to_excel('CSW/Root_Cost_sensitive_Re_weighting_Kcat_5_cv'+'.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    with open(KCAT_DATASET_PATH, 'r') as file:
+        datasets = json.load(file)
+    # print(len(datasets))
+    # datasets = datasets[:50]
+    sequence = [data['Sequence'] for data in datasets]
+    Smiles = [data['Smiles'] for data in datasets]
+    Label = [float(data['Value']) for data in datasets]
+    ECNumber = [data['ECNumber'] for data in datasets]
+    Organism = [data['Organism'] for data in datasets]
+    Substrate = [data['Substrate'] for data in datasets]
+    Type = [data['Type'] for data in datasets]
+    for i in range(len(Label)):
+        if Label[i] == 0:
+            Label[i] = -10000000000
+        else:
+            Label[i] = math.log(Label[i], 10)
+    Label = np.array(Label)
+    print(max(Label), min(Label))
+    # Feature Extractor
+    # smiles_input = smiles_to_vec(Smiles)
+    # sequence_input = Seq_to_vec(sequence)
+    # feature = np.concatenate((smiles_input, sequence_input), axis=1)
+    with open("PreKcat_new/features_17010_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    # Input dataset
+    feature_new = []
+    Label_new = []
+    sequence_new = []
+    Smiles_new = []
+    ECNumber_new = []
+    Organism_new = []
+    Substrate_new = []
+    Type_new = []
+    for i in range(len(Label)):
+        if -10000000000 < Label[i] and '.' not in Smiles[i]:
+            feature_new.append(feature[i])
+            Label_new.append(Label[i])
+            sequence_new.append(sequence[i])
+            Smiles_new.append(Smiles[i])
+            ECNumber_new.append(ECNumber[i])
+            Organism_new.append(Organism[i])
+            Substrate_new.append(Substrate[i])
+            Type_new.append(Type[i])
+    print(len(Label_new), min(Label_new), max(Label_new))
+    feature_new = np.array(feature_new)
+    Label_new = np.array(Label_new)
+    sl_label = [Label_new[i] for i in range(len(Label_new))]
+    weights = Smooth_Label(sl_label)
+    # weights = np.ones([len(Label_new)], dtype=float)
+    # for i in range(len(weights)):
+    #     if Label_new[i] > 5:
+    #         weights[i] = 2
+    # sum_weights = np.sum(weights)
+    # for i in range(len(weights)):
+    #     weights[i] /= sum_weights
+    # Modelling
+    Kcat_predict(feature_new, Label_new, weights)

scripts/UniKP_DMW.py

@@ -0,0 +1,191 @@
+import _bootstrap
+from collections import Counter
+from scipy.ndimage import convolve1d
+from scipy.ndimage import gaussian_filter1d
+from scipy.signal.windows import triang
+import matplotlib.pyplot as plt
+
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def Kcat_predict(Ifeature, Label, weights):
+    kf = KFold(n_splits=5, shuffle=True)
+    All_pre_label = []
+    All_real_label = []
+    for train_index, test_index in kf.split(Ifeature, Label):
+        Train_data, Train_label = Ifeature[train_index], Label[train_index]
+        Test_data, Test_label = Ifeature[test_index], Label[test_index]
+        model = ExtraTreesRegressor()
+        # , sample_weight=weights[train_index]
+        model.fit(Train_data, Train_label, sample_weight=weights[train_index])
+        Pre_label = model.predict(Test_data)
+        All_pre_label.extend(Pre_label)
+        All_real_label.extend(Test_label)
+    res = pd.DataFrame({'Value': All_real_label, 'Predict_Label': All_pre_label})
+    res.to_excel('DMW/DMW_No_Normalize_2_LDS_Kcat_5_cv'+'.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    with open(KCAT_DATASET_PATH, 'r') as file:
+        datasets = json.load(file)
+    # print(len(datasets))
+    # datasets = datasets[:50]
+    sequence = [data['Sequence'] for data in datasets]
+    Smiles = [data['Smiles'] for data in datasets]
+    Label = [float(data['Value']) for data in datasets]
+    ECNumber = [data['ECNumber'] for data in datasets]
+    Organism = [data['Organism'] for data in datasets]
+    Substrate = [data['Substrate'] for data in datasets]
+    Type = [data['Type'] for data in datasets]
+    for i in range(len(Label)):
+        if Label[i] == 0:
+            Label[i] = -10000000000
+        else:
+            Label[i] = math.log(Label[i], 10)
+    Label = np.array(Label)
+    print(max(Label), min(Label))
+    # Feature Extractor
+    # smiles_input = smiles_to_vec(Smiles)
+    # sequence_input = Seq_to_vec(sequence)
+    # feature = np.concatenate((smiles_input, sequence_input), axis=1)
+    with open("PreKcat_new/features_17010_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    # Input dataset
+    feature_new = []
+    Label_new = []
+    sequence_new = []
+    Smiles_new = []
+    ECNumber_new = []
+    Organism_new = []
+    Substrate_new = []
+    Type_new = []
+    for i in range(len(Label)):
+        if -10000000000 < Label[i] and '.' not in Smiles[i]:
+            feature_new.append(feature[i])
+            Label_new.append(Label[i])
+            sequence_new.append(sequence[i])
+            Smiles_new.append(Smiles[i])
+            ECNumber_new.append(ECNumber[i])
+            Organism_new.append(Organism[i])
+            Substrate_new.append(Substrate[i])
+            Type_new.append(Type[i])
+    print(len(Label_new), min(Label_new), max(Label_new))
+    feature_new = np.array(feature_new)
+    Label_new = np.array(Label_new)
+    # sl_label = [Label_new[i] for i in range(len(Label_new))]
+    # weights = Smooth_Label(sl_label)
+    weights = np.ones([len(Label_new)], dtype=float)
+    for i in range(len(weights)):
+        if Label_new[i] > 4:
+            weights[i] = 2
+    # sum_weights = np.sum(weights)
+    # for i in range(len(weights)):
+    #     weights[i] /= sum_weights
+    # Modelling
+    Kcat_predict(feature_new, Label_new, weights)

scripts/UniKP_Km.py

@@ -0,0 +1,147 @@
+import _bootstrap
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from sklearn.model_selection import train_test_split
+import xgboost
+import lightgbm
+from sklearn.model_selection import GridSearchCV
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def Kcat_predict(Ifeature_ini, Label_ini):
+    Myseed = random.randint(0, 1000)
+    print(Myseed)
+    Ifeature, Ifeature_test, Label, Label_test = train_test_split(Ifeature_ini, Label_ini, test_size=0.2,
+                                                                  random_state=Myseed)
+    model = ExtraTreesRegressor()
+    model.fit(Ifeature, Label)
+    Pre_label = model.predict(Ifeature_test)
+    Pcc = np.corrcoef(Label_test, Pre_label)[1][0]
+    RMSE = np.sqrt(mean_squared_error(Label_test, Pre_label))
+    MAE = mean_absolute_error(Label_test, Pre_label)
+    r2 = r2_score(Label_test, Pre_label)
+    print(r2, Pcc, RMSE, MAE)
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    with open(KM_TEST_PATH, 'rb') as file:
+        datasets = pickle.load(file)
+    # datasets = datasets
+    # print(datasets)
+    sequence = datasets['Sequence']
+    smiles = datasets['smiles']
+    Label = datasets['log10_KM']
+    print(len(smiles), len(Label))
+    # smiles_input = smiles_to_vec(smiles)
+    # sequence_input = Seq_to_vec(sequence)
+    # feature = np.concatenate((smiles_input, sequence_input), axis=1)
+    # with open("Km/Km_features_11722_PreKcat.pkl", "wb") as f:
+    #     pickle.dump(feature, f)
+    with open("Km/Km_features_11722_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    feature = np.array(feature)
+    Label = np.array(Label)
+    Kcat_predict(feature, Label)

scripts/UniKP_LDS.py

@@ -0,0 +1,220 @@
+import _bootstrap
+from collections import Counter
+from scipy.ndimage import convolve1d
+from scipy.ndimage import gaussian_filter1d
+from scipy.signal.windows import triang
+import matplotlib.pyplot as plt
+
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def get_lds_kernel_window(kernel, ks, sigma):
+    assert kernel in ['gaussian', 'triang', 'laplace']
+    half_ks = (ks - 1) // 2
+    if kernel == 'gaussian':
+        base_kernel = [0.] * half_ks + [1.] + [0.] * half_ks
+        kernel_window = gaussian_filter1d(base_kernel, sigma=sigma) / max(gaussian_filter1d(base_kernel, sigma=sigma))
+    elif kernel == 'triang':
+        kernel_window = triang(ks)
+    else:
+        laplace = lambda x: np.exp(-abs(x) / sigma) / (2. * sigma)
+        kernel_window = list(map(laplace, np.arange(-half_ks, half_ks + 1))) / max(map(laplace, np.arange(-half_ks, half_ks + 1)))
+    return kernel_window
+
+
+def Smooth_Label(Label_new):
+    labels = Label_new
+    for i in range(len(labels)):
+        labels[i] = labels[i] - min(labels)
+    bin_index_per_label = [int(label*4) for label in labels]
+    # print(bin_index_per_label)
+    Nb = max(bin_index_per_label) + 1
+    num_samples_of_bins = dict(Counter(bin_index_per_label))
+    emp_label_dist = [num_samples_of_bins.get(i, 0) for i in range(Nb)]
+    print(emp_label_dist, len(emp_label_dist))
+    lds_kernel_window = get_lds_kernel_window(kernel='gaussian', ks=3, sigma=1)
+    print(lds_kernel_window)
+    eff_label_dist = convolve1d(np.array(emp_label_dist), weights=lds_kernel_window, mode='constant')
+    print(eff_label_dist, emp_label_dist)
+    eff_num_per_label = [eff_label_dist[bin_idx] for bin_idx in bin_index_per_label]
+    weights = [np.float32(1 / x) for x in eff_num_per_label]
+    weights = np.array(weights)
+    # print(weights)
+    return weights
+
+
+def Kcat_predict(Ifeature, Label, weights):
+    kf = KFold(n_splits=5, shuffle=True)
+    All_pre_label = []
+    All_real_label = []
+    for train_index, test_index in kf.split(Ifeature, Label):
+        Train_data, Train_label = Ifeature[train_index], Label[train_index]
+        Test_data, Test_label = Ifeature[test_index], Label[test_index]
+        model = ExtraTreesRegressor()
+        # , sample_weight=weights[train_index]
+        model.fit(Train_data, Train_label, sample_weight=weights[train_index])
+        Pre_label = model.predict(Test_data)
+        All_pre_label.extend(Pre_label)
+        All_real_label.extend(Test_label)
+    res = pd.DataFrame({'Value': All_real_label, 'Predict_Label': All_pre_label})
+    res.to_excel('LDS/31_LDS_Kcat_5_cv'+'.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    with open(KCAT_DATASET_PATH, 'r') as file:
+        datasets = json.load(file)
+    # print(len(datasets))
+    # datasets = datasets[:50]
+    sequence = [data['Sequence'] for data in datasets]
+    Smiles = [data['Smiles'] for data in datasets]
+    Label = [float(data['Value']) for data in datasets]
+    ECNumber = [data['ECNumber'] for data in datasets]
+    Organism = [data['Organism'] for data in datasets]
+    Substrate = [data['Substrate'] for data in datasets]
+    Type = [data['Type'] for data in datasets]
+    for i in range(len(Label)):
+        if Label[i] == 0:
+            Label[i] = -10000000000
+        else:
+            Label[i] = math.log(Label[i], 10)
+    Label = np.array(Label)
+    print(max(Label), min(Label))
+    # Feature Extractor
+    # smiles_input = smiles_to_vec(Smiles)
+    # sequence_input = Seq_to_vec(sequence)
+    # feature = np.concatenate((smiles_input, sequence_input), axis=1)
+    with open("PreKcat_new/features_17010_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    # feature = feature[:50]
+    # Input dataset
+    feature_new = []
+    Label_new = []
+    sequence_new = []
+    Smiles_new = []
+    ECNumber_new = []
+    Organism_new = []
+    Substrate_new = []
+    Type_new = []
+    for i in range(len(Label)):
+        if -10000000000 < Label[i] and '.' not in Smiles[i]:
+            feature_new.append(feature[i])
+            Label_new.append(Label[i])
+            sequence_new.append(sequence[i])
+            Smiles_new.append(Smiles[i])
+            ECNumber_new.append(ECNumber[i])
+            Organism_new.append(Organism[i])
+            Substrate_new.append(Substrate[i])
+            Type_new.append(Type[i])
+    print(len(Label_new), min(Label_new), max(Label_new))
+    feature_new = np.array(feature_new)
+    Label_new = np.array(Label_new)
+    sl_label = [Label_new[i] for i in range(len(Label_new))]
+    weights = Smooth_Label(sl_label)
+    # Modelling
+    Kcat_predict(feature_new, Label_new, weights)

scripts/UniKP_No_LDS.py

@@ -0,0 +1,237 @@
+import _bootstrap
+from collections import Counter
+from scipy.ndimage import convolve1d
+from scipy.ndimage import gaussian_filter1d
+from scipy.signal.windows import triang
+import matplotlib.pyplot as plt
+
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def get_lds_kernel_window(kernel, ks, sigma):
+    assert kernel in ['gaussian', 'triang', 'laplace']
+    half_ks = (ks - 1) // 2
+    if kernel == 'gaussian':
+        base_kernel = [0.] * half_ks + [1.] + [0.] * half_ks
+        kernel_window = gaussian_filter1d(base_kernel, sigma=sigma) / max(gaussian_filter1d(base_kernel, sigma=sigma))
+    elif kernel == 'triang':
+        kernel_window = triang(ks)
+    else:
+        laplace = lambda x: np.exp(-abs(x) / sigma) / (2. * sigma)
+        kernel_window = list(map(laplace, np.arange(-half_ks, half_ks + 1))) / max(map(laplace, np.arange(-half_ks, half_ks + 1)))
+    return kernel_window
+
+
+def Smooth_Label(Label_new):
+    labels = Label_new
+    for i in range(len(labels)):
+        labels[i] = labels[i] - min(labels)
+    bin_index_per_label = [int(label*4) for label in labels]
+    # print(bin_index_per_label)
+    Nb = max(bin_index_per_label) + 1
+    num_samples_of_bins = dict(Counter(bin_index_per_label))
+    emp_label_dist = [num_samples_of_bins.get(i, 0) for i in range(Nb)]
+    print(emp_label_dist, len(emp_label_dist))
+    plt.figure(figsize=(6,4))
+    # plt.savefig('emp_label_dist.jpg')
+    lds_kernel_window = get_lds_kernel_window(kernel='gaussian', ks=5, sigma=2)
+    print(lds_kernel_window)
+    eff_label_dist = convolve1d(np.array(emp_label_dist[:-10]), weights=lds_kernel_window, mode='constant')
+    # plt.figure(figsize=(6,4))
+    print(eff_label_dist, emp_label_dist[-10:])
+    eff_label_dist = eff_label_dist.tolist()
+    eff_label_dist.extend(emp_label_dist[-10:])
+    # print(eff_label_dist)
+    plt.bar([i for i in range(len(eff_label_dist))], eff_label_dist, color='r')
+    plt.bar([i for i in range(len(emp_label_dist))], emp_label_dist, color='b')
+    plt.savefig('Square_52_label_dist_distribution.jpg')
+    print(eff_label_dist, len(eff_label_dist))
+    eff_num_per_label = [eff_label_dist[bin_idx] for bin_idx in bin_index_per_label]
+    weights = [np.float32(1 / x)*np.float32(1 / x) for x in eff_num_per_label]
+    weights = np.array(weights)
+    # print(weights)
+    return weights
+
+
+def Kcat_predict(Ifeature, Label, weights):
+    for i in range(3):
+        kf = KFold(n_splits=5, shuffle=True)
+        All_pre_label = []
+        All_real_label = []
+        for train_index, test_index in kf.split(Ifeature, Label):
+            Train_data, Train_label = Ifeature[train_index], Label[train_index]
+            Test_data, Test_label = Ifeature[test_index], Label[test_index]
+            model = ExtraTreesRegressor()
+            # , sample_weight=weights[train_index]
+            model.fit(Train_data, Train_label)
+            Pre_label = model.predict(Test_data)
+            All_pre_label.extend(Pre_label)
+            All_real_label.extend(Test_label)
+        res = pd.DataFrame({'Value': All_real_label, 'Predict_Label': All_pre_label})
+        res.to_excel(str(i+1)+'_No_LDS_Kcat_5_cv'+'.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    with open(KCAT_DATASET_PATH, 'r') as file:
+        datasets = json.load(file)
+    # print(len(datasets))
+    # datasets = datasets[:50]
+    sequence = [data['Sequence'] for data in datasets]
+    Smiles = [data['Smiles'] for data in datasets]
+    Label = [float(data['Value']) for data in datasets]
+    ECNumber = [data['ECNumber'] for data in datasets]
+    Organism = [data['Organism'] for data in datasets]
+    Substrate = [data['Substrate'] for data in datasets]
+    Type = [data['Type'] for data in datasets]
+    for i in range(len(Label)):
+        if Label[i] == 0:
+            Label[i] = -10000000000
+        else:
+            Label[i] = math.log(Label[i], 10)
+    Label = np.array(Label)
+    print(max(Label), min(Label))
+    # Feature Extractor
+    # smiles_input = smiles_to_vec(Smiles)
+    # sequence_input = Seq_to_vec(sequence)
+    # feature = np.concatenate((smiles_input, sequence_input), axis=1)
+    with open("PreKcat_new/features_17010_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    # Input dataset
+    feature_new = []
+    Label_new = []
+    sequence_new = []
+    Smiles_new = []
+    ECNumber_new = []
+    Organism_new = []
+    Substrate_new = []
+    Type_new = []
+    for i in range(len(Label)):
+        if -10000000000 < Label[i] and '.' not in Smiles[i]:
+            feature_new.append(feature[i])
+            Label_new.append(Label[i])
+            sequence_new.append(sequence[i])
+            Smiles_new.append(Smiles[i])
+            ECNumber_new.append(ECNumber[i])
+            Organism_new.append(Organism[i])
+            Substrate_new.append(Substrate[i])
+            Type_new.append(Type[i])
+    print(len(Label_new), min(Label_new), max(Label_new))
+    feature_new = np.array(feature_new)
+    Label_new = np.array(Label_new)
+    # sl_label = [Label_new[i] for i in range(len(Label_new))]
+    # weights = Smooth_Label(sl_label)
+    weights = np.ones([len(Label_new)], dtype=float)
+    for i in range(len(weights)):
+        if Label_new[i] > 5:
+            weights[i] = 2
+    sum_weights = np.sum(weights)
+    for i in range(len(weights)):
+        weights[i] /= sum_weights
+    # Modelling
+    Kcat_predict(feature_new, Label_new, weights)

scripts/UniKP_kcat.py

@@ -0,0 +1,180 @@
+import _bootstrap
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            # print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def Kcat_predict(Ifeature, Label, sequence_new, Smiles_new, ECNumber_new, Organism_new, Substrate_new, Type_new):
+    for i in range(10):
+        # Generate training or test set index
+        ALL_index = [j for j in range(len(Ifeature))]
+        train_index = np.array(random.sample(ALL_index, int(len(ALL_index)*0.9)))
+        Training_or_test = []
+        for j in range(len(ALL_index)):
+            if ALL_index[j] in train_index:
+                Training_or_test.append(0)
+            else:
+                Training_or_test.append(1)
+        Training_or_test = np.array(Training_or_test)
+        Train_data, Train_label = Ifeature[train_index], Label[train_index]
+        model = ExtraTreesRegressor()
+        model.fit(Train_data, Train_label)
+        Pre_all_label = model.predict(Ifeature)
+        res = pd.DataFrame({'sequence': sequence_new, 'smiles': Smiles_new, 'ECNumber': ECNumber_new,
+                            'Organism': Organism_new, 'Substrate': Substrate_new, 'Type': Type_new,
+                            'Label': Label, 'Predict_Label': Pre_all_label, 'Training or test': Training_or_test})
+        res.to_excel('PreKcat_new/'+str(i+1)+'_all_samples_metrics.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    with open(KCAT_DATASET_PATH, 'r') as file:
+        datasets = json.load(file)
+    # print(len(datasets))
+    sequence = [data['Sequence'] for data in datasets]
+    Smiles = [data['Smiles'] for data in datasets]
+    Label = [float(data['Value']) for data in datasets]
+    ECNumber = [data['ECNumber'] for data in datasets]
+    Organism = [data['Organism'] for data in datasets]
+    Substrate = [data['Substrate'] for data in datasets]
+    Type = [data['Type'] for data in datasets]
+    for i in range(len(Label)):
+        if Label[i] == 0:
+            Label[i] = -10000000000
+        else:
+            Label[i] = math.log(Label[i], 10)
+    print(max(Label), min(Label))
+    # Feature Extractor
+    smiles_input = smiles_to_vec(Smiles)
+    sequence_input = Seq_to_vec(sequence)
+    feature = np.concatenate((smiles_input, sequence_input), axis=1)
+    with open("PreKcat_new/features_16838_PreKcat.pkl", "wb") as f:
+        pickle.dump(feature, f)
+    Label = np.array(Label)
+    # Input dataset
+    feature_new = []
+    Label_new = []
+    sequence_new = []
+    Smiles_new = []
+    ECNumber_new = []
+    Organism_new = []
+    Substrate_new = []
+    Type_new = []
+    for i in range(len(Label)):
+        if -10000000000 < Label[i] and '.' not in Smiles[i]:
+            feature_new.append(feature[i])
+            Label_new.append(Label[i])
+            sequence_new.append(sequence[i])
+            Smiles_new.append(Smiles[i])
+            ECNumber_new.append(ECNumber[i])
+            Organism_new.append(Organism[i])
+            Substrate_new.append(Substrate[i])
+            Type_new.append(Type[i])
+    print(len(Label_new), min(Label_new), max(Label_new))
+    Label_new = np.array(Label_new)
+    feature_new = np.array(feature_new)
+    # Modelling
+    Kcat_predict(feature_new, Label_new, sequence_new, Smiles_new, ECNumber_new,
+                Organism_new, Substrate_new, Type_new)

scripts/UniKP_kcat_Km.py

@@ -0,0 +1,144 @@
+import _bootstrap
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from sklearn.model_selection import train_test_split
+import xgboost
+import lightgbm
+from sklearn.model_selection import GridSearchCV
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def Kcat_predict(Ifeature, Label):
+    kf = KFold(n_splits=5, shuffle=True)
+    All_pre_label = []
+    All_real_label = []
+    for train_index, test_index in kf.split(Ifeature, Label):
+        Train_data, Train_label = Ifeature[train_index], Label[train_index]
+        Test_data, Test_label = Ifeature[test_index], Label[test_index]
+        model = ExtraTreesRegressor()
+        model.fit(Train_data, Train_label)
+        Pre_label = model.predict(Test_data)
+        All_pre_label.extend(Pre_label)
+        All_real_label.extend(Test_label)
+    res = pd.DataFrame({'Value': All_real_label, 'Predict_Label': All_pre_label})
+    res.to_excel('Kcat_Km/Kcat_Km_5_cv.xlsx')
+
+
+if __name__ == '__main__':
+    res = np.array(pd.read_excel(KCAT_KM_SAMPLES_PATH, sheet_name='main')).T
+    Smiles = res[1]
+    sequences = res[2]
+    Value = res[0]
+    for i in range(len(Value)):
+        Value[i] = math.log(Value[i], 10)
+    print(max(Value), min(Value))
+    smiles_input = smiles_to_vec(Smiles)
+    sequence_input = Seq_to_vec(sequences)
+    feature = np.concatenate((smiles_input, sequence_input), axis=1)
+    with open("Kcat_Km/Kcat_Km_features_910.pkl", "wb") as f:
+        pickle.dump(feature, f)
+    feature = np.array(feature)
+    Label = np.array(Value)
+    Kcat_predict(feature, Label)

scripts/UniKP_kcat_model.py

@@ -0,0 +1,63 @@
+import _bootstrap
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from sklearn.model_selection import KFold
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+def Kcat_predict(Ifeature, Label):
+    for i in range(5):
+        model = ExtraTreesRegressor()
+        model.fit(Ifeature, Label)
+        with open('PreKcat_new/'+str(i)+"_model.pkl", "wb") as f:
+            pickle.dump(model, f)
+
+
+if __name__ == '__main__':
+    with open(KCAT_DATASET_PATH, 'r') as file:
+        datasets = json.load(file)
+    # print(len(datasets))
+    Label = [float(data['Value']) for data in datasets]
+    Smiles = [data['Smiles']for data in datasets]
+    for i in range(len(Label)):
+        if Label[i] == 0:
+            Label[i] = -10000000000
+        else:
+            Label[i] = math.log(Label[i], 10)
+    with open("PreKcat_new/features_16838_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    Label = np.array(Label)
+    Label_new = []
+    feature_new = []
+    for i in range(len(Label)):
+        if -10000000000 < Label[i] and '.' not in Smiles[i]:
+            Label_new.append(Label[i])
+            feature_new.append(feature[i])
+    print(len(Label_new))
+    Label_new = np.array(Label_new)
+    feature_new = np.array(feature_new)
+    Kcat_predict(feature_new, Label_new)

scripts/UniKP_pH.py

@@ -0,0 +1,145 @@
+import _bootstrap
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def Kcat_predict(Ifeature, Label):
+    kf = KFold(n_splits=5, shuffle=True)
+    All_pre_label = []
+    All_real_label = []
+    for train_index, test_index in kf.split(Ifeature, Label):
+        Train_data, Train_label = Ifeature[train_index], Label[train_index]
+        Test_data, Test_label = Ifeature[test_index], Label[test_index]
+        model = ExtraTreesRegressor()
+        model.fit(Train_data, Train_label)
+        Pre_label = model.predict(Test_data)
+        All_pre_label.extend(Pre_label)
+        All_real_label.extend(Test_label)
+    res = pd.DataFrame({'Value': All_real_label, 'Predict_Label': All_pre_label})
+    res.to_excel('pH/pH_Kcat_5_cv.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    database = np.array(pd.read_excel(PH_SMILES_PATH)).T
+    sequence = database[1]
+    smiles = database[3]
+    pH = database[5].reshape([len(smiles), 1])
+    Label = database[4]
+    for i in range(len(Label)):
+        Label[i] = math.log(Label[i], 10)
+    print(max(Label), min(Label))
+    # Feature Extractor
+    # smiles_input = smiles_to_vec(smiles)
+    # sequence_input = Seq_to_vec(sequence)
+    # print(sequence_input.shape, sequence_input.shape, pH.shape)
+    # feature = np.concatenate((smiles_input, sequence_input, pH), axis=1)
+    # with open("pH/features_636_pH_PreKcat.pkl", "wb") as f:
+    #     pickle.dump(feature, f)
+    with open("pH/features_636_pH_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    # Modelling
+    Kcat_predict(feature, Label)

scripts/UniKP_pH_s2.py

@@ -0,0 +1,186 @@
+import _bootstrap
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+from sklearn.linear_model import LinearRegression
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def Kcat_predict(feature, pH, sequence, smiles, Label):
+    # Generate index
+    Train_Validation_index = random.sample(range(len(feature)), int(len(feature)*0.8))
+    Test_index = []
+    for i in range(len(feature)):
+        if i not in Train_Validation_index:
+            Test_index.append(i)
+    Validation_index = random.sample(Train_Validation_index, int(len(Train_Validation_index)*0.2))
+    Train_index = []
+    for i in range(len(feature)):
+        if i not in Validation_index and i not in Test_index:
+            Train_index.append(i)
+    print(len(Train_index), len(Validation_index), len(Test_index))
+    Training_Validation_Test = []
+    for i in range(len(feature)):
+        if i in Train_index:
+            Training_Validation_Test.append(0)
+        elif i in Validation_index:
+            Training_Validation_Test.append(1)
+        else:
+            Training_Validation_Test.append(2)
+    Train_index = np.array(Train_index)
+    Validation_index = np.array(Validation_index)
+    Test_index = np.array(Test_index)
+    print(Train_index.shape, Validation_index.shape, Test_index.shape)
+    # First model
+    print(feature[Train_index].shape, pH[Train_index].shape)
+    model_1_input = np.concatenate((feature[Train_index], pH[Train_index]), axis=1)
+    model_first = ExtraTreesRegressor()
+    model_first.fit(model_1_input, Label[Train_index])
+    # Second model
+    with open("PreKcat_new/0_model.pkl", "rb") as f:
+        model_base = pickle.load(f)
+    Kcat_baseline = model_base.predict(feature[Validation_index]).reshape([len(Validation_index), 1])
+    model_1_2_input = np.concatenate((feature[Validation_index], pH[Validation_index]), axis=1)
+    Kcat_calibrated = model_first.predict(model_1_2_input).reshape([len(Validation_index), 1])
+    kcat_fused = np.concatenate((Kcat_baseline, Kcat_calibrated), axis=1)
+    model_second = LinearRegression()
+    model_second.fit(kcat_fused, Label[Validation_index])
+    # Final prediction
+    model_1_3_input = np.concatenate((feature, pH), axis=1)
+    Kcat_calibrated_3 = model_first.predict(model_1_3_input).reshape([len(feature), 1])
+    Kcat_baseline_3 = model_base.predict(feature).reshape([len(feature), 1])
+    kcat_fused_3 = np.concatenate((Kcat_baseline_3, Kcat_calibrated_3), axis=1)
+    Predicted_value = model_second.predict(kcat_fused_3).reshape([len(feature)])
+    Training_Validation_Test = np.array(Training_Validation_Test).reshape([len(feature)])
+    pH = np.array(pH).reshape([len(Label)])
+    Kcat_baseline_3 = np.array(Kcat_baseline_3).reshape([len(feature)])
+    Kcat_calibrated_3 = np.array(Kcat_calibrated_3).reshape([len(feature)])
+    print(Training_Validation_Test.shape)
+    # save
+    res = pd.DataFrame({'Value': Label,
+                        'sequence': sequence,
+                        'smiles': smiles,
+                        'pH': pH,
+                        'Prediction_first_base': Kcat_baseline_3,
+                        'Prediction_first_pH': Kcat_calibrated_3,
+                        'Prediction_second': Predicted_value,
+                        'Training_Validation_Test': Training_Validation_Test})
+    res.to_excel('pH/s2_pH_Kcat.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    database = np.array(pd.read_excel(PH_SMILES_PATH)).T
+    sequence = database[1]
+    smiles = database[3]
+    pH = database[5]
+    Label = database[4]
+    for i in range(len(Label)):
+        Label[i] = math.log(Label[i], 10)
+    print(max(Label), min(Label))
+    pH = np.array(pH).reshape([len(Label), 1])
+    with open("pH/features_636_pH_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    # Modelling
+    Kcat_predict(feature, pH, sequence, smiles, Label)

scripts/UniKP_temperature.py

@@ -0,0 +1,145 @@
+import _bootstrap
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def Kcat_predict(Ifeature, Label):
+    kf = KFold(n_splits=5, shuffle=True)
+    All_pre_label = []
+    All_real_label = []
+    for train_index, test_index in kf.split(Ifeature, Label):
+        Train_data, Train_label = Ifeature[train_index], Label[train_index]
+        Test_data, Test_label = Ifeature[test_index], Label[test_index]
+        model = ExtraTreesRegressor()
+        model.fit(Train_data, Train_label)
+        Pre_label = model.predict(Test_data)
+        All_pre_label.extend(Pre_label)
+        All_real_label.extend(Test_label)
+    res = pd.DataFrame({'Value': All_real_label, 'Predict_Label': All_pre_label})
+    res.to_excel('degree/degree_Kcat_5_cv.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    database = np.array(pd.read_excel(DEGREE_SMILES_PATH)).T
+    sequence = database[1]
+    smiles = database[3]
+    pH = database[5].reshape([len(smiles), 1])
+    Label = database[4]
+    for i in range(len(Label)):
+        Label[i] = math.log(Label[i], 10)
+    print(max(Label), min(Label))
+    # Feature Extractor
+    # smiles_input = smiles_to_vec(smiles)
+    # sequence_input = Seq_to_vec(sequence)
+    # print(sequence_input.shape, sequence_input.shape, pH.shape)
+    # feature = np.concatenate((smiles_input, sequence_input), axis=1)
+    # with open("degree/features_572_degree_PreKcat.pkl", "wb") as f:
+    #     pickle.dump(feature, f)
+    with open("degree/features_572_degree_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    # Modelling
+    Kcat_predict(feature, Label)

scripts/UniKP_temperature_s2.py

@@ -0,0 +1,186 @@
+import _bootstrap
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+import json
+from transformers import T5EncoderModel, T5Tokenizer
+import re
+import gc
+from sklearn import metrics
+from sklearn.ensemble import ExtraTreesRegressor
+from sklearn.linear_model import LinearRegression
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import KFold
+from sklearn.metrics import mean_squared_error, mean_absolute_error
+from sklearn.metrics import r2_score
+from sklearn.model_selection import train_test_split
+import random
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+
+def smiles_to_vec(Smiles):
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+    mask_index = 4
+    vocab = WordVocab.load_vocab(VOCAB_PATH)
+    def get_inputs(sm):
+        seq_len = 220
+        sm = sm.split()
+        if len(sm)>218:
+            print('SMILES is too long ({:d})'.format(len(sm)))
+            sm = sm[:109]+sm[-109:]
+        ids = [vocab.stoi.get(token, unk_index) for token in sm]
+        ids = [sos_index] + ids + [eos_index]
+        seg = [1]*len(ids)
+        padding = [pad_index]*(seq_len - len(ids))
+        ids.extend(padding), seg.extend(padding)
+        return ids, seg
+    def get_array(smiles):
+        x_id, x_seg = [], []
+        for sm in smiles:
+            a,b = get_inputs(sm)
+            x_id.append(a)
+            x_seg.append(b)
+        return torch.tensor(x_id), torch.tensor(x_seg)
+    trfm = TrfmSeq2seq(len(vocab), 256, len(vocab), 4)
+    trfm.load_state_dict(torch.load(TRFM_PATH))
+    trfm.eval()
+    x_split = [split(sm) for sm in Smiles]
+    xid, xseg = get_array(x_split)
+    X = trfm.encode(torch.t(xid))
+    return X
+
+
+def Seq_to_vec(Sequence):
+    for i in range(len(Sequence)):
+        if len(Sequence[i]) > 1000:
+            Sequence[i] = Sequence[i][:500] + Sequence[i][-500:]
+    sequences_Example = []
+    for i in range(len(Sequence)):
+        zj = ''
+        for j in range(len(Sequence[i]) - 1):
+            zj += Sequence[i][j] + ' '
+        zj += Sequence[i][-1]
+        sequences_Example.append(zj)
+    tokenizer = T5Tokenizer.from_pretrained(PROT_T5_MODEL, do_lower_case=False)
+    model = T5EncoderModel.from_pretrained(PROT_T5_MODEL)
+    gc.collect()
+    print(torch.cuda.is_available())
+    # 'cuda:0' if torch.cuda.is_available() else
+    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
+    model = model.to(device)
+    model = model.eval()
+    features = []
+    for i in range(len(sequences_Example)):
+        print('For sequence ', str(i+1))
+        sequences_Example_i = sequences_Example[i]
+        sequences_Example_i = [re.sub(r"[UZOB]", "X", sequences_Example_i)]
+        ids = tokenizer.batch_encode_plus(sequences_Example_i, add_special_tokens=True, padding=True)
+        input_ids = torch.tensor(ids['input_ids']).to(device)
+        attention_mask = torch.tensor(ids['attention_mask']).to(device)
+        with torch.no_grad():
+            embedding = model(input_ids=input_ids, attention_mask=attention_mask)
+        embedding = embedding.last_hidden_state.cpu().numpy()
+        for seq_num in range(len(embedding)):
+            seq_len = (attention_mask[seq_num] == 1).sum()
+            seq_emd = embedding[seq_num][:seq_len - 1]
+            features.append(seq_emd)
+    features_normalize = np.zeros([len(features), len(features[0][0])], dtype=float)
+    for i in range(len(features)):
+        for k in range(len(features[0][0])):
+            for j in range(len(features[i])):
+                features_normalize[i][k] += features[i][j][k]
+            features_normalize[i][k] /= len(features[i])
+    return features_normalize
+
+
+def Kcat_predict(feature, pH, sequence, smiles, Label):
+    # Generate index
+    Train_Validation_index = random.sample(range(len(feature)), int(len(feature)*0.8))
+    Test_index = []
+    for i in range(len(feature)):
+        if i not in Train_Validation_index:
+            Test_index.append(i)
+    Validation_index = random.sample(Train_Validation_index, int(len(Train_Validation_index)*0.2))
+    Train_index = []
+    for i in range(len(feature)):
+        if i not in Validation_index and i not in Test_index:
+            Train_index.append(i)
+    print(len(Train_index), len(Validation_index), len(Test_index))
+    Training_Validation_Test = []
+    for i in range(len(feature)):
+        if i in Train_index:
+            Training_Validation_Test.append(0)
+        elif i in Validation_index:
+            Training_Validation_Test.append(1)
+        else:
+            Training_Validation_Test.append(2)
+    Train_index = np.array(Train_index)
+    Validation_index = np.array(Validation_index)
+    Test_index = np.array(Test_index)
+    print(Train_index.shape, Validation_index.shape, Test_index.shape)
+    # First model
+    print(feature[Train_index].shape, pH[Train_index].shape)
+    model_1_input = np.concatenate((feature[Train_index], pH[Train_index]), axis=1)
+    model_first = ExtraTreesRegressor()
+    model_first.fit(model_1_input, Label[Train_index])
+    # Second model
+    with open("PreKcat_new/0_model.pkl", "rb") as f:
+        model_base = pickle.load(f)
+    Kcat_baseline = model_base.predict(feature[Validation_index]).reshape([len(Validation_index), 1])
+    model_1_2_input = np.concatenate((feature[Validation_index], pH[Validation_index]), axis=1)
+    Kcat_calibrated = model_first.predict(model_1_2_input).reshape([len(Validation_index), 1])
+    kcat_fused = np.concatenate((Kcat_baseline, Kcat_calibrated), axis=1)
+    model_second = LinearRegression()
+    model_second.fit(kcat_fused, Label[Validation_index])
+    # Final prediction
+    model_1_3_input = np.concatenate((feature, pH), axis=1)
+    Kcat_calibrated_3 = model_first.predict(model_1_3_input).reshape([len(feature), 1])
+    Kcat_baseline_3 = model_base.predict(feature).reshape([len(feature), 1])
+    kcat_fused_3 = np.concatenate((Kcat_baseline_3, Kcat_calibrated_3), axis=1)
+    Predicted_value = model_second.predict(kcat_fused_3).reshape([len(feature)])
+    Training_Validation_Test = np.array(Training_Validation_Test).reshape([len(feature)])
+    pH = np.array(pH).reshape([len(Label)])
+    Kcat_baseline_3 = np.array(Kcat_baseline_3).reshape([len(feature)])
+    Kcat_calibrated_3 = np.array(Kcat_calibrated_3).reshape([len(feature)])
+    print(Training_Validation_Test.shape)
+    # save
+    res = pd.DataFrame({'Value': Label,
+                        'sequence': sequence,
+                        'smiles': smiles,
+                        'pH': pH,
+                        'Prediction_first_base': Kcat_baseline_3,
+                        'Prediction_first_pH': Kcat_calibrated_3,
+                        'Prediction_second': Predicted_value,
+                        'Training_Validation_Test': Training_Validation_Test})
+    res.to_excel('degree/s2_degree_Kcat.xlsx')
+
+
+if __name__ == '__main__':
+    # Dataset Load
+    database = np.array(pd.read_excel(DEGREE_SMILES_PATH)).T
+    sequence = database[1]
+    smiles = database[3]
+    pH = database[5]
+    Label = database[4]
+    for i in range(len(Label)):
+        Label[i] = math.log(Label[i], 10)
+    print(max(Label), min(Label))
+    pH = np.array(pH).reshape([len(Label), 1])
+    with open("degree/features_572_degree_PreKcat.pkl", "rb") as f:
+        feature = pickle.load(f)
+    # Modelling
+    Kcat_predict(feature, pH, sequence, smiles, Label)

scripts/_bootstrap.py

@@ -0,0 +1,8 @@
+from pathlib import Path
+import sys
+
+
+ROOT_DIR = Path(__file__).resolve().parents[1]
+root = str(ROOT_DIR)
+if root not in sys.path:
+    sys.path.insert(0, root)

scripts/build_corpus.py

@@ -0,0 +1,23 @@
+import _bootstrap
+import argparse
+import pandas as pd
+from tqdm import tqdm
+
+from model.utils import split
+
+def main():
+    parser = argparse.ArgumentParser(description='Build a corpus file')
+    parser.add_argument('--in_path', '-i', type=str, default='data/chembl24_bert_train.csv', help='input file')
+    parser.add_argument('--out_path', '-o', type=str, default='data/chembl24_corpus.txt', help='output file')
+    args = parser.parse_args()
+
+    smiles = pd.read_csv(args.in_path)['first'].values
+    with open(args.out_path, 'a') as f:
+        for sm in tqdm(smiles):
+            f.write(split(sm)+'\n')
+    print('Built a corpus file!')
+
+if __name__=='__main__':
+    main()
+
+

scripts/demo_kcat.py

@@ -0,0 +1,482 @@
+import _bootstrap
+import torch
+from model.build_vocab import WordVocab
+from model.pretrain_trfm import TrfmSeq2seq
+from model.utils import split
+
+from transformers import T5EncoderModel, T5Tokenizer
+
+import re
+import gc
+import numpy as np
+import pandas as pd
+import pickle
+import math
+from project_paths import (
+
+    DEGREE_SMILES_PATH, KCAT_DATASET_PATH, KCAT_KM_SAMPLES_PATH, KM_TEST_PATH,
+
+    PH_SMILES_PATH, PROT_T5_MODEL, TRFM_PATH, UNIKP_MODEL_DIR, VOCAB_PATH,
+
+)
+
+
+# ============================
+# SMILES -> vector
+# ============================
+
+def smiles_to_vec(Smiles):
+
+    pad_index = 0
+    unk_index = 1
+    eos_index = 2
+    sos_index = 3
+
+    vocab = WordVocab.load_vocab(
+        VOCAB_PATH
+    )
+
+
+    def get_inputs(sm):
+
+        seq_len = 220
+
+        sm = sm.split()
+
+        if len(sm) > 218:
+            print(
+                "SMILES too long:",
+                len(sm)
+            )
+            sm = sm[:109] + sm[-109:]
+
+
+        ids = [
+            vocab.stoi.get(
+                token,
+                unk_index
+            )
+            for token in sm
+        ]
+
+
+        ids = (
+            [sos_index]
+            +
+            ids
+            +
+            [eos_index]
+        )
+
+
+        seg = [1] * len(ids)
+
+
+        padding = [
+            pad_index
+        ] * (
+            seq_len - len(ids)
+        )
+
+
+        ids.extend(padding)
+        seg.extend(padding)
+
+
+        return ids, seg
+
+
+
+    def get_array(smiles):
+
+        x_id = []
+        x_seg = []
+
+        for sm in smiles:
+
+            a, b = get_inputs(sm)
+
+            x_id.append(a)
+            x_seg.append(b)
+
+
+        return (
+            torch.tensor(x_id),
+            torch.tensor(x_seg)
+        )
+
+
+
+    # load SMILES Transformer
+
+    trfm = TrfmSeq2seq(
+        len(vocab),
+        256,
+        len(vocab),
+        4
+    )
+
+
+    trfm.load_state_dict(
+        torch.load(
+            TRFM_PATH,
+            map_location="cpu"
+        )
+    )
+
+
+    trfm.eval()
+
+
+
+    x_split = [
+        split(sm)
+        for sm in Smiles
+    ]
+
+
+    xid, xseg = get_array(
+        x_split
+    )
+
+
+    X = trfm.encode(
+        torch.t(xid)
+    )
+
+
+    return X
+
+
+
+# ============================
+# Protein sequence -> vector
+# ============================
+
+
+def Seq_to_vec(Sequence):
+
+
+    for i in range(len(Sequence)):
+
+        if len(Sequence[i]) > 1000:
+
+            Sequence[i] = (
+                Sequence[i][:500]
+                +
+                Sequence[i][-500:]
+            )
+
+
+
+    sequences_Example = []
+
+
+    for seq in Sequence:
+
+        spaced = ""
+
+        for aa in seq[:-1]:
+
+            spaced += aa + " "
+
+        spaced += seq[-1]
+
+
+        sequences_Example.append(
+            spaced
+        )
+
+
+
+    # load ProtT5
+
+    tokenizer = T5Tokenizer.from_pretrained(
+        PROT_T5_MODEL,
+        do_lower_case=False
+    )
+
+
+    model = T5EncoderModel.from_pretrained(
+        PROT_T5_MODEL
+    )
+
+
+    gc.collect()
+
+
+    device = torch.device(
+        "cuda:0"
+        if torch.cuda.is_available()
+        else "cpu"
+    )
+
+
+    print(
+        "Using device:",
+        device
+    )
+
+
+    model = model.to(device)
+
+    model.eval()
+
+
+
+    features = []
+
+
+
+    for i, seq in enumerate(sequences_Example):
+
+        print(
+            "Processing protein:",
+            i + 1
+        )
+
+
+        seq = [
+            re.sub(
+                r"[UZOB]",
+                "X",
+                seq
+            )
+        ]
+
+
+        ids = tokenizer.batch_encode_plus(
+            seq,
+            add_special_tokens=True,
+            padding=True
+        )
+
+
+        input_ids = torch.tensor(
+            ids["input_ids"]
+        ).to(device)
+
+
+        attention_mask = torch.tensor(
+            ids["attention_mask"]
+        ).to(device)
+
+
+
+        with torch.no_grad():
+
+            embedding = model(
+                input_ids=input_ids,
+                attention_mask=attention_mask
+            )
+
+
+
+        embedding = (
+            embedding
+            .last_hidden_state
+            .cpu()
+            .numpy()
+        )
+
+
+
+        seq_len = (
+            attention_mask[0] == 1
+        ).sum()
+
+
+        seq_embedding = (
+            embedding[0]
+            [:seq_len-1]
+        )
+
+
+        features.append(
+            seq_embedding
+        )
+
+
+
+    # mean pooling
+
+    features_normalize = np.zeros(
+        [
+            len(features),
+            len(features[0][0])
+        ],
+        dtype=float
+    )
+
+
+
+    for i in range(len(features)):
+
+        for k in range(
+            len(features[0][0])
+        ):
+
+            for j in range(
+                len(features[i])
+            ):
+
+                features_normalize[i][k] += (
+                    features[i][j][k]
+                )
+
+
+            features_normalize[i][k] /= (
+                len(features[i])
+            )
+
+
+    return features_normalize
+
+
+
+
+# ============================
+# Main
+# ============================
+
+
+if __name__ == "__main__":
+
+
+    # 示例蛋白序列
+    sequences = [
+        "MEDIPDTSRPPLKYVKGIPLIKYFAEALESLQDFQAQPDDLLISTYPKSGTTWVSEILDMIYQDGDVEKCRRAPVFIRVPFLEFKAPGIPTGLEVLKDTPAPRLIKTHLPLALLPQTLLDQKVKVVYVARNAKDVAVSYYHFYRMAKVHPDPDTWDSFLEKFMAGEVSYGSWYQHVQEWWELSHTHPVLYLFYEDMKENPKREIQKILKFVGRSLPEETVDLIVQHTSFKEMKNNSMANYTTLSPDIMDHSISAFMRKGISGDWKTTFTVAQNERFDADYAKKMEGCGLSFRTQL"
+    ]
+
+
+    # 示例底物 SMILES
+
+    Smiles = [
+        "OC1=CC=C(C[C@@H](C(O)=O)N)C=C1"
+    ]
+
+
+
+    print(
+        "Extracting protein embedding..."
+    )
+
+    seq_vec = Seq_to_vec(
+        sequences
+    )
+
+
+    print(
+        "Extracting SMILES embedding..."
+    )
+
+    smiles_vec = smiles_to_vec(
+        Smiles
+    )
+
+
+    print(
+        "Protein vector:",
+        seq_vec.shape
+    )
+
+
+    print(
+        "SMILES vector:",
+        smiles_vec.shape
+    )
+
+
+
+    # concatenate
+
+    fused_vector = np.concatenate(
+        (
+            smiles_vec,
+            seq_vec
+        ),
+        axis=1
+    )
+
+
+
+    print(
+        "Fused vector:",
+        fused_vector.shape
+    )
+
+
+
+    # load UniKP model
+
+    with open(
+        UNIKP_MODEL_DIR / "UniKP for kcat.pkl",
+        "rb"
+    ) as f:
+
+        model = pickle.load(f)
+
+
+
+    # prediction
+
+    pred = model.predict(
+        fused_vector
+    )
+
+
+    # log10 inverse transform
+
+    pred_value = [
+        math.pow(
+            10,
+            x
+        )
+        for x in pred
+    ]
+
+
+
+    print("\n========== Result ==========")
+
+    print(
+        "Protein:",
+        sequences[0][:50],
+        "..."
+    )
+
+    print(
+        "SMILES:",
+        Smiles[0]
+    )
+
+
+    print(
+        "Predicted kcat:",
+        pred_value[0],
+        "s^-1"
+    )
+
+
+
+    # save
+
+    result = pd.DataFrame(
+        {
+            "sequence": sequences,
+            "SMILES": Smiles,
+            "kcat": pred_value
+        }
+    )
+
+
+    result.to_excel(
+        "UniKP_kcat_prediction.xlsx",
+        index=False
+    )
+
+
+    print(
+        "\nSaved:"
+        " UniKP_kcat_prediction.xlsx"
+    )

scripts/project_paths.py

@@ -0,0 +1,23 @@
+from pathlib import Path
+
+
+ROOT_DIR = Path(__file__).resolve().parents[1]
+CONF_DIR = ROOT_DIR / "conf"
+DATASETS_DIR = CONF_DIR / "datasets"
+FIGURES_DIR = CONF_DIR / "Figures"
+WEIGHT_DIR = ROOT_DIR / "weight"
+UNIKP_MODEL_DIR = WEIGHT_DIR / "UniKP_model"
+PROT_T5_DIR = WEIGHT_DIR / "prot_t5_xl_uniref50"
+PROT_T5_MODEL = str(PROT_T5_DIR if PROT_T5_DIR.exists() else "prot_t5_xl_uniref50")
+
+VOCAB_PATH = WEIGHT_DIR / "vocab.pkl"
+TRFM_PATH = WEIGHT_DIR / "trfm_12_23000.pkl"
+KCAT_DATASET_PATH = DATASETS_DIR / "Kcat_combination_0918_wildtype_mutant.json"
+KM_TEST_PATH = DATASETS_DIR / "Km_test_11722.pkl"
+KCAT_KM_SAMPLES_PATH = DATASETS_DIR / "kcat_km_samples.xlsx"
+PH_SMILES_PATH = DATASETS_DIR / "Generated_pH_unified_smiles_636.xlsx"
+DEGREE_SMILES_PATH = DATASETS_DIR / "Generated_degree_unified_smiles_572.xlsx"
+
+
+def output_path(*parts):
+    return ROOT_DIR.joinpath(*parts)

scripts/smi2csv.py

@@ -0,0 +1,36 @@
+import argparse
+import numpy as np
+import pandas as pd
+
+def main():
+    parser = argparse.ArgumentParser(description='Convert smi file to csv file')
+    parser.add_argument('--in_path', '-i', type=str, default='data/GDB/GDB08.smi', help='input file')
+    parser.add_argument('--out_path_1', '-o1', type=str, default='data/gdb08_bert_train.csv', help='output file (train)')
+    parser.add_argument('--out_path_2', '-o2', type=str, default='data/gdb08_bert_test.csv', help='output file (test)')
+    parser.add_argument('--max', '-m', type=int, default=2e6, help='Maximum number of molecules')
+    args = parser.parse_args()
+    print('Input file: {}'.format(args.in_path))
+    print('Start preprocessing')
+
+    smiles = []
+    with open(args.in_path) as f:
+        lines = f.readlines()
+    for l in lines:
+        smiles.append(l.replace('\n', ''))
+    del lines
+    smiles = np.array(smiles)
+    N = len(smiles)
+    print('The dataset contains {} molecules'.format(N))
+    
+    rands = np.random.choice(N, min(N,args.max), replace=False)
+    smiles_train = smiles[rands[:N//2]]
+    df_train = pd.DataFrame(data=smiles_train, columns=['canonical_smiles'])
+    df_train.to_csv(args.out_path_1, index=False)
+    del smiles_train, df_train
+    smiles_test = smiles[rands[N//2:]]
+    df_test = pd.DataFrame(data=smiles_test, columns=['canonical_smiles'])
+    df_test.to_csv(args.out_path_2, index=False)
+    print('Each set contains {} molecules'.format(N//2))
+
+if __name__=='__main__':
+    main()

weight/UniKP_model/.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

weight/UniKP_model/README.md

@@ -0,0 +1,3 @@
+---
+license: gpl-3.0
+---

weight/UniKP_model/UniKP for Km.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0b8d597cf4e5f73431980950cb89423415a260dbbf9be0bb1d8810712bf9c07
+size 147957236

weight/UniKP_model/UniKP for kcat.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe90811273401698a2c25ab32959f13c0087a14feb8ca310cf4b44dcad819fd5
+size 205501172

weight/UniKP_model/UniKP for kcat_Km.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f81f8715a87790c542023b1bab1da6055a60b9db22c20c04a8846d9b09ba844
+size 11476980

weight/trfm_12_23000.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b56c8c05d048e7c7d143c4e3ba2bc6f76e5eda2358798cf636210406a700eb2
+size 22128521

weight/vocab.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21a66c850a3222547ec0fbd30c05fe587d66d22d3de2ee2195c58250fe486fb7
+size 1446