| --- |
| license: cc-by-4.0 |
| datasets: |
| - J-MADRAL/AmazonESCI |
| - J-MADRAL/SearchESCI |
| - J-MADRAL/TREC_Product_Search_2023 |
| - J-MADRAL/TrainingData |
| language: |
| - en |
| metrics: |
| - recall |
| - trec_eval |
| base_model: |
| - google-bert/bert-base-uncased |
| pipeline_tag: text-ranking |
| tags: |
| - ProductSearch |
| - ReviewSearch |
| library_name: transformers |
| --- |
| |
|
|
| **J-BiBERT** is a BERT-sized dense retriever initialized from [BERT](https://huggingface.co/google-bert/bert-base-uncased) public checkpoint, |
| further pre-trained on e-commerce product and review data, and fine-tuned jointly on product and review search retrieval tasks on the |
| [Amazon ESCI](https://huggingface.co/datasets/J-MADRAL/AmazonESCI) and [Search ESCI](https://huggingface.co/datasets/J-MADRAL/SearchESCI) datasets. |
| We use symmetric encoder architecture, with a single shared encoder for both queries and products. |
| The similarity function is *dot product*. |
|
|
|
|
| ## Paper and Repository ## |
|
|
| J-BiBERT has been described in the *Multi-Aspect Joint Retrieval for E-Commerce: Bridging Product Catalogs and Customer Reviews* paper. |
| The associated GitHub repository is available at [https://anonymous.4open.science/r/J-MADRAL-C4CC](https://anonymous.4open.science/r/J-MADRAL-C4CC). |
|
|
|
|
| ## Usage (HuggingFace Transformers) ## |
|
|
| Using the model directly in HuggingFace transformers requires additional code available in the [repository](https://anonymous.4open.science/r/J-MADRAL-C4CC). |
|
|
| ```python |
| import modeling |
| import torch |
| import transformers |
| |
| # We use a couple of training queries from Amazon ESCI and Search ESCI as an example. |
| queries = [ |
| "iphone 11 pro max case" |
| "cotton summer dress care instructions" |
| ] |
| products_and_reviews = [ |
| "OtterBox Symmetry Series Case for iPhone 11 Pro Max - Black [...]", |
| "Camera Lens Protector for iPhone 11 Pro/Pro Max, Tempered Glass 9H Hardness Anti-Scratch Camera Screen Protective [...]" |
| "Cute, cool and comfy summer dress [...] Hand wash and line drys easily, material is crinkly so no ironing needed. [...]", |
| "Excellent machine JET J-2530 15-Inch 3/4-Horsepower Bench Drill Press. Two common Amazon reviewer complaints about higher-end drill presses [...]" |
| ] |
| |
| # Load the tokenizer and model. |
| tokenizer = transformers.AutoTokenizer.from_pretrained("J-MADRAL/J-BiBERT") |
| model = modeling.BiEncoderModel.from_pretrained("J-MADRAL/J-BiBERT") |
| |
| # Tokenize the input data. |
| q_input = tokenizer(queries, |
| add_special_tokens=True, |
| truncation=True, |
| padding=True, |
| max_length=128, |
| return_tensors="pt") |
| pr_input = tokenizer(products_and_reviews, |
| add_special_tokens=True, |
| truncation=True, |
| padding=True, |
| max_length=128, |
| return_tensors="pt") |
| |
| # Compute embeddings: take the "pooled_output". |
| q_emb = model(**q_input).pooled_output |
| pr_emb = model(**pr_input).pooled_output |
| |
| # Compute similarity scores, using dot product similarity. |
| scores = torch.matmul(q_emb, pr_emb.transpose(0, 1)) |
| ``` |
|
|
|
|
| ## Training Hyperparameters ## |
|
|
| Training Stage | Num. Epochs | Learning Rate | AP Scaling Factor | Max Num Tokens | Batch Size | Num Negatives |
| |---|---|---|---|---|---|--- |
| Pre-training | 20 | 1e-4 | 0.10 | 128 | 64 | --- |
| Fine-tuning | 20 | 5e-6 | 0.05 | 128 | 64 | 7 |
|
|
| The data used for fine-tuning is available at [https://huggingface.co/datasets/J-MADRAL/TrainingData](https://huggingface.co/datasets/J-MADRAL/TrainingData). |
|
|
|
|
| ## Evaluation Results ## |
|
|
| #### [Amazon ESCI](https://huggingface.co/datasets/J-MADRAL/AmazonESCI) #### |
|
|
| Model | R@100 | R@500 | MRR | nDCG@10 | nDCG@50 |
| |---|---|---|---|---|--- |
| BM25 | 0.4949 | 0.6603 | 0.4056 | 0.2599 | 0.3160 |
| [DRAGON](https://huggingface.co/facebook/dragon-plus-context-encoder) | 0.5490 | 0.7155 | 0.4541 | 0.2929 | 0.3540 |
| [P-BiBERT](https://huggingface.co/J-MADRAL/P-BiBERT) | 0.6018 | 0.7640 | 0.4939 | 0.3276 | 0.3967 |
| [P-MADRAL](https://huggingface.co/J-MADRAL/P-MADRAL) | **0.6235** | **0.7806** | **0.5060** | **0.3382** | **0.4104** |
| **J-BiBERT** | 0.5889 | 0.7556 | 0.4808 | 0.3195 | 0.3858 |
| [J-MADRAL](https://huggingface.co/J-MADRAL/J-MADRAL) | *0.6083* | *0.7729* | *0.4988* | *0.3330* | *0.4028* |
|
|
| #### [TREC Product Search 2023](https://huggingface.co/datasets/J-MADRAL/TREC_Product_Search_2023) #### |
|
|
| Model | R@100 | R@500 | MRR | nDCG@10 | nDCG@50 |
| |---|---|---|---|---|--- |
| BM25 | 0.7252 | 0.8650 | 0.7746 | 0.6231 | 0.5988 |
| [DRAGON](https://huggingface.co/facebook/dragon-plus-context-encoder) | 0.7373 | 0.8692 | 0.8135 | 0.6526 | 0.6293 |
| [P-BiBERT](https://huggingface.co/J-MADRAL/P-BiBERT) | 0.7432 | 0.8757 | 0.8256 | 0.6612 | 0.6287 |
| [P-MADRAL](https://huggingface.co/J-MADRAL/P-MADRAL) | **0.7547** | *0.8840* | **0.8337** | **0.6719** | **0.6480** |
| **J-BiBERT** | 0.7432 | 0.8684 | 0.8114 | 0.6487 | 0.6213 |
| [J-MADRAL](https://huggingface.co/J-MADRAL/J-MADRAL) | *0.7527* | **0.8888** | *0.8333* | *0.6702* | *0.6411* |
|
|
| #### [Search ESCI](https://huggingface.co/datasets/J-MADRAL/SearchESCI) #### |
|
|
| Model | R@100 | R@500 | MRR | nDCG@10 | nDCG@50 |
| |---|---|---|---|---|--- |
| BM25 | 0.5875 | 0.7288 | 0.2539 | 0.2766 | 0.3101 |
| [DRAGON](https://huggingface.co/facebook/dragon-plus-context-encoder) | 0.5451 | 0.6751 | 0.2347 | 0.2567 | 0.2873 |
| [R-BiBERT](https://huggingface.co/J-MADRAL/R-BiBERT) | 0.5972 | 0.7350 | 0.2602 | 0.2855 | 0.3176 |
| [R-MADRAL](https://huggingface.co/J-MADRAL/R-MADRAL) | *0.6405* | *0.7626* | *0.2944* | *0.3215* | *0.3541* |
| **J-BiBERT** | 0.6300 | 0.7593 | 0.2879 | 0.3140 | 0.3474 |
| [J-MADRAL](https://huggingface.co/J-MADRAL/J-MADRAL) | **0.6488** | **0.7729** | **0.3007** | **0.3281** | **0.3611** |
