Upload folder using huggingface_hub

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+    "word_embedding_dimension": 4096,
+    "pooling_mode_cls_token": false,
+    "pooling_mode_mean_tokens": false,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false,
+    "pooling_mode_weightedmean_tokens": false,
+    "pooling_mode_lasttoken": true,
+    "include_prompt": true
+}

2_SparseEmbedding/config.json

@@ -0,0 +1,8 @@
+{
+    "n_in": 4096,
+    "init_mean": 0.5,
+    "init_std": 0.01,
+    "temperature": 1.0,
+    "stretch": 0.1,
+    "eps": 1e-06
+}

2_SparseEmbedding/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55820e46cd6fa739d6844a20567ecf18f7c969c484db91da09420188d9e6b862
+size 18044

README.md

@@ -0,0 +1,882 @@
+---
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- dense
+- generated_from_trainer
+- dataset_size:14166
+- loss:SparseLoss
+- loss:MultipleNegativesRankingLoss
+widget:
+- source_sentence: 'Instruct: Given a web search query, retrieve relevant passages
+    that answer the query
+
+    Query: Does borrowing from my 401(k) make sense in my specific circumstance?'
+  sentences:
+  - 'I''m not sure why you think that it matters that the distribution goes to an
+    S-Corp vs an individual tax payer. You seem to think it has any relevance to your
+    question, but it doesn''t. It only confuses your readers. The situation is like
+    this: LLC X is deriving income in State #2. It has two members (I and S) residents
+    of State #1. Members I and S pay all their taxes to State #1, and don''t pay taxes
+    to State #2. State #2 audited member I and that member now needs to pay back taxes
+    and penalties to State #2 on income derived from that State. Your question: Does
+    that mean that member S should be worried, since that member was essentially doing
+    the exact same thing as member I? My answer: Yes.'
+  - Interest rates are market driven. They tend to be based on the prime rate set
+    by the federal reserve bank because of the tremendous lending capacity of that
+    institution and that other loan originators will often fund their own lending
+    (at least in part) with fed loans. However, there is no mandatory link between
+    the federal reserve rate and the market rate. No law stipulates that rates cannot
+    rise or fall. They will rise and fall as lenders see necessary to use their capital.
+    Though a lender asking 10% interest might make no loans when others are willing
+    to lend for 9%. The only protection you have is that we are (mostly) economically
+    free. As a borrower, you are protected by the fact that there are many lenders.
+    Likewise, as a lender, because there are many borrowers. Stability is simply by
+    virtue of the fact that one market participant with inordinate pricing will find
+    fewer counterparties to transact.
+  - '"You''re getting great wisdom and options. Establishing your actionable path
+    will require the details that only you know, such as how much is actually in each
+    paycheck (and how much tax is withheld), how much do you spend each month (and
+    yearly expenses too), how much spending can you actually cut or replace, how comfortable
+    are you with considering (or not considering) unexpected/emergency spending. You
+    mentioned you were cash-poor, but only you know what your current account balances
+    are, which will affect your actions and priorities. Btw, interestingly, your ""increase
+    401k contributions by 2% each year"" will need to end before hitting the $18K
+    contribution limit. I took some time and added the details you posted into a cash-flow
+    program to see your scenario over the next few years. There isn''t a ""401k loan""
+    activity in this program yet, so I build the scenario from other simple activities.
+    You seem financially minded enough to continue modeling on your own. I''m posting
+    the more difficult one for you (borrow from 401k), but you''ll have to input your
+    actual balances, paycheck and spending. My spending assumptions must be low, and
+    I entered $70K as ""take-home,"" so the model looks like you''ve got lots of cash.
+    If you choose to play with it, then consider modeling some other scenarios from
+    the advice in the other posts. Here''s the ""Borrow $6500 from 401k"" scenario
+    model at Whatll.Be: https://whatll.be/d1x1ndp26i/2 To me, it''s all about trying
+    the scenarios and see which one seems to work with all of the details. The trick
+    is knowing what scenarios to try, and how to model them. Full disclosure: I needed
+    to do similar planning, so I wrote Whatll.Be and I now share it with other people.
+    It''s in beta, so I''m testing it with scenarios like yours. (Notice most of the
+    extra activity occurs on 2018-Jan-01)"'
+- source_sentence: 'Instruct: Given a web search query, retrieve relevant passages
+    that answer the query
+
+    Query: Finding a good small business CPA?'
+  sentences:
+  - I have had better experiences with accountants in smaller towns.  It seems they
+    are used to working with small businesses and their reputation is very important
+    to them.
+  - '"It''s a scam. The cashier''s check will be forged. Craigslist has a warning
+    about it here (item #3). What kind of payment do you think is not fakable? Or
+    at least not   likely to be used in scams? When on craigslist - deal only locally
+    and in person. You can ask to see the person''s ID if you''re being paid by check
+    When being paid by check, how can seeing his/her ID help? In case the   check
+    isn''t cashable, I can find that person by keeping record of   his/her ID?  If
+    you''re paid by check, the payers details should be printed on the check. By checking
+    the ID you can verify that the details match (name/address), so you can find the
+    payer later. Of course the ID can be faked too, but there''s so much you can do
+    to protect yourself. You''ll get better protection (including verified escrow
+    service) by selling on eBay. Is being paid by cash the safest way currently,   although
+    cash can be faked too, but it is the least common thing that   is faked currently?
+    Do you recommend to first deposit the cash into a   bank (so that let the bank
+    verify if the cash is faked), before   delivering the good?  For Craigslist, use
+    cash and meet locally. That rules out most scams as a seller. What payment methods
+    do you think are relatively safe currently?   Then getting checks must be the
+    least favorite way of being paid. Do   you think cash is better than money order
+    or cashier order?  You should only accept cash. If it is a large transaction,
+    you can meet them at your bank, have them get cash, and you receive the cash from
+    the bank. Back to the quoted scam, how will they later manipulate me? Are they
+    interested in   my stuffs on moving sale, or in my money? They will probably ""accidentally""
+    overpay you and ask for a refund of some portion of the overpayment. In that case
+    you will be out the entire amount that you send back to them and possibly some
+    fees from your bank for cashing a bad check."'
+  - '"Putting them on line 10 is best suited for your situation. According to Quickbooks:  Commissions
+    and Fees (Line 10) Commissions/fees paid to nonemployees to generate revenue (e.g.
+    agent   fees). It seems like this website you are using falls under the term ""nonemployees""."'
+- source_sentence: 'Instruct: Given a web search query, retrieve relevant passages
+    that answer the query
+
+    Query: Are Investment Research websites worth their premiums?'
+  sentences:
+  - Anyone who claims they can consistently beat the market and asks you to pay them
+    to tell you how is a liar. This cannot be done, as the market adjusts itself.
+    There's nothing they could possibly learn that analysts and institutional investors
+    don't already know. They earn their money through the subscription fees, not through
+    capital gains on their beat-the-market suggestions, that means that they don't
+    have to rely on themselves to earn money, they only need you to rely on them.
+    They have to provide proof because they cannot lie in advertisements, but if you
+    read carefully, there are many small letters and disclaimers that basically remove
+    any liability from them by saying that they don't take responsibility for anything
+    and don't guarantee anything.
+  - '"I don''t see EWQ6 in any of your links, so I can''t say for certain, but when
+    you buy an option contract on a future, the option will be for a specific future
+    (and strike).  So the page you''re looking at may be for options on E-mini S&P
+    500 futures in general, and when you actually purchase one through your broker,
+    you pick a specific expiry (which will be based on the ""prompt"" future, meaning
+    the next future that expires after the option) and strike. UPDATE: Based on this
+    page mirror, the option EWQ7 is an option on the ESU7 (SEP 2017) future.  The
+    next 3 monthly options use ESZ7 as the underlier, which confirms that they use
+    the next prompt future as the underlier."'
+  - There are a few factors at play here. Depending on the bank that has offered you
+    the card there are different types of overdraft protection that may have been
+    set up. Typically, if they attempt to run the card with no money, if one of these
+    is in play, you will be spared any overdraft fees by the transaction charging
+    to a designated overdraft account, usually savings, or by the transaction failing
+    due to insufficient funds.  If you know the transaction went through, and you
+    know there were not enough funds in the account to cover the transactions, then
+    you have a few options. If you have overdraft protection that auto charges insufficient
+    funds charges to a separate account, then you have nothing to worry about. If
+    you do not, most banks offer a grace period where you have until the end of the
+    day to zero out your account, that is to say pay the overdraft amount and bring
+    your balance to at least $0.  If this is a charge that occurred in the past, and
+    you have already been charged an overdraft fee, there may still be hope. I cannot
+    speak for all banks, but I know that Chase Bank offers a once per year overdraft
+    forgiveness, where they will get rid of the charges if you agree to bring the
+    account out of the negative. There is a chance other banks will do the same if
+    you call their customer service.
+- source_sentence: 'Instruct: Given a web search query, retrieve relevant passages
+    that answer the query
+
+    Query: Ballpark salary equivalent today of “healthcare benefits” in the US?'
+  sentences:
+  - While in the interview stage you need one good outfit. Take care of them and they
+    will see you through this stage of the process. Shoes, ties, shirt, and a suit
+    can all be purchased on sale. The fact that you have months before graduation
+    give you time to purchase them when there is a sale.  Off-the-rack is good enough
+    for a suit for this stage of your life. There is no need to go custom made when
+    you are just starting out. In fact you may find you never need more than one or
+    two suits, and they never need to be custom made.
+  - Fiduciary They are obligated by the rules of the exchanges they are listed with.
+    Furthermore, there is a strong chance that people running the company also have
+    stock, so it personally benefits them to create higher prices. Finally, maybe
+    they don't care about the prices directly, but by being a good company with a
+    good product or service, they are desirable and that is expressed as a higher
+    stock price.  Not every action is because it will raise the stock price, but because
+    it is good for business which happens to make the stock more valuable.
+  - 'There is some magic involved in that calculation, because what health insurance
+    is worth to you is not necessarily the same it is worth for the employer. Two
+    examples that illustrate the extreme ends of the spectrum: let''s say you or a
+    family member have a chronic or a serious illness, especially if it is a preexisting
+    condition - for instance, cancer. In that case, health insurance can be worth
+    literally millions of dollars to you. Even if you are a diabetic, the value of
+    health insurance can be substantial. Sometimes, it could even make financial sense
+    in that case to accept a very low-paying job. On the other extreme of the scale,
+    if you are very young and healthy, many people decide to forego insurance. In
+    that case, the value of health insurance can be as little as the penalty (usually,
+    2% of your taxable income, I believe).'
+- source_sentence: 'Instruct: Given a web search query, retrieve relevant passages
+    that answer the query
+
+    Query: I am under 18 years old, in the US, my parents have terrible credit, how
+    can I take out a loan?'
+  sentences:
+  - What about web-hosting fees? Cost of Internet service? Cost of computer equipment
+    to do the work? Amortized cost of development? Time for support calls/email? Phone
+    service used for sales? Advertising/marketing expenses? Look hard--I bet there
+    are some costs.
+  - 'In the equity markets, the P/E is usually somewhere around 15.  The P/E can be
+    viewed as the inverse of the rate of a perpetuity. Since the average is 15, and
+    the E/P of that would be 6.7%, r should be 6.7% on average. If your business is
+    growing, the growth rate can be incorporated like so: As you can see, a high g
+    would make the price negative, in essence the seller should actually pay someone
+    to take the business, but in reality, r is determined from the p and an estimated
+    g. For a business of any growth rate, it''s best to compare the multiple to the
+    market, so for the average business in the market with your business''s growth
+    rate and industry, that P/E would be best applied to your company''s income.'
+  - Depending on the state this might not be possible. Loans are considered contracts,
+    and various states regulate how minors may enter into them. For example, in the
+    state of Oregon, a minor may NOT enter into a contract without their parent being
+    on the contract as well. So you are forced to wait until you turn 18. At that
+    time you won't have a credit history, and to lenders that often is worse than
+    having bad credit. I can't help with the car (other than to recommend you buy
+    a junker for $500-$1,000 and just live with it for now), but you could certainly
+    get a secured credit card or line of credit from your local bank. The way they
+    are arranged is, you make a deposit of an amount of your choosing (generally at
+    least $200 for credit cards, and $1,000 for lines of credit), and receive a revolving
+    line with a limit of that same amount. As you use and pay on this loan, it will
+    be reported in your credit history. If you start that now, by the time you turn
+    18 you will have much better options for purchasing vehicles.
+pipeline_tag: sentence-similarity
+library_name: sentence-transformers
+---
+
+# SentenceTransformer
+
+This is a [sentence-transformers](https://www.SBERT.net) model trained on the fiqa dataset. It maps sentences & paragraphs to a 4096-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+
+## Model Details
+
+### Model Description
+- **Model Type:** Sentence Transformer
+<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
+- **Maximum Sequence Length:** 32768 tokens
+- **Output Dimensionality:** 4096 dimensions
+- **Similarity Function:** Cosine Similarity
+- **Training Dataset:**
+    - fiqa
+<!-- - **Language:** Unknown -->
+<!-- - **License:** Unknown -->
+
+### Model Sources
+
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
+
+### Full Model Architecture
+
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'MistralModel'})
+  (1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
+  (2): SparseEmbedding(
+    (sparsifyer): ZeroNeuron(in_features=4096, out_features=4096)
+  )
+)
+```
+
+## Usage
+
+### Direct Usage (Sentence Transformers)
+
+First install the Sentence Transformers library:
+
+```bash
+pip install -U sentence-transformers
+```
+
+Then you can load this model and run inference.
+```python
+from sentence_transformers import SentenceTransformer
+
+# Download from the 🤗 Hub
+model = SentenceTransformer("sentence_transformers_model_id")
+# Run inference
+sentences = [
+    'Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: I am under 18 years old, in the US, my parents have terrible credit, how can I take out a loan?',
+    "Depending on the state this might not be possible. Loans are considered contracts, and various states regulate how minors may enter into them. For example, in the state of Oregon, a minor may NOT enter into a contract without their parent being on the contract as well. So you are forced to wait until you turn 18. At that time you won't have a credit history, and to lenders that often is worse than having bad credit. I can't help with the car (other than to recommend you buy a junker for $500-$1,000 and just live with it for now), but you could certainly get a secured credit card or line of credit from your local bank. The way they are arranged is, you make a deposit of an amount of your choosing (generally at least $200 for credit cards, and $1,000 for lines of credit), and receive a revolving line with a limit of that same amount. As you use and pay on this loan, it will be reported in your credit history. If you start that now, by the time you turn 18 you will have much better options for purchasing vehicles.",
+    "In the equity markets, the P/E is usually somewhere around 15.  The P/E can be viewed as the inverse of the rate of a perpetuity. Since the average is 15, and the E/P of that would be 6.7%, r should be 6.7% on average. If your business is growing, the growth rate can be incorporated like so: As you can see, a high g would make the price negative, in essence the seller should actually pay someone to take the business, but in reality, r is determined from the p and an estimated g. For a business of any growth rate, it's best to compare the multiple to the market, so for the average business in the market with your business's growth rate and industry, that P/E would be best applied to your company's income.",
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 4096]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities)
+# tensor([[1.0000, 0.6277, 0.2807],
+#         [0.6277, 1.0000, 0.2775],
+#         [0.2807, 0.2775, 1.0000]])
+```
+
+<!--
+### Direct Usage (Transformers)
+
+<details><summary>Click to see the direct usage in Transformers</summary>
+
+</details>
+-->
+
+<!--
+### Downstream Usage (Sentence Transformers)
+
+You can finetune this model on your own dataset.
+
+<details><summary>Click to expand</summary>
+
+</details>
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Dataset
+
+#### fiqa
+
+* Dataset: fiqa
+* Size: 14,166 training samples
+* Columns: <code>anchor</code> and <code>positive</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | anchor                                                                             | positive                                                                             |
+  |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
+  | type    | string                                                                             | string                                                                               |
+  | details | <ul><li>min: 24 tokens</li><li>mean: 36.38 tokens</li><li>max: 62 tokens</li></ul> | <ul><li>min: 2 tokens</li><li>mean: 223.61 tokens</li><li>max: 1683 tokens</li></ul> |
+* Samples:
+  | anchor                                                                                                                                                                                                    | positive                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
+  |:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | <code>Instruct: Given a web search query, retrieve relevant passages that answer the query<br>Query: What is considered a business expense on a business trip?</code>                                     | <code>The IRS Guidance pertaining to the subject.  In general the best I can say is your business expense may be deductible.  But it depends on the circumstances and what it is you want to deduct. Travel Taxpayers who travel away from home on business may deduct related   expenses, including the cost of reaching their destination, the cost   of lodging and meals and other ordinary and necessary expenses.   Taxpayers are considered “traveling away from home” if their duties   require them to be away from home substantially longer than an   ordinary day’s work and they need to sleep or rest to meet the demands   of their work. The actual cost of meals and incidental expenses may be   deducted or the taxpayer may use a standard meal allowance and reduced   record keeping requirements. Regardless of the method used, meal   deductions are generally limited to 50 percent as stated earlier.    Only actual costs for lodging may be claimed as an expense and   receipts must be kept for documentation. ...</code> |
+  | <code>Instruct: Given a web search query, retrieve relevant passages that answer the query<br>Query: Business Expense - Car Insurance Deductible For Accident That Occurred During a Business Trip</code> | <code>As a general rule, you must choose between a mileage deduction or an actual expenses deduction.  The idea is that the mileage deduction is supposed to cover all costs of using the car.  Exceptions include parking fees and tolls, which can be deducted separately under either method.  You explicitly cannot deduct insurance costs if you claim a mileage deduction.   Separately, you probably won't be able to deduct the deductible for your car as a casualty loss.  You first subtract $100 from the deductible and then divide it by your Adjusted Gross Income (AGI) from your tax return.  If your deductible is over 10% of your AGI, you can deduct it.   Note that even with a $1500 deductible, you won't be able to deduct anything if you made more than $14,000 for the year.  For most people, the insurance deductible just isn't large enough relative to income to be tax deductible.   Source</code>                                                                                                                     |
+  | <code>Instruct: Given a web search query, retrieve relevant passages that answer the query<br>Query: Starting a new online business</code>                                                                | <code>Most US states have rules that go something like this: You will almost certainly have to pay some registration fees, as noted above.  Depending on how you organize, you may or may not need to file a separate tax return for the business. (If you're sole proprietor for tax purposes, then you file on Schedule C on your personal Form 1040.)  Whether or not you pay taxes depends on whether you have net income.  It's possible that some losses might also be deductible. (Note that you may have to file a return even if you don't have net income - Filing and needing to pay are not the same since your return may indicate no tax due.)  In addition, at the state level, you may have to pay additional fees or taxes beyond income tax depending on what you sell and how you sell it. (Sales tax, for example, might come into play as might franchise taxes.)  You'll need to check your own state law for that. As always, it could be wise to get professional tax and accounting advice that's tailored to your si...</code> |
+* Loss: <code>zero-neuron.src.embedding.sparse_loss.SparseLoss</code>
+
+### Training Hyperparameters
+#### Non-Default Hyperparameters
+
+- `per_device_eval_batch_size`: 16
+- `learning_rate`: 0.0005
+- `num_train_epochs`: 2
+- `lr_scheduler_type`: cosine
+- `warmup_ratio`: 0.01
+- `save_safetensors`: False
+- `bf16`: True
+- `remove_unused_columns`: False
+- `batch_sampler`: no_duplicates
+
+#### All Hyperparameters
+<details><summary>Click to expand</summary>
+
+- `overwrite_output_dir`: False
+- `do_predict`: False
+- `eval_strategy`: no
+- `prediction_loss_only`: True
+- `per_device_train_batch_size`: 8
+- `per_device_eval_batch_size`: 16
+- `per_gpu_train_batch_size`: None
+- `per_gpu_eval_batch_size`: None
+- `gradient_accumulation_steps`: 1
+- `eval_accumulation_steps`: None
+- `torch_empty_cache_steps`: None
+- `learning_rate`: 0.0005
+- `weight_decay`: 0.0
+- `adam_beta1`: 0.9
+- `adam_beta2`: 0.999
+- `adam_epsilon`: 1e-08
+- `max_grad_norm`: 1.0
+- `num_train_epochs`: 2
+- `max_steps`: -1
+- `lr_scheduler_type`: cosine
+- `lr_scheduler_kwargs`: {}
+- `warmup_ratio`: 0.01
+- `warmup_steps`: 0
+- `log_level`: passive
+- `log_level_replica`: warning
+- `log_on_each_node`: True
+- `logging_nan_inf_filter`: True
+- `save_safetensors`: False
+- `save_on_each_node`: False
+- `save_only_model`: False
+- `restore_callback_states_from_checkpoint`: False
+- `no_cuda`: False
+- `use_cpu`: False
+- `use_mps_device`: False
+- `seed`: 42
+- `data_seed`: None
+- `jit_mode_eval`: False
+- `use_ipex`: False
+- `bf16`: True
+- `fp16`: False
+- `fp16_opt_level`: O1
+- `half_precision_backend`: auto
+- `bf16_full_eval`: False
+- `fp16_full_eval`: False
+- `tf32`: None
+- `local_rank`: 0
+- `ddp_backend`: None
+- `tpu_num_cores`: None
+- `tpu_metrics_debug`: False
+- `debug`: []
+- `dataloader_drop_last`: False
+- `dataloader_num_workers`: 0
+- `dataloader_prefetch_factor`: None
+- `past_index`: -1
+- `disable_tqdm`: False
+- `remove_unused_columns`: False
+- `label_names`: None
+- `load_best_model_at_end`: False
+- `ignore_data_skip`: False
+- `fsdp`: []
+- `fsdp_min_num_params`: 0
+- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
+- `fsdp_transformer_layer_cls_to_wrap`: None
+- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
+- `deepspeed`: None
+- `label_smoothing_factor`: 0.0
+- `optim`: adamw_torch
+- `optim_args`: None
+- `adafactor`: False
+- `group_by_length`: False
+- `length_column_name`: length
+- `ddp_find_unused_parameters`: None
+- `ddp_bucket_cap_mb`: None
+- `ddp_broadcast_buffers`: False
+- `dataloader_pin_memory`: True
+- `dataloader_persistent_workers`: False
+- `skip_memory_metrics`: True
+- `use_legacy_prediction_loop`: False
+- `push_to_hub`: False
+- `resume_from_checkpoint`: None
+- `hub_model_id`: None
+- `hub_strategy`: every_save
+- `hub_private_repo`: None
+- `hub_always_push`: False
+- `hub_revision`: None
+- `gradient_checkpointing`: False
+- `gradient_checkpointing_kwargs`: None
+- `include_inputs_for_metrics`: False
+- `include_for_metrics`: []
+- `eval_do_concat_batches`: True
+- `fp16_backend`: auto
+- `push_to_hub_model_id`: None
+- `push_to_hub_organization`: None
+- `mp_parameters`: 
+- `auto_find_batch_size`: False
+- `full_determinism`: False
+- `torchdynamo`: None
+- `ray_scope`: last
+- `ddp_timeout`: 1800
+- `torch_compile`: False
+- `torch_compile_backend`: None
+- `torch_compile_mode`: None
+- `include_tokens_per_second`: False
+- `include_num_input_tokens_seen`: False
+- `neftune_noise_alpha`: None
+- `optim_target_modules`: None
+- `batch_eval_metrics`: False
+- `eval_on_start`: False
+- `use_liger_kernel`: False
+- `liger_kernel_config`: None
+- `eval_use_gather_object`: False
+- `average_tokens_across_devices`: False
+- `prompts`: None
+- `batch_sampler`: no_duplicates
+- `multi_dataset_batch_sampler`: proportional
+- `router_mapping`: {}
+- `learning_rate_mapping`: {}
+
+</details>
+
+### Training Logs
+<details><summary>Click to expand</summary>
+
+| Epoch  | Step | Training Loss |
+|:------:|:----:|:-------------:|
+| 0.0056 | 10   | 1.1694        |
+| 0.0113 | 20   | 1.2809        |
+| 0.0169 | 30   | 1.2538        |
+| 0.0226 | 40   | 1.2237        |
+| 0.0282 | 50   | 1.1809        |
+| 0.0339 | 60   | 1.1602        |
+| 0.0395 | 70   | 1.2008        |
+| 0.0452 | 80   | 1.1064        |
+| 0.0508 | 90   | 1.0857        |
+| 0.0565 | 100  | 1.0553        |
+| 0.0621 | 110  | 1.0513        |
+| 0.0678 | 120  | 0.9236        |
+| 0.0734 | 130  | 0.8998        |
+| 0.0791 | 140  | 0.8509        |
+| 0.0847 | 150  | 0.7769        |
+| 0.0903 | 160  | 0.7268        |
+| 0.0960 | 170  | 0.7287        |
+| 0.1016 | 180  | 0.6437        |
+| 0.1073 | 190  | 0.6653        |
+| 0.1129 | 200  | 0.5888        |
+| 0.1186 | 210  | 0.6097        |
+| 0.1242 | 220  | 0.6939        |
+| 0.1299 | 230  | 0.5969        |
+| 0.1355 | 240  | 0.5333        |
+| 0.1412 | 250  | 0.5143        |
+| 0.1468 | 260  | 0.6152        |
+| 0.1525 | 270  | 0.4779        |
+| 0.1581 | 280  | 0.5182        |
+| 0.1637 | 290  | 0.5724        |
+| 0.1694 | 300  | 0.5073        |
+| 0.1750 | 310  | 0.4924        |
+| 0.1807 | 320  | 0.5219        |
+| 0.1863 | 330  | 0.5621        |
+| 0.1920 | 340  | 0.4535        |
+| 0.1976 | 350  | 0.4818        |
+| 0.2033 | 360  | 0.4773        |
+| 0.2089 | 370  | 0.4948        |
+| 0.2146 | 380  | 0.4277        |
+| 0.2202 | 390  | 0.5043        |
+| 0.2259 | 400  | 0.5746        |
+| 0.2315 | 410  | 0.4762        |
+| 0.2372 | 420  | 0.4432        |
+| 0.2428 | 430  | 0.4771        |
+| 0.2484 | 440  | 0.5298        |
+| 0.2541 | 450  | 0.4352        |
+| 0.2597 | 460  | 0.5714        |
+| 0.2654 | 470  | 0.508         |
+| 0.2710 | 480  | 0.5215        |
+| 0.2767 | 490  | 0.5096        |
+| 0.2823 | 500  | 0.4598        |
+| 0.2880 | 510  | 0.5843        |
+| 0.2936 | 520  | 0.5581        |
+| 0.2993 | 530  | 0.4686        |
+| 0.3049 | 540  | 0.4956        |
+| 0.3106 | 550  | 0.4209        |
+| 0.3162 | 560  | 0.4181        |
+| 0.3219 | 570  | 0.4847        |
+| 0.3275 | 580  | 0.5193        |
+| 0.3331 | 590  | 0.4235        |
+| 0.3388 | 600  | 0.4626        |
+| 0.3444 | 610  | 0.4309        |
+| 0.3501 | 620  | 0.451         |
+| 0.3557 | 630  | 0.4742        |
+| 0.3614 | 640  | 0.4892        |
+| 0.3670 | 650  | 0.4478        |
+| 0.3727 | 660  | 0.4461        |
+| 0.3783 | 670  | 0.5197        |
+| 0.3840 | 680  | 0.4692        |
+| 0.3896 | 690  | 0.4272        |
+| 0.3953 | 700  | 0.4196        |
+| 0.4009 | 710  | 0.4737        |
+| 0.4065 | 720  | 0.4015        |
+| 0.4122 | 730  | 0.4786        |
+| 0.4178 | 740  | 0.3968        |
+| 0.4235 | 750  | 0.4499        |
+| 0.4291 | 760  | 0.478         |
+| 0.4348 | 770  | 0.4003        |
+| 0.4404 | 780  | 0.4679        |
+| 0.4461 | 790  | 0.4129        |
+| 0.4517 | 800  | 0.452         |
+| 0.4574 | 810  | 0.4238        |
+| 0.4630 | 820  | 0.4761        |
+| 0.4687 | 830  | 0.4324        |
+| 0.4743 | 840  | 0.4535        |
+| 0.4800 | 850  | 0.4914        |
+| 0.4856 | 860  | 0.5368        |
+| 0.4912 | 870  | 0.4106        |
+| 0.4969 | 880  | 0.419         |
+| 0.5025 | 890  | 0.3884        |
+| 0.5082 | 900  | 0.4833        |
+| 0.5138 | 910  | 0.4295        |
+| 0.5195 | 920  | 0.3673        |
+| 0.5251 | 930  | 0.4245        |
+| 0.5308 | 940  | 0.4636        |
+| 0.5364 | 950  | 0.3897        |
+| 0.5421 | 960  | 0.4342        |
+| 0.5477 | 970  | 0.442         |
+| 0.5534 | 980  | 0.4443        |
+| 0.5590 | 990  | 0.3737        |
+| 0.5647 | 1000 | 0.441         |
+| 0.5703 | 1010 | 0.4247        |
+| 0.5759 | 1020 | 0.4583        |
+| 0.5816 | 1030 | 0.4077        |
+| 0.5872 | 1040 | 0.5236        |
+| 0.5929 | 1050 | 0.4307        |
+| 0.5985 | 1060 | 0.5054        |
+| 0.6042 | 1070 | 0.4787        |
+| 0.6098 | 1080 | 0.4521        |
+| 0.6155 | 1090 | 0.4011        |
+| 0.6211 | 1100 | 0.3864        |
+| 0.6268 | 1110 | 0.4191        |
+| 0.6324 | 1120 | 0.436         |
+| 0.6381 | 1130 | 0.4469        |
+| 0.6437 | 1140 | 0.4416        |
+| 0.6494 | 1150 | 0.4475        |
+| 0.6550 | 1160 | 0.3857        |
+| 0.6606 | 1170 | 0.3571        |
+| 0.6663 | 1180 | 0.441         |
+| 0.6719 | 1190 | 0.4144        |
+| 0.6776 | 1200 | 0.4108        |
+| 0.6832 | 1210 | 0.4051        |
+| 0.6889 | 1220 | 0.489         |
+| 0.6945 | 1230 | 0.3881        |
+| 0.7002 | 1240 | 0.4971        |
+| 0.7058 | 1250 | 0.415         |
+| 0.7115 | 1260 | 0.4048        |
+| 0.7171 | 1270 | 0.3805        |
+| 0.7228 | 1280 | 0.3869        |
+| 0.7284 | 1290 | 0.3804        |
+| 0.7340 | 1300 | 0.4141        |
+| 0.7397 | 1310 | 0.4223        |
+| 0.7453 | 1320 | 0.3836        |
+| 0.7510 | 1330 | 0.4012        |
+| 0.7566 | 1340 | 0.4725        |
+| 0.7623 | 1350 | 0.3946        |
+| 0.7679 | 1360 | 0.4424        |
+| 0.7736 | 1370 | 0.4256        |
+| 0.7792 | 1380 | 0.4381        |
+| 0.7849 | 1390 | 0.3634        |
+| 0.7905 | 1400 | 0.3568        |
+| 0.7962 | 1410 | 0.4158        |
+| 0.8018 | 1420 | 0.3982        |
+| 0.8075 | 1430 | 0.4734        |
+| 0.8131 | 1440 | 0.3787        |
+| 0.8187 | 1450 | 0.4492        |
+| 0.8244 | 1460 | 0.4504        |
+| 0.8300 | 1470 | 0.4125        |
+| 0.8357 | 1480 | 0.4059        |
+| 0.8413 | 1490 | 0.419         |
+| 0.8470 | 1500 | 0.4269        |
+| 0.8526 | 1510 | 0.5586        |
+| 0.8583 | 1520 | 0.4664        |
+| 0.8639 | 1530 | 0.5185        |
+| 0.8696 | 1540 | 0.422         |
+| 0.8752 | 1550 | 0.5141        |
+| 0.8809 | 1560 | 0.4576        |
+| 0.8865 | 1570 | 0.372         |
+| 0.8922 | 1580 | 0.4194        |
+| 0.8978 | 1590 | 0.4074        |
+| 0.9034 | 1600 | 0.3894        |
+| 0.9091 | 1610 | 0.4172        |
+| 0.9147 | 1620 | 0.4274        |
+| 0.9204 | 1630 | 0.4013        |
+| 0.9260 | 1640 | 0.4072        |
+| 0.9317 | 1650 | 0.3616        |
+| 0.9373 | 1660 | 0.3485        |
+| 0.9430 | 1670 | 0.4478        |
+| 0.9486 | 1680 | 0.4543        |
+| 0.9543 | 1690 | 0.4229        |
+| 0.9599 | 1700 | 0.4186        |
+| 0.9656 | 1710 | 0.378         |
+| 0.9712 | 1720 | 0.3753        |
+| 0.9768 | 1730 | 0.4575        |
+| 0.9825 | 1740 | 0.4291        |
+| 0.9881 | 1750 | 0.4288        |
+| 0.9938 | 1760 | 0.3678        |
+| 0.9994 | 1770 | 0.5298        |
+| 1.0051 | 1780 | 0.3926        |
+| 1.0107 | 1790 | 0.3799        |
+| 1.0164 | 1800 | 0.4288        |
+| 1.0220 | 1810 | 0.4323        |
+| 1.0277 | 1820 | 0.4371        |
+| 1.0333 | 1830 | 0.4652        |
+| 1.0390 | 1840 | 0.3565        |
+| 1.0446 | 1850 | 0.4567        |
+| 1.0503 | 1860 | 0.3947        |
+| 1.0559 | 1870 | 0.3868        |
+| 1.0615 | 1880 | 0.4143        |
+| 1.0672 | 1890 | 0.482         |
+| 1.0728 | 1900 | 0.3763        |
+| 1.0785 | 1910 | 0.3795        |
+| 1.0841 | 1920 | 0.4413        |
+| 1.0898 | 1930 | 0.4761        |
+| 1.0954 | 1940 | 0.3907        |
+| 1.1011 | 1950 | 0.4066        |
+| 1.1067 | 1960 | 0.3905        |
+| 1.1124 | 1970 | 0.3944        |
+| 1.1180 | 1980 | 0.4022        |
+| 1.1237 | 1990 | 0.398         |
+| 1.1293 | 2000 | 0.3473        |
+| 1.1350 | 2010 | 0.4357        |
+| 1.1406 | 2020 | 0.3823        |
+| 1.1462 | 2030 | 0.3628        |
+| 1.1519 | 2040 | 0.403         |
+| 1.1575 | 2050 | 0.3965        |
+| 1.1632 | 2060 | 0.3837        |
+| 1.1688 | 2070 | 0.5012        |
+| 1.1745 | 2080 | 0.3959        |
+| 1.1801 | 2090 | 0.3661        |
+| 1.1858 | 2100 | 0.4603        |
+| 1.1914 | 2110 | 0.4607        |
+| 1.1971 | 2120 | 0.4241        |
+| 1.2027 | 2130 | 0.5183        |
+| 1.2084 | 2140 | 0.3533        |
+| 1.2140 | 2150 | 0.3877        |
+| 1.2196 | 2160 | 0.4298        |
+| 1.2253 | 2170 | 0.4228        |
+| 1.2309 | 2180 | 0.4131        |
+| 1.2366 | 2190 | 0.4034        |
+| 1.2422 | 2200 | 0.3834        |
+| 1.2479 | 2210 | 0.4183        |
+| 1.2535 | 2220 | 0.5475        |
+| 1.2592 | 2230 | 0.4755        |
+| 1.2648 | 2240 | 0.4478        |
+| 1.2705 | 2250 | 0.3763        |
+| 1.2761 | 2260 | 0.4493        |
+| 1.2818 | 2270 | 0.4001        |
+| 1.2874 | 2280 | 0.3765        |
+| 1.2931 | 2290 | 0.3379        |
+| 1.2987 | 2300 | 0.337         |
+| 1.3043 | 2310 | 0.4143        |
+| 1.3100 | 2320 | 0.4794        |
+| 1.3156 | 2330 | 0.4004        |
+| 1.3213 | 2340 | 0.3674        |
+| 1.3269 | 2350 | 0.3963        |
+| 1.3326 | 2360 | 0.3896        |
+| 1.3382 | 2370 | 0.5062        |
+| 1.3439 | 2380 | 0.4114        |
+| 1.3495 | 2390 | 0.3955        |
+| 1.3552 | 2400 | 0.4682        |
+| 1.3608 | 2410 | 0.3551        |
+| 1.3665 | 2420 | 0.3536        |
+| 1.3721 | 2430 | 0.3784        |
+| 1.3778 | 2440 | 0.3456        |
+| 1.3834 | 2450 | 0.4273        |
+| 1.3890 | 2460 | 0.4005        |
+| 1.3947 | 2470 | 0.3957        |
+| 1.4003 | 2480 | 0.3371        |
+| 1.4060 | 2490 | 0.3451        |
+| 1.4116 | 2500 | 0.4735        |
+| 1.4173 | 2510 | 0.4013        |
+| 1.4229 | 2520 | 0.3751        |
+| 1.4286 | 2530 | 0.365         |
+| 1.4342 | 2540 | 0.3548        |
+| 1.4399 | 2550 | 0.4227        |
+| 1.4455 | 2560 | 0.3626        |
+| 1.4512 | 2570 | 0.404         |
+| 1.4568 | 2580 | 0.4055        |
+| 1.4625 | 2590 | 0.4513        |
+| 1.4681 | 2600 | 0.4147        |
+| 1.4737 | 2610 | 0.3623        |
+| 1.4794 | 2620 | 0.404         |
+| 1.4850 | 2630 | 0.4           |
+| 1.4907 | 2640 | 0.3854        |
+| 1.4963 | 2650 | 0.4082        |
+| 1.5020 | 2660 | 0.3502        |
+| 1.5076 | 2670 | 0.4022        |
+| 1.5133 | 2680 | 0.4479        |
+| 1.5189 | 2690 | 0.3456        |
+| 1.5246 | 2700 | 0.3992        |
+| 1.5302 | 2710 | 0.4143        |
+| 1.5359 | 2720 | 0.3925        |
+| 1.5415 | 2730 | 0.366         |
+| 1.5471 | 2740 | 0.4254        |
+| 1.5528 | 2750 | 0.4337        |
+| 1.5584 | 2760 | 0.471         |
+| 1.5641 | 2770 | 0.4201        |
+| 1.5697 | 2780 | 0.4357        |
+| 1.5754 | 2790 | 0.4289        |
+| 1.5810 | 2800 | 0.4287        |
+| 1.5867 | 2810 | 0.4349        |
+| 1.5923 | 2820 | 0.4551        |
+| 1.5980 | 2830 | 0.3562        |
+| 1.6036 | 2840 | 0.4925        |
+| 1.6093 | 2850 | 0.4104        |
+| 1.6149 | 2860 | 0.4691        |
+| 1.6206 | 2870 | 0.383         |
+| 1.6262 | 2880 | 0.3612        |
+| 1.6318 | 2890 | 0.4584        |
+| 1.6375 | 2900 | 0.3828        |
+| 1.6431 | 2910 | 0.3784        |
+| 1.6488 | 2920 | 0.4148        |
+| 1.6544 | 2930 | 0.4535        |
+| 1.6601 | 2940 | 0.3523        |
+| 1.6657 | 2950 | 0.3501        |
+| 1.6714 | 2960 | 0.3703        |
+| 1.6770 | 2970 | 0.388         |
+| 1.6827 | 2980 | 0.3846        |
+| 1.6883 | 2990 | 0.4212        |
+| 1.6940 | 3000 | 0.4192        |
+| 1.6996 | 3010 | 0.4265        |
+| 1.7053 | 3020 | 0.4385        |
+| 1.7109 | 3030 | 0.4197        |
+| 1.7165 | 3040 | 0.3488        |
+| 1.7222 | 3050 | 0.3666        |
+| 1.7278 | 3060 | 0.3909        |
+| 1.7335 | 3070 | 0.5085        |
+| 1.7391 | 3080 | 0.3495        |
+| 1.7448 | 3090 | 0.4198        |
+| 1.7504 | 3100 | 0.3647        |
+| 1.7561 | 3110 | 0.3873        |
+| 1.7617 | 3120 | 0.4038        |
+| 1.7674 | 3130 | 0.4471        |
+| 1.7730 | 3140 | 0.4078        |
+| 1.7787 | 3150 | 0.3823        |
+| 1.7843 | 3160 | 0.4852        |
+| 1.7899 | 3170 | 0.3891        |
+| 1.7956 | 3180 | 0.4334        |
+| 1.8012 | 3190 | 0.3836        |
+| 1.8069 | 3200 | 0.4           |
+| 1.8125 | 3210 | 0.4126        |
+| 1.8182 | 3220 | 0.3767        |
+| 1.8238 | 3230 | 0.4085        |
+| 1.8295 | 3240 | 0.3919        |
+| 1.8351 | 3250 | 0.358         |
+| 1.8408 | 3260 | 0.3709        |
+| 1.8464 | 3270 | 0.4131        |
+| 1.8521 | 3280 | 0.4082        |
+| 1.8577 | 3290 | 0.4547        |
+| 1.8634 | 3300 | 0.4317        |
+| 1.8690 | 3310 | 0.3981        |
+| 1.8746 | 3320 | 0.4585        |
+| 1.8803 | 3330 | 0.3698        |
+| 1.8859 | 3340 | 0.3662        |
+| 1.8916 | 3350 | 0.3955        |
+| 1.8972 | 3360 | 0.4387        |
+| 1.9029 | 3370 | 0.5001        |
+| 1.9085 | 3380 | 0.3708        |
+| 1.9142 | 3390 | 0.4448        |
+| 1.9198 | 3400 | 0.3632        |
+| 1.9255 | 3410 | 0.4589        |
+| 1.9311 | 3420 | 0.4085        |
+| 1.9368 | 3430 | 0.3993        |
+| 1.9424 | 3440 | 0.4598        |
+| 1.9481 | 3450 | 0.4019        |
+| 1.9537 | 3460 | 0.4179        |
+| 1.9593 | 3470 | 0.3804        |
+| 1.9650 | 3480 | 0.4229        |
+| 1.9706 | 3490 | 0.3933        |
+| 1.9763 | 3500 | 0.4217        |
+| 1.9819 | 3510 | 0.4182        |
+| 1.9876 | 3520 | 0.5265        |
+| 1.9932 | 3530 | 0.403         |
+| 1.9989 | 3540 | 0.3758        |
+
+</details>
+
+### Framework Versions
+- Python: 3.10.16
+- Sentence Transformers: 5.1.1
+- Transformers: 4.55.4
+- PyTorch: 2.6.0+cu124
+- Accelerate: 1.8.0.dev0
+- Datasets: 3.2.0
+- Tokenizers: 0.21.4
+
+## Citation
+
+### BibTeX
+
+#### Sentence Transformers
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084",
+}
+```
+
+#### MultipleNegativesRankingLoss
+```bibtex
+@misc{henderson2017efficient,
+    title={Efficient Natural Language Response Suggestion for Smart Reply},
+    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
+    year={2017},
+    eprint={1705.00652},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "MistralModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": null,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mistral",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.55.4",
+  "use_cache": false,
+  "vocab_size": 32000
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "SentenceTransformer",
+  "__version__": {
+    "sentence_transformers": "5.1.1",
+    "transformers": "4.55.4",
+    "pytorch": "2.6.0+cu124"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b854e9aacc16eaa0006a0f7cc62b0c3be2829841c470830f69b5c117f08230c7
+size 4943161760

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8bb4d21e13ec794602e019ddfb3fd6513d6f977e27de30ee109fb554f3ac4be
+size 4999818704

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a222fc5797efc6e22d094ccd0bfd3475598df41cb69a435fcbafd0a61e9c7493
+size 4278371712

model.safetensors.index.json

@@ -0,0 +1,298 @@
+{
+  "metadata": {
+    "total_parameters": 7110660096,
+    "total_size": 14221320192
+  },
+  "weight_map": {
+    "embed_tokens.weight": "model-00001-of-00003.safetensors",
+    "layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.10.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.10.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.11.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.12.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.13.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.14.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.15.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.16.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.17.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.18.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.18.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.18.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.19.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.19.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.19.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.20.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.20.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.20.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.20.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.input_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.21.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
+    "layers.21.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.21.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.22.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.22.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.22.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.22.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.22.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.22.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.22.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.22.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.22.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
+    "layers.23.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.23.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.23.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.23.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.24.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.24.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.24.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.25.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.25.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.25.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.26.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.26.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.26.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.27.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.27.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.27.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.28.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.28.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.28.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.29.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.29.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.29.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.30.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.30.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.30.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.30.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.input_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.31.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
+    "layers.31.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.31.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
+    "layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.5.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.6.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.7.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.8.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.input_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.9.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
+    "layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
+    "layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
+    "norm.weight": "model-00003-of-00003.safetensors"
+  }
+}

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_SparseEmbedding",
+    "type": "sparse_embedding.SparseEmbedding"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 32768,
+    "do_lower_case": false
+}

sparse_embedding.py

@@ -0,0 +1,41 @@
+from .zero_neuron import ZeroNeuron
+from sentence_transformers.models import Module
+import torch
+
+class SparseEmbedding(Module):
+    """ This module should be applied last (after Pooling, Normalize, etc.) """
+    config_keys = ["n_in", "init_mean", "init_std", "temperature", "stretch", "eps"]
+    def __init__(self,
+                 n_in: int,
+                 init_mean: float = 0.5,
+                 init_std: float = 0.01,
+                 temperature: float = 1.0,
+                 stretch: float = 0.1,
+                 eps: float = 1e-6):
+        super(SparseEmbedding, self).__init__()
+        self.n_in = n_in
+        self.init_mean = init_mean
+        self.init_std = init_std
+        self.temperature = temperature
+        self.stretch = stretch
+        self.eps = eps
+        self.sparsifyer = ZeroNeuron(
+            in_features=n_in,
+            out_features=n_in,
+            init_mean=init_mean,
+            init_std=init_std,
+            temperature=temperature,
+            stretch=stretch,
+            eps=eps
+            )
+        
+    def forward(self, features, *args, **kwargs):
+        mask = self.sparsifyer(features["sentence_embedding"], dim=kwargs.get("dim", None))
+        features["mask"] = mask
+        features["sparsity_loss"] = self.sparsifyer.l0_norm(features["sentence_embedding"])
+
+        return features
+    
+    def save(self, output_path: str):
+        self.save_config(output_path)
+        torch.save(self.sparsifyer.state_dict(), output_path + "/pytorch_model.bin")

special_tokens_map.json

@@ -0,0 +1,35 @@
+{
+  "additional_special_tokens": [
+    "<unk>",
+    "<s>",
+    "</s>"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

tokenizer_config.json

@@ -0,0 +1,48 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": true,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<unk>",
+    "<s>",
+    "</s>"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "model_max_length": 32768,
+  "pad_token": "</s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

zero_neuron.py

@@ -0,0 +1,134 @@
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class ZeroNeuron(nn.Module):
+    """
+    Adapted from: https://github.com/asappresearch/flop/blob/master/flop/hardconcrete.py
+    We replace 'self.log_alpha = nn.Parameter...' with something input-dependant: 'self.log_alpha = nn.Linear(...)'
+
+    >>> import torch
+    >>> x = torch.rand(12, 100)
+    >>> module = HardConcrete(in_features=100, out_features=100)
+    >>> mask = module(x)
+    >>> norm = module.l0_norm()
+
+    """
+
+    def __init__(self,
+                 in_features: int,
+                 out_features: int,
+                 init_mean: float = 0.5,
+                 init_std: float = 0.01,
+                 temperature: float = 1.0,
+                 stretch: float = 0.1,
+                 eps: float = 1e-6) -> None:
+        """Initialize the HardConcrete module.
+
+        Parameters
+        ----------
+        in_features : int
+            The features of the input X.
+        out_features: int
+            The dimension of the sparsity (should be 1 if you want sparsity to be applied on the penultimate dimension of X)
+        init_mean : float, optional
+            Initialization value for hard concrete parameter,
+            by default 0.5.,
+        init_std: float, optional
+            Used to initialize the hard concrete parameters,
+            by default 0.01.
+        temperature : float, optional
+            Temperature used to control the sharpness of the
+            distribution, by default 1.0
+        stretch : float, optional
+            Stretch the sampled value from [0, 1] to the interval
+            [-stretch, 1 + stretch], by default 0.1.
+
+        """
+        super().__init__()
+
+        self.in_features = in_features
+        self.out_features = out_features
+        self.limit_l = -stretch
+        self.limit_r = 1.0 + stretch
+        # we use a low-rank structure to reduce the computation cost.
+        if self.out_features > 1:
+            self.log_alpha = nn.Sequential(nn.Linear(in_features, 1, bias=False), nn.Linear(1, out_features, bias=False))
+        else:
+            self.log_alpha = nn.Linear(in_features, 1, bias=False)
+
+        self.beta = temperature
+        self.init_mean = init_mean
+        self.init_std = init_std
+        self.bias = -self.beta * math.log(-self.limit_l / self.limit_r)
+
+        self.eps = eps
+        self.log_alpha.apply(self.reset_parameters)
+
+    @torch.no_grad()
+    def reset_parameters(self, module):
+        """Reset the parameters of this module."""
+        mean = math.log(1 - self.init_mean) - math.log(self.init_mean)
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean, self.init_std)
+
+    def l0_norm(self, x: torch.Tensor, log_alpha=None) -> torch.Tensor:
+        """Compute the expected L0 norm of this mask.
+
+        Returns
+        -------
+        torch.Tensor
+            The expected L0 norm.
+
+        """
+        log_alpha = self.log_alpha(x).squeeze(-1) if log_alpha is None else log_alpha
+        return (log_alpha + self.bias).sigmoid().mean()
+
+    def forward(self, x: torch.Tensor, dim=None) -> torch.Tensor:  # type: ignore
+        """Sample a harconcrete mask.
+
+        Returns
+        -------
+        torch.Tensor
+            The sampled binary mask
+
+        """
+        log_alpha = self.log_alpha(x).squeeze(-1)
+        
+        if self.training:
+            # print(self.log_alpha[0].weight)
+            # Sample mask dynamically
+            u = torch.rand_like(log_alpha).clamp(self.eps, 1 - self.eps)
+            s = F.sigmoid((torch.log(u / (1 - u)) + log_alpha) / self.beta)
+            s = s * (self.limit_r - self.limit_l) + self.limit_l
+            mask = s.clamp(min=0., max=1.)
+
+        else:
+            # TODO: use this approach when dim is specified, other wise use per-sample / per-token sparsity
+            if dim is not None:
+                expected_num_zeros = dim
+            else:
+                # Get expected sparsity
+                sparsity_axis = self.out_features if self.out_features != 1 else x.shape[-1]
+                # b, s
+                expected_num_zeros = sparsity_axis - (log_alpha + self.bias).sigmoid().mean().item()
+            num_zeros = round(expected_num_zeros)
+            # Approximate expected value of each mask variable z;
+            # We use an empirically validated magic number 0.8
+            soft_mask = F.sigmoid(log_alpha / self.beta * 0.8)
+            # Prune small values to set to 0
+            _, indices = torch.topk(soft_mask, k=num_zeros, largest=False)
+            soft_mask[..., indices] = 0.
+            self.compiled_mask = soft_mask
+            mask = self.compiled_mask
+
+        return mask
+
+    def extre_repr(self) -> str:
+        return f"in_features={self.in_features}, out_features={self.out_features}"
+
+    def __repr__(self) -> str:
+        return "{}({})".format(self.__class__.__name__, self.extre_repr())