---
license: cc-by-4.0
language:
- en
metrics:
- matthews_correlation
- accuracy
base_model:
- state-spaces/mamba-130m
tags:
- text-classification
- nli
- mamba
---

# Model Card for 11128093-11066053-nli

<!-- Provide a quick summary of what the model is/does. -->

A binary Natural Language Inference classifier fine-tuned on the provided COMP34812 dataset using the Mamba state space model.


## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

This model extends the state-spaces/mamba-130m architecture for binary NLI tasks (entailment vs. non-entailment). It uses a custom classification head and was fine-tuned on the COMP34812 NLI dataset.

- **Developed by:** Patrick Mermelstein Lyons and Dev Soneji
- **Language(s):** English
- **Model type:** Supervised
- **Model architecture:** Non-Transformers (Selective State Spaces)
- **Finetuned from model [optional]:** state-spaces/mamba-130m

### Model Resources

<!-- Provide links where applicable. -->

- **Repository:** https://huggingface.co/state-spaces/mamba-130m
- **Paper or documentation:** https://arxiv.org/pdf/2312.00752.pdf

## Training Details

### Training Data

<!-- This is a short stub of information on the training data that was used, and documentation related to data pre-processing or additional filtering (if applicable). -->

The COMP34812 NLI train dataset (closed-source task-specific dataset). 24.4K pairs of premise-hypothesis pairs, each with a binary entailment label.

### Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

#### Training Hyperparameters

<!-- This is a summary of the values of hyperparameters used in training the model. -->


      - learning_rate: 5e-5
      - train_batch_size: 4
      - eval_batch_size: 16
      - num_train_epochs: 5
      - lr_scheduler_type: cosine
      - warmup_ratio: 0.1
    

#### Speeds, Sizes, Times

<!-- This section provides information about how roughly how long it takes to train the model and the size of the resulting model. -->


      - total training time: 1 hour 17 minutes
      - number of epochs: 5
      - model size: ~500MB
    

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data & Metrics

#### Testing Data

<!-- This should describe any evaluation data used (e.g., the development/validation set provided). -->

The COMP34812 NLI dev dataset (closed-source task-specific dataset). 6.7K pairs of premise-hypothesis pairs, each with a binary entailment label.

#### Metrics

<!-- These are the evaluation metrics being used. -->


      - Accuracy
      - Matthews Correlation Coefficient (MCC)
    

### Results

The model achieved an accuracy of 82.4% and an MCC of 0.649.

## Technical Specifications

### Hardware


      - GPU: NVIDIA T4 (Google Colab)
      - VRAM: 15.0 GB
      - RAM: 12.7 GB
      - Disk: 2 GB for model and data
    

### Software


      - Python 3.10+
      - PyTorch
      - HuggingFace Transformers
      - mamba-ssm
      - datasets, evaluate, accelerate
    

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The model is limited to binary entailment detection and is trained exclusively on the COMP34812 dataset. Generalization outside of this dataset is untested. Sentence pairs longer than 128 tokens will be trunacted.

## Additional Information

<!-- Any other information that would be useful for other people to know. -->

Model checkpoints and tokenizer available at https://huggingface.co/patrickmlml/mamba_nli_ensemble. Hyperparameters were determined by closely following referenced literature.