README.md

metadata

base_model: bert-base-uncased
datasets:
  - rajpurkar/squad
language: en
library_name: sklearn
license: apache-2.0
pipeline_tag: question-answering
tags:
  - question-answering
  - bert
  - squad
  - feature-extraction
  - logistic-regression
  - baseline
model-index:
  - name: bert-squad-qa-feature-based
    results:
      - task:
          type: question-answering
        dataset:
          name: SQuAD v1.1
          type: rajpurkar/squad
          split: validation
        metrics:
          - type: exact_match
            value: 14.35
          - type: f1
            value: 24.57

bert-squad-qa-feature-based

Baseline for extractive question answering on SQuAD v1.1: the frozen, original bert-base-uncased is used only as a feature extractor, and two scikit-learn Logistic Regression classifiers predict the answer span on top of its features. No BERT weight is trained.

The fully fine-tuned counterpart is published separately at Giobbva/bert-squad-qa-full-finetuning.

Model	Exact Match	F1
This model (frozen BERT + Logistic Regression)	14.35	24.57
Full fine-tuning (bert-squad-qa-full-finetuning)	69.04	79.53

Evaluated on the full SQuAD v1.1 validation set (10,570 questions).

Files in this repository

Two StandardScaler + LogisticRegression pipelines, stored as .joblib files:

• model1_start_classifier.joblib: scores each context word as the answer start
• model1_end_classifier.joblib: scores each context word as the answer end

This repository contains no transformer weights: load bert-base-uncased from its own repository.

Features

• Input: last_hidden_state (768-d) of the frozen bert-base-uncased at the first sub-token of each context word
• Question and context are encoded together (question [SEP] context), max length 384, stride 128

Training

• 15,000 SQuAD v1.1 training examples; per window, the start word, the end word and 10 randomly sampled negative context words
• LogisticRegression(max_iter=300) after StandardScaler, seed 42
• Span decoding: best start + end score with end >= start and at most 30 words

Usage

import joblib
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer

start_classifier = joblib.load(hf_hub_download("Giobbva/bert-squad-qa-feature-based", "model1_start_classifier.joblib"))
end_classifier = joblib.load(hf_hub_download("Giobbva/bert-squad-qa-feature-based", "model1_end_classifier.joblib"))
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
bert = AutoModel.from_pretrained("bert-base-uncased").eval()

question, context = "Where is the Eiffel Tower?", "The Eiffel Tower is in Paris."
encoded = tokenizer(question, context, truncation="only_second", max_length=384, return_tensors="pt")
with torch.no_grad():
    hidden = bert(**encoded).last_hidden_state[0].numpy()

# first sub-token of each context word
word_ids, sequence_ids = encoded.word_ids(), encoded.sequence_ids()
positions = [i for i, w in enumerate(word_ids) if sequence_ids[i] == 1 and word_ids[i - 1] != w]
start_scores = start_classifier.decision_function(hidden[positions])
end_scores = end_classifier.decision_function(hidden[positions])

s = int(start_scores.argmax())
e = s + int(end_scores[s:s + 30].argmax())
start_char = encoded.word_to_chars(word_ids[positions[s]], sequence_index=1).start
end_char = encoded.word_to_chars(word_ids[positions[e]], sequence_index=1).end
print(context[start_char:end_char])