README.md

---
developed_by: GivingTuesday Data Commons
license: apache-2.0
model_type: Classifier (BERT)
training_data:
  description: Dataset contained over 2k examples for training and validation
accuracy:
  test_examples: 500
  weighted_f1: 0.93
  macro_f1: 0.76
datasets:
  - GivingTuesday/religious_orgs_training
language: en
---


**Technical Specifications Document is available at**: [Technical Specifications](https://docs.google.com/document/d/1eLUFC-8FtJkaQT9dUhjwRRKn8bXrHaZsXdMlIvCoeT4/edit?usp=sharing)



# Non-Profit Mapping Project Documentation: Religious Orgs Segmentation

**Author**: Zilun Lin \- GivingTuesday Data Commons   
**Note for external readers:** Some Databricks links in this document point to internal notebooks and may not be accessible to people outside GivingTuesday.

# 1\. Approach

## Definition

We use the following definition for categorizing religious orgs provided in the academic literature:  
“Religious organizations are organizations whose identity and mission are derived from a religious or spiritual tradition and which operate as registered or unregistered, nonprofit, voluntary entities.” ([Source: Montclair.ed](https://www.montclair.edu/profilepages/media/11259/user/religiousorganizationsglobalencyclope.pdf))  
This definition is operationalized in how we prompt GPT 4 to classify the training and testing datasets. Namely, we give it information on the name, mission statement and key activities and prompt it to find mentions/wording/terminology that reveal an org’s religious affiliations.

## Religious Recipient Orgs

Using the 990 CN file and BERT classifier:  
(1) Categorise orgs by religion (if any) using name (23F990-LINE-C), mission statement (23F990-PART-03-LINE-1) and activities (20F990-PART-03-LINE-4A, 4B, 4C).

Using the 990 EZ file and BERT classifier:  
(1) Categorise orgs by religion (if any) using name (23F990-LINE-C), primary exempt purpose (F9\_03\_PZ\_MISSIODESCES) and program accomplishment description(F9\_03\_PZ\_PRSRACACDEES).

# 2\. Code documentation

The code has two parts. The first consists of a notebook where the fine-tuning and testing datasets are constructed and uploaded to huggingface. This latter notebook is stored on databricks. The second consists of a Google Colab notebook where the actual fine-tuning of the model is done and its accuracy is tested. We use Google Colab because fine-tuning libraries can’t seem to be run on my databricks environment and I couldn’t figure out how to fix it. Google Colab also has the upside of giving me access to the much more powerful A100 GPUs (after paying a small fee) that is 10x quicker for fine-tuning.

All of the notebooks should be reasonably documented. Please message Zilun Lin if there are any mistakes or missing documentation.

## Classifying and formatting datasets

This notebook randomly samples from the 990 datamart and classifies the sample orgs using GPT4. It also generates a curated dataset of artificial orgs that are associated with under-represented religions. These two datasets are combined, formatted into an appropriate instruct-prompt-output format for fine-tuning and uploaded to HuggingFace. The final dataset has over 2k examples for training and validation, and 500 examples for testing.

[Link to EDA Notebook (Databricks)](https://dbc-3a4d04f2-8cab.cloud.databricks.com/editor/notebooks/1182041857993717?o=4203893953353865)

## Fine-tuning the LLM and testing for accuracy

We downloaded the fine-tuning dataset from HuggingFace and fine-tuned a set of LLMs. The resulting models are uploaded to HuggingFace. We also test these model’s accuracy on an unseen testing dataset.

(Llama Models)  
[Llama Model Fine-tuning (Google Collab)](https://colab.research.google.com/drive/1tZBVcQ_XQeb11HUBKxKjPTGBhMwJCiDF?usp=sharing)

(Bert models)  
[BERT Model Fine-tuning (Google Collab)](https://colab.research.google.com/drive/1OaV9wwqCzWqRXFmKzzDYW3Hwq_zwaUE5?usp=sharing)

# 3\. Outputs and Results

The two notebooks’ final outputs are two models:

- BERT base curated  
- Llama 3.2 3B curated

The curated models are fine-tuned using the combined dataset of actual and artificial orgs. The curated models should have better accuracy on under-represented orgs.

The following are the accuracy measures (higher is better) for the fine-tuned/trained models.

- BERT base curated:  
  - Weighted F1 score: 0.93  
  - Macro F1 score: 0.76  
- Llama 3.2 3B curated:  
  - Weighted F1 score: 0.85  
  - Macro F1 score: 0.27

The following are the speed of inference measures (lower is better) for the fine-tuned/trained models.

- BERT base curated time taken for 500 inferences: 3 seconds.  
- Llama 3.2 3B curated time taken for 500 inferences: \~10 minutes

When comparing models for text classification, BERT stands out for its fast inference speed and strong classification performance, as indicated by its F1 scores. The F1 score is a measure that balances precision and recall, offering a robust metric for evaluating classification tasks. BERT performs better than LLaMa in both the weighted and the macro F1 metrics, suggesting that it has a high accuracy in general and is adept at categorizing less frequent religious affiliations as well.

Aside from less accurate classification, the larger Llama 3.2 3B model also has a major drawback: its inference speed is significantly slower, and hosting the model is resource-intensive. For example, a linear extrapolation suggests that categorizing 200,000 organizations with Llama 3.2 3B could take approximately 4,000 minutes. This makes it less practical for scenarios where processing speed is a priority.

In comparison, BERT is much faster for inference, thanks to its streamlined model architecture, which is specifically optimized for tasks like classification. This makes BERT a more practical choice for applications requiring both speed and great classification performance.

# 4\. Deployment

The curated BERT model is now hosted on MLFlow (Databricks) in the model registry under the name \`religious\_orgs\_model\`, and has been released to the public under the apache-2 license on [Huggingface](https://huggingface.co/GivingTuesday/religious_org_v1). The processed data will be available for download in a data mart or API.

The API endpoint will output five fields, three for BERT classification and two based on 1023 EZ data availability:  
BERT Natural Language Outputs:   
(1) Religious Classification (and classification probability)  
(2) Classification probability for the designated religious affiliations (and classification probability)  
(3) Classification probability for whether the organisation is religious or not (and probability)

# 5\. Example usage

This is a minimal-code example of how to use this model to classify texts as related to a religion (or none). This model is public and doesn't require login to download. Use `snapshot_download` to get the latest version of the full model.
```
repo_id = 'GivingTuesday/religious_org_v1'
from huggingface_hub import snapshot_download
snapshot_download(repo_id)

# snapshot_download will return / print a location for the model. Pass that into the function below.
from transformers import pipeline
# 1. Define the local path to your downloaded model directory
# Replace './my_local_model_directory' with the actual path on your machine
model_path = './my_local_model_directory' # copied from output of previous cell

# 2. Load the text classification pipeline from the local directory
# The 'pipeline' function automatically loads the model, tokenizer, and config
# from the specified local path.
try:
    classifier = pipeline(
        task="text-classification",
        model=model_path,
        tokenizer=model_path
    )
    print(f"Model loaded successfully from {model_path}")
except Exception as e:
    print(f"Error loading model: {e}")
    print("Please ensure the directory contains the model weights, config, and tokenizer files.")
    exit()

# 3. Define the string(s) you want to classify
sentences = [
    """St. John's Episcopal is a really beautiful community that preaches "faith in action" -- they welcome and assist migrants, partner with community organizations, run a housing first apartment building for unhoused folks, and they are part of the Together Colorado coalition of faith-based organizations that lobbies for anti-poverty legislation. Bonus: the cathedral is gorgeous and historic.""",
    """I’m atheist but rehearse in a community choir at Christ Church United Methodist and they have pro LGBTQ+ stuff everywhere. They even do a reconciliation Sunday service every year in June during pride month""",
]

# 4. Run the classification on the input string(s)
results = classifier(sentences)

# 5. Print the results
for i, result in enumerate(results):
    print(f"\nText: '{sentences[i]}'")
    print(f"Classification Results:")
    # Results can be a list of dictionaries if top_k is used
    if isinstance(result, list):
        for label_info in result:
            print(f"- Label: {label_info['label']}, Score: {label_info['score']:.4f}")
    else:
        print(f"- Label: {result['label']}, Score: {result['score']:.4f}")

```

### Output (actual reddit comments):
```
Text: 'St. John's Episcopal is a really beautiful community that preaches "faith in action" -- they welcome and assist migrants, partner with community organizations, run a housing first apartment building for unhoused folks, and they are part of the Together Colorado coalition of faith-based organizations that lobbies for anti-poverty legislation. Bonus: the cathedral is gorgeous and historic.'
Classification Results:
- Label: Christianity, Score: 0.8822

Text: 'I’m atheist but rehearse in a community choir at Christ Church United Methodist and they have pro LGBTQ+ stuff everywhere. They even do a reconciliation Sunday service every year in June during pride month'
Classification Results:
- Label: Christianity, Score: 0.9899
```