--- developed_by: GivingTuesday Data Commons license: apache-2.0 model_type: Classifier (BERT) training_data: description: Dataset contained over 2k examples for training and validation accuracy: test_examples: 500 weighted_f1: 0.93 macro_f1: 0.76 datasets: - GivingTuesday/religious_orgs_training language: en --- **Technical Specifications Document is available at**: [Technical Specifications](https://docs.google.com/document/d/1eLUFC-8FtJkaQT9dUhjwRRKn8bXrHaZsXdMlIvCoeT4/edit?usp=sharing) # Non-Profit Mapping Project Documentation: Religious Orgs Segmentation **Author**: Zilun Lin \- GivingTuesday Data Commons **Note for external readers:** Some Databricks links in this document point to internal notebooks and may not be accessible to people outside GivingTuesday. # 1\. Approach ## Definition We use the following definition for categorizing religious orgs provided in the academic literature: “Religious organizations are organizations whose identity and mission are derived from a religious or spiritual tradition and which operate as registered or unregistered, nonprofit, voluntary entities.” ([Source: Montclair.ed](https://www.montclair.edu/profilepages/media/11259/user/religiousorganizationsglobalencyclope.pdf)) This definition is operationalized in how we prompt GPT 4 to classify the training and testing datasets. Namely, we give it information on the name, mission statement and key activities and prompt it to find mentions/wording/terminology that reveal an org’s religious affiliations. ## Religious Recipient Orgs Using the 990 CN file and BERT classifier: (1) Categorise orgs by religion (if any) using name (23F990-LINE-C), mission statement (23F990-PART-03-LINE-1) and activities (20F990-PART-03-LINE-4A, 4B, 4C). Using the 990 EZ file and BERT classifier: (1) Categorise orgs by religion (if any) using name (23F990-LINE-C), primary exempt purpose (F9\_03\_PZ\_MISSIODESCES) and program accomplishment description(F9\_03\_PZ\_PRSRACACDEES). # 2\. Code documentation The code has two parts. The first consists of a notebook where the fine-tuning and testing datasets are constructed and uploaded to huggingface. This latter notebook is stored on databricks. The second consists of a Google Colab notebook where the actual fine-tuning of the model is done and its accuracy is tested. We use Google Colab because fine-tuning libraries can’t seem to be run on my databricks environment and I couldn’t figure out how to fix it. Google Colab also has the upside of giving me access to the much more powerful A100 GPUs (after paying a small fee) that is 10x quicker for fine-tuning. All of the notebooks should be reasonably documented. Please message Zilun Lin if there are any mistakes or missing documentation. ## Classifying and formatting datasets This notebook randomly samples from the 990 datamart and classifies the sample orgs using GPT4. It also generates a curated dataset of artificial orgs that are associated with under-represented religions. These two datasets are combined, formatted into an appropriate instruct-prompt-output format for fine-tuning and uploaded to HuggingFace. The final dataset has over 2k examples for training and validation, and 500 examples for testing. [Link to EDA Notebook (Databricks)](https://dbc-3a4d04f2-8cab.cloud.databricks.com/editor/notebooks/1182041857993717?o=4203893953353865) ## Fine-tuning the LLM and testing for accuracy We downloaded the fine-tuning dataset from HuggingFace and fine-tuned a set of LLMs. The resulting models are uploaded to HuggingFace. We also test these model’s accuracy on an unseen testing dataset. (Llama Models) [Llama Model Fine-tuning (Google Collab)](https://colab.research.google.com/drive/1tZBVcQ_XQeb11HUBKxKjPTGBhMwJCiDF?usp=sharing) (Bert models) [BERT Model Fine-tuning (Google Collab)](https://colab.research.google.com/drive/1OaV9wwqCzWqRXFmKzzDYW3Hwq_zwaUE5?usp=sharing) # 3\. Outputs and Results The two notebooks’ final outputs are two models: - BERT base curated - Llama 3.2 3B curated The curated models are fine-tuned using the combined dataset of actual and artificial orgs. The curated models should have better accuracy on under-represented orgs. The following are the accuracy measures (higher is better) for the fine-tuned/trained models. - BERT base curated: - Weighted F1 score: 0.93 - Macro F1 score: 0.76 - Llama 3.2 3B curated: - Weighted F1 score: 0.85 - Macro F1 score: 0.27 The following are the speed of inference measures (lower is better) for the fine-tuned/trained models. - BERT base curated time taken for 500 inferences: 3 seconds. - Llama 3.2 3B curated time taken for 500 inferences: \~10 minutes When comparing models for text classification, BERT stands out for its fast inference speed and strong classification performance, as indicated by its F1 scores. The F1 score is a measure that balances precision and recall, offering a robust metric for evaluating classification tasks. BERT performs better than LLaMa in both the weighted and the macro F1 metrics, suggesting that it has a high accuracy in general and is adept at categorizing less frequent religious affiliations as well. Aside from less accurate classification, the larger Llama 3.2 3B model also has a major drawback: its inference speed is significantly slower, and hosting the model is resource-intensive. For example, a linear extrapolation suggests that categorizing 200,000 organizations with Llama 3.2 3B could take approximately 4,000 minutes. This makes it less practical for scenarios where processing speed is a priority. In comparison, BERT is much faster for inference, thanks to its streamlined model architecture, which is specifically optimized for tasks like classification. This makes BERT a more practical choice for applications requiring both speed and great classification performance. # 4\. Deployment The curated BERT model is now hosted on MLFlow (Databricks) in the model registry under the name \`religious\_orgs\_model\`, and has been released to the public under the apache-2 license on [Huggingface](https://huggingface.co/GivingTuesday/religious_org_v1). The processed data will be available for download in a data mart or API. The API endpoint will output five fields, three for BERT classification and two based on 1023 EZ data availability: BERT Natural Language Outputs: (1) Religious Classification (and classification probability) (2) Classification probability for the designated religious affiliations (and classification probability) (3) Classification probability for whether the organisation is religious or not (and probability) # 5\. Example usage This is a minimal-code example of how to use this model to classify texts as related to a religion (or none). This model is public and doesn't require login to download. Use `snapshot_download` to get the latest version of the full model. ``` repo_id = 'GivingTuesday/religious_org_v1' from huggingface_hub import snapshot_download snapshot_download(repo_id) # snapshot_download will return / print a location for the model. Pass that into the function below. from transformers import pipeline # 1. Define the local path to your downloaded model directory # Replace './my_local_model_directory' with the actual path on your machine model_path = './my_local_model_directory' # copied from output of previous cell # 2. Load the text classification pipeline from the local directory # The 'pipeline' function automatically loads the model, tokenizer, and config # from the specified local path. try: classifier = pipeline( task="text-classification", model=model_path, tokenizer=model_path ) print(f"Model loaded successfully from {model_path}") except Exception as e: print(f"Error loading model: {e}") print("Please ensure the directory contains the model weights, config, and tokenizer files.") exit() # 3. Define the string(s) you want to classify sentences = [ """St. John's Episcopal is a really beautiful community that preaches "faith in action" -- they welcome and assist migrants, partner with community organizations, run a housing first apartment building for unhoused folks, and they are part of the Together Colorado coalition of faith-based organizations that lobbies for anti-poverty legislation. Bonus: the cathedral is gorgeous and historic.""", """I’m atheist but rehearse in a community choir at Christ Church United Methodist and they have pro LGBTQ+ stuff everywhere. They even do a reconciliation Sunday service every year in June during pride month""", ] # 4. Run the classification on the input string(s) results = classifier(sentences) # 5. Print the results for i, result in enumerate(results): print(f"\nText: '{sentences[i]}'") print(f"Classification Results:") # Results can be a list of dictionaries if top_k is used if isinstance(result, list): for label_info in result: print(f"- Label: {label_info['label']}, Score: {label_info['score']:.4f}") else: print(f"- Label: {result['label']}, Score: {result['score']:.4f}") ``` ### Output (actual reddit comments): ``` Text: 'St. John's Episcopal is a really beautiful community that preaches "faith in action" -- they welcome and assist migrants, partner with community organizations, run a housing first apartment building for unhoused folks, and they are part of the Together Colorado coalition of faith-based organizations that lobbies for anti-poverty legislation. Bonus: the cathedral is gorgeous and historic.' Classification Results: - Label: Christianity, Score: 0.8822 Text: 'I’m atheist but rehearse in a community choir at Christ Church United Methodist and they have pro LGBTQ+ stuff everywhere. They even do a reconciliation Sunday service every year in June during pride month' Classification Results: - Label: Christianity, Score: 0.9899 ```