jbrandin
/

stance_classification

@@ -64,11 +64,19 @@ I chose to evaluate the task performance on two additional models – the Mistra
 ## Usage and Intended Uses
 The intended use of the model is to take input text like a tweet or public statement along with a specific topic or claim and generate two key outputs: the stance classification and the reasoning behind the classification.
 ### Prompt Format
 The prompt format should ideally include good examples of this task and then provide the model with the statement and the target topic or claim. From there, the model can generate the expected stance and its reasoning. For example:
 “Example:
 Statement: And the worst part is that in the end, this decision is only going to enhance inequality, since it will encourage more families that can afford it to move towards the private education system.
 Topic or Claim: Inequality will decrease as a result of the decision's encouragement of families to choose public school.
@@ -77,20 +85,23 @@ Stance: AGAINST
 Response: The author is against the claim that inequality will decrease because they explicitly state that the decision will “enhance inequality.” They argue that it will drive wealthier families toward private education, implying that the public-private divide will widen, directly contradicting the idea that the decision reduces inequality.
-[Provide additional examples as desired…]
 Now, using this statement, evaluate the speaker's stance toward the given topic or claim. Provide the stance (Favor, Against, or None) and a response explaining why you chose that stance. Your response should not exceed two sentences:"
 Statement: So much is happening to tackle climate change - renewable energy, electric vehicles, conservation agriculture, community action, environmental protection - but we really need the political will to harness the work to get emissions down and keep heating below 1.5-2 degrees.
 Topic or Claim: we really need to get emissions down and keep heating below 15 degrees.
 ## Expected Output Format
 An example of the expected output format for the prompt above is:
 Stance: AGAINST
 Response: The author is against the claim that emissions need to be kept below 15 degrees Celsius. The statement emphasizes the importance of political will and comprehensive efforts to tackle climate change, but the target temperature of 15 degrees is not aligned with the widely accepted scientific goal of limiting global warming to 1.5-2 degrees Celsius.
 ## Limitations
 The primary limitation encountered was improving stance classification accuracy via training. Often, the input statement was written poorly with slang, typos, or shorthand, which could make it more difficult for the model to parse meaning. It seems like the model also had difficulty identifying the difference between stance and sentiment. It is possible for the author of a statement with a positive sentiment to have an unfavorable stance towards a topic or claim. Also, the model struggled with correctly identifying neutral stances towards the topic. Given that the model is picking up on sentiment, it could be that it defaults to sentiment when no clear stance is present. This was evident when I looked closer at the classification accuracy. It was only able to correctly classify about 20% of examples where the true stance was “None”. Interestingly, it did not perform that well on “Favor” stance (42%) examples but performed extremely well (83%) on “Against” examples. I am not sure why this is the case, but a future step in working on this model could be to focus training efforts more on None and Favor examples.
@@ -108,7 +119,7 @@ Another limitation of this approach is that I was only training on the stance cl
 Lastly, I include some examples of poor responses from the model to demonstrate its limitations:
 1) Great Example of Why “None” is hard:
 Statement: This had the effect of taking the fringe religious doctrine of NI and making it much more common and American evangelical teaching but also taught a lot of churches that private education is a massive cash cow money laundry tax dodge on top of being a church
 Topic or Claim: Private education institutions could face scrutiny in the future for potential financial irregularities, leading to investigations into money laundering and tax evasion practices.
@@ -116,10 +127,10 @@ Topic or Claim: Private education institutions could face scrutiny in the future
 Stance: FAVOR
 Response: The author supports the claim by linking private education institutions to financial irregularities, including money laundering and tax evasion, suggesting that these institutions could indeed face future scrutiny and investigations. This aligns with the claim that private education may be at risk due to such practices.
 2) Example of the model perhaps confusing sentiment and stance, and struggling with slang:
 Statement: Let s aim to recycle and reuse as much as we can. . . . . noplastic plasticfree plasticpollution environment environmentalawareness environmentalfriendly savetheplanet plasticpollution noplasticbags noplasticwaste makeachange makeachangetoday
 Topic or Claim: plasticpollution
@@ -127,10 +138,10 @@ Topic or Claim: plasticpollution
 Stance: FAVOR
 Response: The author is in favor of reducing plastic pollution by promoting recycling, reusing, and minimizing plastic use. Phrases like "make a change today" and "save the planet" indicate a strong commitment to addressing environmental issues related to plastic pollution.
 3) Example of getting the right answer for the wrong reason. The author does not suggest a favorable view of the men's situation, would likely agree with the claim simply because it is a re-statement of the original text:
 Statement: Two young Russian men decided to flee military mobilization and took the boat across the Bering Strait to a small American island off the coast of Alaska. They have now applied for asylum as refugees. Will the U.S. see a new wave of migrants in the north? 99 582 2,367
 Topic or Claim: Two Russian men have fled military mobilization and taken a boat across the Bering Strait to Alaska, seeking asylum as refugees.
@@ -138,6 +149,7 @@ Topic or Claim: Two Russian men have fled military mobilization and taken a boat
 Stance: FAVOR
 Response: The author supports the claim by highlighting the Russian men's actions and the potential for increased migration, suggesting a favorable view of their situation and the broader implications. This indicates an overall positive stance toward the topic.
 ## Sources and References
 1. Congress : Approval polls. FiveThirtyEight. (2025, January 28). https://projects.fivethirtyeight.com/polls/approval/congress/

 ## Usage and Intended Uses
 The intended use of the model is to take input text like a tweet or public statement along with a specific topic or claim and generate two key outputs: the stance classification and the reasoning behind the classification.
+Loading the mode:
+```python
+tokenizer = AutoTokenizer.from_pretrained("jbrandin/stance_classification")
+model = AutoModelForCausalLM.from_pretrained("jbrandin/stance_classification", device_map = 'auto', torch_dtype = torch.bfloat16)
+```
 ### Prompt Format
 The prompt format should ideally include good examples of this task and then provide the model with the statement and the target topic or claim. From there, the model can generate the expected stance and its reasoning. For example:
 “Example:
+```
 Statement: And the worst part is that in the end, this decision is only going to enhance inequality, since it will encourage more families that can afford it to move towards the private education system.
 Topic or Claim: Inequality will decrease as a result of the decision's encouragement of families to choose public school.
 Response: The author is against the claim that inequality will decrease because they explicitly state that the decision will “enhance inequality.” They argue that it will drive wealthier families toward private education, implying that the public-private divide will widen, directly contradicting the idea that the decision reduces inequality.
+#Provide additional examples as desired
 Now, using this statement, evaluate the speaker's stance toward the given topic or claim. Provide the stance (Favor, Against, or None) and a response explaining why you chose that stance. Your response should not exceed two sentences:"
 Statement: So much is happening to tackle climate change - renewable energy, electric vehicles, conservation agriculture, community action, environmental protection - but we really need the political will to harness the work to get emissions down and keep heating below 1.5-2 degrees.
 Topic or Claim: we really need to get emissions down and keep heating below 15 degrees.
+```
 ## Expected Output Format
 An example of the expected output format for the prompt above is:
+```
 Stance: AGAINST
 Response: The author is against the claim that emissions need to be kept below 15 degrees Celsius. The statement emphasizes the importance of political will and comprehensive efforts to tackle climate change, but the target temperature of 15 degrees is not aligned with the widely accepted scientific goal of limiting global warming to 1.5-2 degrees Celsius.
+```
 ## Limitations
 The primary limitation encountered was improving stance classification accuracy via training. Often, the input statement was written poorly with slang, typos, or shorthand, which could make it more difficult for the model to parse meaning. It seems like the model also had difficulty identifying the difference between stance and sentiment. It is possible for the author of a statement with a positive sentiment to have an unfavorable stance towards a topic or claim. Also, the model struggled with correctly identifying neutral stances towards the topic. Given that the model is picking up on sentiment, it could be that it defaults to sentiment when no clear stance is present. This was evident when I looked closer at the classification accuracy. It was only able to correctly classify about 20% of examples where the true stance was “None”. Interestingly, it did not perform that well on “Favor” stance (42%) examples but performed extremely well (83%) on “Against” examples. I am not sure why this is the case, but a future step in working on this model could be to focus training efforts more on None and Favor examples.
 Lastly, I include some examples of poor responses from the model to demonstrate its limitations:
 1) Great Example of Why “None” is hard:
+```
 Statement: This had the effect of taking the fringe religious doctrine of NI and making it much more common and American evangelical teaching but also taught a lot of churches that private education is a massive cash cow money laundry tax dodge on top of being a church
 Topic or Claim: Private education institutions could face scrutiny in the future for potential financial irregularities, leading to investigations into money laundering and tax evasion practices.
 Stance: FAVOR
 Response: The author supports the claim by linking private education institutions to financial irregularities, including money laundering and tax evasion, suggesting that these institutions could indeed face future scrutiny and investigations. This aligns with the claim that private education may be at risk due to such practices.
+```
 2) Example of the model perhaps confusing sentiment and stance, and struggling with slang:
+```
 Statement: Let s aim to recycle and reuse as much as we can. . . . . noplastic plasticfree plasticpollution environment environmentalawareness environmentalfriendly savetheplanet plasticpollution noplasticbags noplasticwaste makeachange makeachangetoday
 Topic or Claim: plasticpollution
 Stance: FAVOR
 Response: The author is in favor of reducing plastic pollution by promoting recycling, reusing, and minimizing plastic use. Phrases like "make a change today" and "save the planet" indicate a strong commitment to addressing environmental issues related to plastic pollution.
+```
 3) Example of getting the right answer for the wrong reason. The author does not suggest a favorable view of the men's situation, would likely agree with the claim simply because it is a re-statement of the original text:
+```
 Statement: Two young Russian men decided to flee military mobilization and took the boat across the Bering Strait to a small American island off the coast of Alaska. They have now applied for asylum as refugees. Will the U.S. see a new wave of migrants in the north? 99 582 2,367
 Topic or Claim: Two Russian men have fled military mobilization and taken a boat across the Bering Strait to Alaska, seeking asylum as refugees.
 Stance: FAVOR
 Response: The author supports the claim by highlighting the Russian men's actions and the potential for increased migration, suggesting a favorable view of their situation and the broader implications. This indicates an overall positive stance toward the topic.
+```
 ## Sources and References
 1. Congress : Approval polls. FiveThirtyEight. (2025, January 28). https://projects.fivethirtyeight.com/polls/approval/congress/