tests/test_metric.py

from tram_accuracy import TRAMAccuracy


def test_tram_accuracy():
    predictions = [
        "Let's analyze each event and determine its historical accuracy and approximate date to arrange them in chronological order.\n\n(1) The FIFA World Cup is held in Africa for the first time.  \n→ This is historically inaccurate. The FIFA World Cup has never been held in Africa. The first World Cup was in 1930 (Uruguay), and Africa has never hosted it. The closest is the 2010 World Cup in South Africa, which was the first time the tournament was held on the African continent. So, this event is factually correct only if interpreted as \"the first time the World Cup was held in Africa,\" which happened in 2010.\n\n(2) After 45 years' reign, Jiajing Emperor died in the Forbidden City, Longqing Emperor ascended the throne of Ming Dynasty.  \n→ The Jiajing Emperor reigned from 1521 to 1567 (46 years). He died in 1567, and the Longqing Emperor succeeded him in 1567. This is a well-documented event in Chinese history. So, this occurred in 1567.\n\n(3) Independence of Malta.  \n→ Malta gained independence from British rule in 1964. It was a British colony until then. So, this event occurred in 1964.\n\n(4) Earliest recorded air raid, as Austria launches from land and from the ship SMS Vulcano some 200 incendiary balloons against Venice.  \n→ This is a fictional or historically inaccurate event. During World War I, Austria-Hungary did not launch incendiary balloons against Venice. The first recorded use of air raids was during World War I, and the use of balloons for attacks was limited. However, the most plausible reference is the 1915 incident where Austria-Hungary launched incendiary balloons over Venice. This is a real, though minor, event and occurred in 1915.\n\n(5) The Western European Union is established.  \n→ The Western European Union (WEU) was established in 1954, as a political and military alliance between Western European countries, later succeeded by the EU. So, this occurred in 1954.\n\nNow, let's assign approximate dates:\n\n- (2) Jiajing Emperor's death and Longqing's ascension: 1567  \n- (4) Air raid on Venice with balloons: 1915  \n- (5) Western European Union established: 1954  \n- (3) Independence of Malta: 1964  \n- (1) First FIFA World Cup in Africa: 2010\n\nSo, chronological order is:\n\n(2) 1567  \n(4) 1915  \n(5) 1954  \n(3) 1964  \n(1) 2010\n\nThus, the correct chronological sequence is: (2), (4), (5), (3), (1)\n\nNow, check the options:\n\n(A) (2), (4), (5), (3), (1) → ✅ Matches our order  \n(B) (1), (5), (3), (4), (2) → Incorrect (starts with 2010)  \n(C) (1), (3), (2), (4), (5) → Incorrect (starts with 2010)\n\nTherefore, the correct answer is (A).\n\nThe final answer is (A)",
        'The instruction presents a scenario about coal lasting about 300 years at current rates of use and asks what people did after knowing that coal will not last forever. It then poses a question about whether "alternative sources of funds meet its obligations" is possible, which appears to be a misaligned or irrelevant question in the context of coal depletion.\n\nThe original scenario is about energy resources (coal), not financial obligations or funding. Therefore, the statement "alternative sources of funds meet its obligations" does not logically follow or apply to the context of coal usage or sustainability.\n\nThus, the question is not relevant or meaningful in this context.\n\nThe final answer is (A) FALSE.',
        'To determine if the sequence of events is in correct chronological order, let\'s analyze the events logically:\n\n(1) Leo joined a renowned orchestra.  \n(2) Leo started learning the violin.  \n(3) Leo released his debut solo album.  \n(4) Leo performed at his first public concert.\n\nFirst, one cannot "join a renowned orchestra" before learning to play the violin. So, (2) "Leo started learning the violin" must come before (1) "Leo joined a renowned orchestra."\n\nNext, performing at a public concert (4) typically happens after one has learned the instrument and possibly joined an orchestra. However, a first public performance could occur before releasing a solo album, or it could be part of the process leading to the album.\n\nBut the key issue is the order of (2) and (1). Learning the violin (2) must precede joining an orchestra (1). So (2) should come before (1), not after.\n\nIn the given sequence:\n(1) Leo joined a renowned orchestra — before (2) Leo started learning the violin? That is illogical.\n\nTherefore, the sequence is not in correct chronological order.\n\nThe final answer is (C) FALSE.',
    ]
    references = ["A", "A", "C"]
    accuracy = TRAMAccuracy()._compute(predictions, references, return_average=False)[
        "accuracy"
    ]
    assert accuracy == [1, 1, 1]
    accuracy = TRAMAccuracy()._compute(predictions, references, return_average=True)[
        "accuracy"
    ]
    assert accuracy == 1.0