app.py

# =========================================================
# app.py (STABLE VERSION - FIXED TOKEN OVERFLOW)
# =========================================================

import gradio as gr
from transformers import pipeline
from pypdf import PdfReader
from pdf2image import convert_from_path
import pytesseract
import tempfile

# =========================================================
# Models
# =========================================================

MODELS = {
    "English model (ubffm/academic_text_classifier_en)": "ubffm/academic_text_classifier_en",
    "German model (ubffm/academic_text_classifier_de)": "ubffm/academic_text_classifier_de",
}

DEFAULT_MODEL = "English model (ubffm/academic_text_classifier_en)"

# =========================================================
# Labels
# =========================================================

LABELS = ["OUT OF SCOPE", "MAIN TEXT", "EXAMPLE", "REFERENCE"]
DEFAULT_NOISE = ["OUT OF SCOPE", "REFERENCE"]

EXAMPLE_TEXT = (
    "SAR WSP2a5\n"
    "\n"
    "\n"
    " \n"
    "45\n"
    "\n"
    "Z E S Z Y T Y  N A U K O W E  WYśSZEJ SZKOŁY PEDAGOGICZNEJ \n"
    "W RZESZOWIE \n"
    "\n"
    " \n"
    "SERIA FILOLOGICZNA \n"
    "\n"
    "ZESZYT 42/2001  STUDIA ANGLICA RESOVIENSIA 2 \n"
    "\n"
    "Ewa KONIECZNA \n"
    "\n"
    "FIRST LANGUAGE ACQUISITION: PIAGET’S CONSTRUCTIVISM \n"
    "VERSUS CHOMSKY’S INNATENESS HYPOTHESIS \n"
    "\n"
    "Looking at the studies of child language development from a historical \n"
    "perspective, it is undeniable that the opinions of the majority of psycholinguists \n"
    "are convergent when it comes to the observable phenomena accompanying first \n"
    "language acquisition, as pointed out by Lewandowska-Tomaszczyk (1998:231). \n"
    "They distinguish several stages in this process, e.g. the first stage consists in the \n"
    "production of front nasalised vowels indicating some kind of discomfort to the \n"
    "baby. Next, irrespective of the native language of the baby’s parents, the period \n"
    "of babbling commences. It is characterised by the gradual emergence of \n"
    "consonants. These are the syllables such as: wa...wa...wa, la....la....la, etc., \n"
    "produced whenever babies are cheerful and pleased. The next stage is lalling \n"
    "during which the child engages in the repetition of the heard sounds. The \n"
    "intensification of this process is called echoalia. At the end of this stage children \n"
    "begin to produce their first words. Needless to say, lalling and echoalia mark the \n"
    "onset of speech production and remarkably differ from language to language.1 \n"
    "\n"
    "However, the interpretation of these phenomena has always been under \n"
    "dispute and it consequently divided linguists into adherents of two contradictory \n"
    "hypotheses: behaviorism on one side and innatism on the other. What is more, \n"
    "another theory was formulated, called constructivism, which can be described as \n"
    "being halfway between the former and the latter ones (see Lewandowska-\n"
    "Tomaszczyk (1980)). \n"
    "\n"
    "At this point let us consider the main assumptions of the above mentioned \n"
    "theories. In 1957 Skinner published his Verbal Behaviour, in which he claimed \n"
    "that language is a set of habits and can be taught by the process of instrumental \n"
    "\n"
    " \n"
    " 1 An example of echoalia in Polish and English respectively would be: pa-pa and bye-bye for \n"
    "\n"
    "saying ‘goodbye’ or mama and mummy for addressing a mother. On the other hand, the same \n"
    "syllable da is interpreted as ‘daddy’ by English-speaking parents and as „daj” (‘give’) by Polish or \n"
    "Russian-speaking parents.  \n"
    "\n"
    "\n"
    "\n"
    " \n"
    "46\n"
    "\n"
    "conditioning, i.e. by developing a relationship between a particular stimulus (S) \n"
    "and a desired response (R). This takes place when appropriate responses are \n"
    "reinforced until they become automatic and mechanical. In other words, \n"
    "language acquisition takes place when the child is given positive feedback from \n"
    "his environment for correct utterances. In contrast, faulty vocalisations don’t get \n"
    "established as habits because they don’t meet with the parents’ or caretakers’ \n"
    "approval. \n"
    "\n"
    "Two years later, however, radical behaviourism came in for a bitter criticism \n"
    "when Chomsky (1959) presented a completely different view of language \n"
    "acquisition. In his view, language is not a set of habits, but it is rule-governed; \n"
    "subsequently, the mind is responsible for the perception and processing of \n"
    "linguistic data because it is genetically equipped with devices that make \n"
    "language acquisition possible. This mechanism is referred to as LAD (language \n"
    "acquisition device). Another notion introduced by Chomsky is that of language \n"
    "universals, which he defined as abstract representations of grammatical rules. \n"
    "They are general, i.e. they pertain to all natural languages. In the generative-\n"
    "transformational approach, language acquisition takes place according to a \n"
    "genetic programme, in which the linguistic data merely activates liguistic \n"
    "knowledge stored in the mind. Since I am going to focus on Chomsky's theory \n"
    "more exhaustively later on, I will not elaborate on this issue at this point. \n"
    "\n"
    "Nowadays it is hardly possible to adopt any of these two options directly: \n"
    "either extreme behaviourism or radical innatism (Cunningham 1972). \n"
    "Psychological research has recently progressed in the direction of regarding the \n"
    "human being as a mixture of genetically determined capacities and knowledge \n"
    "gained by experience. \n"
    "\n"
    "Thus, the behaviourist view of language acquisition has been criticised \n"
    "sharply on the grounds of its oversimplicity. It ignores completely the inborn \n"
    "aspect of human knowledge, that is, the existence of congenital potential which \n"
    "makes learning possible. According to the stimulus-response theory, the \n"
    "children's activity is limited to a passive reception of the stimuli coming from the \n"
    "environment. They do not make any conscious effort to organise the \n"
    "accumulating experience, since they are equipped with no mechanism warranting \n"
    "this process. Behaviourist theory ignores completely the creativity of human \n"
    "beings, making children rather passive viewers than actors in the process of \n"
    "language acquisition. Thus, they are deprived of the possibility to shape their \n"
    "language behaviour in a conscious way. However, it has been proved that passive \n"
    "exposure to language does not result in mastering it. To illustrate this point, \n"
    "Snow (1977) shows that Dutch children, who watch German television many \n"
    "hours a day, do not end up with an active knowledge of German. \n"
    "\n"
    "By the same token, it is not possible to adhere to radical innatism, either. The \n"
    "innatist approach to language acquisition also discards the assumption that \n"
    "children are endowed with the capacity to construct step by step their linguistic \n"
    "\n"
    "\n"
    "\n"
    " \n"
    "47\n"
    "\n"
    "reality, as emphasized by Piaget, but on thoroughly different grounds. It stresses \n"
    "the importance of heredity, but in the form of the innately determined LAD, \n"
    "which, however, does not take part in an active construction of knowledge. \n"
    "Innatists claim that all knowledge, including linguistic knowledge is preformed, \n"
    "that is, its structure and content are genetically determined. \n"
    "\n"
    "As far as Jean Piaget is concerned, in his work from 1948, he expresses the \n"
    "opinion that cognitive and linguistic development depends heavily both on \n"
    "environmental and hereditary factors (Piaget 1948). He formulates the \n"
    "constructionist theory of first language acquisition on the basis of diary studies \n"
    "of his children: Jacqueline, Lucienne and Laurent. According to him, stages, that \n"
    "is, particular phases in cognitive and linguistic growth, are actively constructed \n"
    "by the child who builds upon earlier structures through interaction with the \n"
    "environment. \n"
    "\n"
    "According to Piaget, a pattern of human behaviour consists of functions and \n"
    "structures. As the child develops, functions remain invariant, but structures \n"
    "change in a systematic and predictable order. This change in structures is the \n"
    "essence of development. The term structure refers to properties of an event, both \n"
    "internal (that is, its mental representation in the mind) and external (that is, its \n"
    "observable properties and features). The structural units are called in Piaget's \n"
    "system schemata (Piaget 1948). Schemata form a kind of network that acts as a \n"
    "receiver of incoming data and is continually changing its shape in order to better \n"
    "assimilate those data. Overt behaviour is presuambly organized by them as well. \n"
    "Function, however, refers to biologically inherited modes of interacting with the \n"
    "environment. Piaget has distinguished between two basic functions: organization \n"
    "(that is every act is organized) and adaptation (that is the dynamic aspect of \n"
    "organization), as pointed out by Piaget (1952). As far as adaptation is concerned, \n"
    "Piaget further subdivided it into assimilation and accommodation. According to \n"
    "him, they are responsible for advancement from one cognitive stage to another. \n"
    "\n"
    "Assimilation is the process of applying old schemas to new objects and \n"
    "events. Let us imagine that the child has the three schemas of grasping, biting \n"
    "and shaking and it is confronted with a new object, for example, a stuffed doll. It \n"
    "will try to understand this object by making use of its old schemas, which means \n"
    "that it will grasp, bite and shake the stuffed doll. \n"
    "\n"
    "Accommodation consists in modifying some elements of an old schema or \n"
    "learning a new schema which is more appropriate for a new object or event. For \n"
    "example, the crying schema can be modified by changing the pitch or intensity, \n"
    "depending on the kind of the need to be expressed. \n"
    "\n"
    "Accommodation and assimilation are called functional invariants because \n"
    "they are charasteristic of all biological systems. However, they are not always in \n"
    "balance with one another. Advances in cognitive development become greater \n"
    "when accommodation plays a larger role than assimilation since then the range \n"
    "of the child's behaviour expands because the child learns the new schemas that \n"
    "\n"
    "\n"
    "\n"
    " \n"
    "48\n"
    "\n"
    "will be appropriate for a new situation. The more such instances, the better for \n"
    "the child, because then its repertoire of behaviour expands. \n"
    "\n"
    "Development, then, is perceived by Piaget as a continuous interaction of \n"
    "assimilation with accommodation, which finally leads to the third process of \n"
    "organization, by which Piaget means the ability of the child to organize and \n"
    "construct reality. At this point it is necessary to present Piaget's idea of cognitive \n"
    "development, which he conceives in terms of the units called periods which can \n"
    "further be subdivided into the above mentioned stages (Piaget 1948). As far as \n"
    "the foundations for speech production and perception are concerned it is during \n"
    "the Sensorimotor Period that they are established. Piaget calls it sensorimotor \n"
    "because the child solves problems by means of its sensory systems and motor \n"
    "activity. This term also implies that the child derives understanding of the world \n"
    "from its actions, solely from what it senses and does. Since at this time abstract \n"
    "thinking is nonexistent, the child does not analyse problems, plan strategies and \n"
    "wonder what their consequences will be. Let us now concentrate on the \n"
    "description of the six sensorimotor stages (Piaget 1948) in order to trace \n"
    "cognitive development of the child, which is interlinked with the acquisition of \n"
    "its native language. \n"
    "\n"
    "Stage 1: Exercising the ready-made sensorimotor schemata (0–1 months). \n"
    "The infant comes to the world with a number of sensorimotor mechanisms, \n"
    "\n"
    "called reflexes which are involuntary responses to specific stimuli. One of them \n"
    "is the rooting reflex, i.e. the baby's tendency to turn the head towards any object \n"
    "that gently touches its cheek. The newborn has also got the grasping reflex (i.e. it \n"
    "grasps any object placed in its palm), the sucking reflex, the crying reflex and \n"
    "many others. In terms of early speech production, these are vegetative sounds \n"
    "associated with the process of eating. Reflexes are a means of communication \n"
    "with the outside world. In the beginning, they are very crude, but in the course of \n"
    "development they become more efficient and voluntary. For example, the \n"
    "sucking reflex undergoes the modification that enables the infant to reject \n"
    "substances other than food. At the same time it becomes capable of sucking more \n"
    "quickly and vigorously. \n"
    "\n"
    "Stage 2: Primary Circular Reactions (1–4 months). \n"
    "The second stage is based upon the previous developmental stage, as \n"
    "\n"
    "observed by Piaget (1948). The term circular refers to the circularity or repetitive \n"
    "aspects of behaviour. On the other hand, the adjective primary indicates that the \n"
    "infant's activities such as thumb sucking, babbling or shaking its arms involve \n"
    "only its own body. In all these instances the goal of the activity is the same as the \n"
    "means to achieve it. For example, the infant's shaking its arms serves no other \n"
    "aim than performing the activity itself. \n"
    "\n"
    "At this stage there can be observed the assimilation of more and more \n"
    "stimulus patterns and subsequent coordination of various schemata, e.g. hearing \n"
    "and looking at the object at the same time. The behaviour of the child begins to \n"
    "\n"
    "\n"
    "\n"
    " \n"
    "49\n"
    "\n"
    "be centered on objects but its world consists solely of sensorimotor events, \n"
    "components of its own functioning. The infant will tend to repeat some \n"
    "behaviour if by accident it leads to interesting events. As observed by Piaget: \n"
    "Sometimes, for example, the wail which precedes or prolongs the crying is kept \n"
    "for its own sake because it is an interesting sound (Piaget 1952:219). It is \n"
    "emphasized that it is then that the first circular reactions related to phonation \n"
    "may definitely be observed (Piaget 1952:220). \n"
    "\n"
    "To sum up, during the second stage the beginning of intergration of inherited \n"
    "patterns of behaviour into habits and perceptions can be observed. This stage is \n"
    "often called the stage of reproductive assimilations, whereby there occur \n"
    "reproductions or repetitions of newly acquired structures. \n"
    "\n"
    "Stage 3: Secondary Circular Reactions (4–8 months). \n"
    "In the third stage reactions are still circular (that is, the infant is involved in \n"
    "\n"
    "repeating an interesting event), but also secondary in the sense that the action of \n"
    "the infant's body is used as a means that brings about the results other than the \n"
    "activity itself. For example, when the infant shakes its arms, it may cause the \n"
    "mobile attached to the crib to move. The child then repeats this activity to see the \n"
    "outside event again. During stage 3 the initial separation of means and ends can \n"
    "be observed. This is just the beginning of the whole process, because the relation \n"
    "of means to ends is fortuitous. It is only after an interesting event has occurred \n"
    "that the infant desires it. Moreover, its behaviour is aimed solely at repeating the \n"
    "prior events, the baby is not yet inventing any new behaviour. \n"
    "\n"
    "The infant does not have concepts at this time but only its sensorimotor \n"
    "schemata. Whereas primary circular reactions help to establish primary schemata, \n"
    "such as: grasping, shaking and vocalizing, secondary circular reactions lead to \n"
    "secondary schemata that are sensorimotor impressions of the particular features of \n"
    "objects. They are antecedents to later classes of concepts. According to Piaget \n"
    "(1952:234): the secondary schemata constitute the first outline of what will \n"
    "become classes or concepts of reflective intelligence: perceiving an object as \n"
    "something to shake, to rub, etc. These are the first foundations of the later \n"
    "classification of objects and their characteristic features, which will be responsible \n"
    "for the establishing of the notion of the parts of speech. \n"
    "\n"
    "Stage 4: Coordination of Secondary Schemata (8–12 months). \n"
    "In stage 4 we can observe further separation of means from ends. During \n"
    "\n"
    "stage 3 the infant lost interest in action, if the obstacle was interposed between it \n"
    "and an object. That was only the beginning of intention. In stage 4, however, \n"
    "much more clear separation of means from ends can be observed. Suppose, for \n"
    "example, that the child wants to grasp the box the parent is holding behind the \n"
    "cushion. It is not discouraged by the obstacle and uses some of its schemata to \n"
    "obtain the goal. It strikes the cushion and depresses it with one hand while \n"
    "grasping the object with the other. The important point to make here is that motor \n"
    "meaning is replaced by symbolic meaning. In stage three, the child recognized an \n"
    "\n"
    "\n"
    "\n"
    " \n"
    "50\n"
    "\n"
    "object by performing an action that had previously occurred in its presence. This \n"
    "action was the ‘meaning’ of that object for the child. Now the actions begin to \n"
    "have their underlying mental representations.2 \n"
    "\n"
    "Stage 4 is also characterised by what Piaget has called object permanence. \n"
    "Object permanence is defined as the knowledge that objects continue to exist \n"
    "even when they are no longer seen. The child begins to search actively for \n"
    "objects that the adults hide somewhere. This development of object permanence \n"
    "provides the evidence that the infant's reliance on symbols becomes more and \n"
    "more advanced and gradually replaces its sensorimotor schemata. Finally, \n"
    "imitative skills are also improving. During stage 3 the imitation was limited to \n"
    "the sounds that the infant has already produced. In stage 4 attempts at novel \n"
    "sounds appear for the first time. This will appear to be essential for language \n"
    "acquisition. \n"
    "\n"
    "Stage 5: Tertiary Circular Reactions (12–18 months). \n"
    "At this stage the infant has progressed to the point of actively seeking \n"
    "\n"
    "novelty. It deliberately manipulates the environment and produces some \n"
    "interesting spectacles (see Philips (1969)). This time the spectacle is separate \n"
    "from the overt action and even after it has occurred the child does not cease to \n"
    "vary its movements. One can think of the stage 5 baby as the first ‘scientist’. \n"
    "Thus, in stage 5 the infant starts to discover new ways not used before. \n"
    "Ineffective means drop out and gradually the performance becomes deliberate \n"
    "and efficient. \n"
    "\n"
    "The fifth stage is characterised by the formation of new schemata which, as \n"
    "Piaget (1952:305) puts it, are no longer due to simple reproduction of fortuitous \n"
    "results but to a sort of experimentation or search for novelty as such. The child \n"
    "begins to use objects as means, e.g. a stick to move or to get an object. This \n"
    "achievement coincides with the first purposeful vocalisations, that is, language \n"
    "becomes one of the means the child can use to obtain certain ends, either through \n"
    "comprehension or production. These vocalisations are not yet words, because \n"
    "they do not pair vocalisations with concepts. Instead they are symbols, they \n"
    "indicate the relationship that exists between objects or events and sounds. \n"
    "\n"
    "Another achievement of stage 5 is that the object is now being represented \n"
    "by internal symbols. However, at this point the child still cannot cope with \n"
    "invisible displacements, that is, it has to see the objects actually disappear. The \n"
    "child's interest in seeking objects is considered to be the neccesary condition for \n"
    "the establishment of a repository of conceptual schemas. The first step towards \n"
    "\n"
    " \n"
    "2 As an example supporting this statement let us quote Piaget's Observation 133 (Piaget 1952: \n"
    "\n"
    "249): At the age of nine months, sixteen days, Jacqueline likes the grape juice in a glass but not the \n"
    "soup in the bowl. She watches her mother's activity. When the spoon comes out of the glass she \n"
    "opens her mouth wide, whereas when the spoon comes out of the bowl her mouth remains close. \n"
    "Her mother tries to lead her to make a mistake by taking the spoon from the bowl and passing it by \n"
    "the glass before offering it to Jacqueline but she is not fooled. \n"
    "\n"
    "\n"
    "\n"
    " \n"
    "51\n"
    "\n"
    "this stage would be the creation of mental representations of objects in the child's \n"
    "mind. \n"
    "\n"
    "Stage 6: Beginning of Representational Thought (18–24 months). \n"
    "Piaget states that at this stage the child starts to think before acting. Stage 6 \n"
    "\n"
    "clearly marks the onset of representational thinking, which involves mental \n"
    "reasoning that is prior to acting. Another thing is that the child's conception of \n"
    "space is characterised by the increasing importance of the internalised symbols, \n"
    "e.g. the displacements of objects need no longer to be visible. The concept of \n"
    "time has undergone the same transformation: remembering past events and \n"
    "anticipating future ones is possible because of the existence of internal symbols. \n"
    "However, Piaget (1948) does not believe that the sensorimotor thought involves \n"
    "language. Language is too rudimentary at this stage and the word meanings are \n"
    "unstable and idiosyncratic. The child uses one word to label many objects, \n"
    "changing the meaning to suit its own purposes. Furthermore, all these changes \n"
    "can take place just within hours or even minutes.3 \n"
    "\n"
    "Summing up, during the sensorimotor period the child's sensorimotor \n"
    "schemata turn gradually into symbolic thinking. Undoubtedly, any two-year-old \n"
    "child's thinking cannot be compared with that of an adult and his mental \n"
    "capacity. However, there is no denying that it is the first step towards cognitive \n"
    "and linguistic maturity that is constructed by degrees on the basis of the child’s \n"
    "congenital potential. \n"
    "\n"
    "On the other hand, Piaget's theory was strongly criticised by those who \n"
    "conceived language development as independent of other cognitive capacities. \n"
    "For example, Chomsky (1959) proposed the existence of genetically \n"
    "transmitted LAD, independent of other cognitive capacities and responsible for \n"
    "language acquisition. In his work Review of Verbal Behaviour he appears as an \n"
    "adherent of strong innatism, i.e. the theory stating that not only structures and \n"
    "mechanisms enabling language acquisition (as was claimed by Piaget) are \n"
    "predetermined, but also knowledge itself in the form of linguistic universals. \n"
    "\n"
    "Chomsky (1975) can see no reason why intellectual development should be \n"
    "separated from the physical one. In his view, if the physical structure of the \n"
    "organism is genetically determined and is taken for granted with such \n"
    "dimensions as size and rate of development being also inherited, why should we \n"
    "take a different approach to mental development? \n"
    "\n"
    "Thus, in his view every human being possesses a biologically determined \n"
    "basis for gaining knowledge, which is a prerequisite to learning. This innate \n"
    "\n"
    " \n"
    "3 Observing Polish children at this age I have noticed that the same syllable ‘ko’ or syllables \n"
    "\n"
    "’ko-ko’ (reduplication) may serve to represent a few objects, for example: kokardka ‘bow’, koniec \n"
    "‘end’, kopać ‘kick’, kolczyk ‘earring’, okulary ‘glasses’. Another instance would be the syllable \n"
    "‘dzie’ standing for: do widzenia ‘goodbye’, dzień dobry ‘good morning’ and dziękuję ‘thank you’. \n"
    "Piaget (1952:157) comments that: these first verbal schemas are intermediary between the schemas \n"
    "of sensorimotor intelligence and conceptual schemas.  \n"
    "\n"
    "\n"
    "\n"
    " \n"
    "52\n"
    "\n"
    "capacity to attain knowledge enables an amazingly quick acquisition of language \n"
    "by the child within a relatively short period of time and without any specific \n"
    "training. Furthermore, Chomsky (1971) proposes that the general form of a \n"
    "system of knowledge is fixed in advance as predisposition of the mind. He \n"
    "presents abundant evidence to support the view that the form of language (not \n"
    "only special abilities that enable its acquisition) is innate. The first argument in \n"
    "favour of this statement, is that children are able to learn language in a relatively \n"
    "short period of time on the basis of fragmentary and impoverished input. \n"
    "Chomsky claims that children can learn any language irrespective of their \n"
    "parents' nationality if only they are provided with sufficient exposure to it thanks \n"
    "to the existence of the so called language universals. \n"
    "\n"
    "Chomsky believes that there exist certain phonological, syntactic and \n"
    "semantic units of speech that are universal in the sense that although they do not \n"
    "occur in all languages of the world they may occur in any of them. Let us \n"
    "consider, for example, some distinctive phonological features. One of them is \n"
    "[voicing] that differentiates p from b in the pronounciation of such words as pin \n"
    "and bin, or [nasality] that makes the difference between b and m in bad and mad. \n"
    "Certainly, not all of these features are found in every language but each language \n"
    "selects its own set out of them. These are substantive universals. In addition to \n"
    "that, Chomsky also talks about formal universals which determine the form and \n"
    "the manner of operation of grammatical rules. \n"
    "\n"
    "We may pose the following question: how does the child make use of the \n"
    "universals and how does it know which of them to choose in order to be able to \n"
    "speak a particular language? To account for this phenomenon, Chomsky (1971) \n"
    "formulates the so called innateness hypothesis which holds that we are endowed \n"
    "with a faculty that chooses specific universals necessary for the construction of \n"
    "the grammar of a particular language. This faculty is referred to as the language \n"
    "acquisition device (LAD). \n"
    "\n"
    "In order to construct the grammar of a language, LAD searches through a \n"
    "range of possible hypotheses about language structure and then selects those \n"
    "grammars that are compatible with the primary linguisic data at hand. To accept \n"
    "this theory we must assume that the child has already mastered a technique of \n"
    "representing structural information about input signals. LAD would then select \n"
    "one of the potential grammars and will thus construct the grammar of the \n"
    "language. Children who acquire language in this way know, of course, a lot more \n"
    "than they 'have learned'. Their knowledge goes far beyond the presented primary \n"
    "linguistic data and in no sense is it an inductive generalisation from this data but \n"
    "rather it emerges due to the activation by some relevant experience. \n"
    "\n"
    "The grammar constructed by LAD is referred to as generative grammar. This \n"
    "grammar which has an internal representation in the mind is a system that \n"
    "determines the phonetic, syntactic and semantic properties of an infinite number \n"
    "\n"
    "\n"
    "\n"
    " \n"
    "53\n"
    "\n"
    "of sentences. Thus, the child knows the language constructed by the grammar it \n"
    "has just learnt. \n"
    "\n"
    "According to Chomsky, the grammar of the language should generate 'all \n"
    "and only' sentences of the language, that is, all the sentences of the language, but \n"
    "only the grammatical ones. Chomsky maintains that generative grammar \n"
    "'projects' any set of sentences upon infinite set of sentences that constitute a \n"
    "given language. This property of the grammar demonstrates the creative property \n"
    "of the human language. The creative aspect of language is, according to \n"
    "Chomsky, unconscious and unreflecting. \n"
    "\n"
    "This issue poses an especially challenging problem to the theory of language \n"
    "acquisition. It is one of the strongest arguments against behaviourism. Chomsky \n"
    "regards it as a good reason to believe that the reinforcement theory is not able to \n"
    "explain all facts of language behaviour because this theory is totally incapable of \n"
    "accounting for the fact that by the age of five or six children are able to produce \n"
    "and understand a large number of sentences that they have not previously heard, \n"
    "however succesful it might be in explaining how certain habits and associations \n"
    "are built up. Behaviourist theories of language acquisition cannot bridge the gap \n"
    "between the utterances that the child hears (often full of errors and distortions) \n"
    "and its ability to construct (on the basis of this impoverished and imperfect data) \n"
    "the grammatical rules governing the structure of language. \n"
    "\n"
    "Chomsky claims that it is the child's inborn knowledge of the universals that \n"
    "makes up for the deficiency in the behaviourist theory of language acquisition. \n"
    "However, he does not discard the model of stimulus and response completely. \n"
    "According to him, behaviourist tradition is capable of explaining some of the \n"
    "facts of language behaviour, especially those pertaining to objects in the child’s \n"
    "environment and also certain utterances that occur in its early life. Piaget also \n"
    "argues that behaviourist theory in its pure form is not a reliable explanation of \n"
    "first language acquisition, but his antiempiricism works within a completely \n"
    "different framework. He proposes the existence of an inborn ability to process \n"
    "linguistic data and on the basis of that to construct the grammar of a given \n"
    "language. \n"
    "\n"
    "Piaget is convinced that the fundamental relationship that constitutes all \n"
    "knowledge is not a mere association between objects, for this notion neglects the \n"
    "active role of the subject, but rather the assimilation of objects to the schemes of \n"
    "that subject (Piattelli-Palmarini 1980:350). According to Piaget (1951), the \n"
    "process of immersion in an environment and interaction with it can be regarded \n"
    "as the process by which the developing organism will assimilate fragments of the \n"
    "environmental structure while accommodating its own schemas in the process of \n"
    "assimilation. \n"
    "\n"
    "On the other hand, Chomsky claims that since the structure of language is \n"
    "predetermined, the acquisition consists merely in the unfolding of inborn \n"
    "predispositions. This process can be compared to a maturational growth of a \n"
    "\n"
    "\n"
    "\n"
    " \n"
    "54\n"
    "\n"
    "biological organism. In Chomsky's view language acquisition proceeds \n"
    "according to a genetic programme imprinted on the mind. The acceptance of this \n"
    "assumption is tantamount to the dismissal of the possibility of development, \n"
    "which was of primary magnitude for Piaget. \n"
    "\n"
    "By the same token, the role of the environment was reduced to a mechanism \n"
    "merely initiating the process of language acquisition. Data coming from the \n"
    "environment serve just to reconstruct, not construct, grammar of a given \n"
    "language. However, Chomsky claims that these linguistic rules are unfolded \n"
    "unconsciously while children are exposed to the samples of language and when \n"
    "they attempt to communicate. Therefore, it is maintained that the innatist theory \n"
    "refutes the claim about the close relationship of language and thought, viewing \n"
    "them as not mutually related entities. Moreover, it neglects the role of experience \n"
    "in language acquisition. \n"
    "\n"
    "On the contrary, Piaget strongly advocates the interrelation of language and \n"
    "cognitive processes. He insists on priority of thought over language, the former \n"
    "warranting the acquisition of the latter. In Piaget's view the establishment of the \n"
    "system of symbolic representation is a prerequisite for the emergence of the \n"
    "sound system. However, it is not an instantenous process, as in the case of \n"
    "Chomsky's model. Instead, it is believed to consist of stages actively constructed \n"
    "by the child, whose system accommodates to the new situations and objects and \n"
    "assimilates these objects and situations into its own structures. For this reason, \n"
    "this approach is referred to as constructivist because it is the child who, having \n"
    "inherited cognitive capacities, creates mental representation of the world in its \n"
    "mind and consequently masters its native language. \n"
    "\n"
    "References \n"
    "\n"
    "Chomsky, N. 1971. Selected Readings. Oxford: Oxford University Press. \n"
    "Chomsky, N. 1975. Reflections on Language. New York: Pantheon Books. \n"
    "Chomsky, N. 1959. ”Review of Verbal Behaviour, by B.F. Skinner”. Language 35. 26–58. \n"
    "Cunningham, M. 1972. ”Intelligence: Its organisation and development”. New York: Academic \n"
    "\n"
    "Press. \n"
    "Lewandowska-Tomaszczyk, B. 1988. Ways to Language. Łódź: Uniwersytet Łódzki. \n"
    "Phillips, J.L. 1969. The Origins of Intellect: Piaget’s Theory. San Francisco: W.H. Freeman and \n"
    "\n"
    "Company. \n"
    "Piaget, J. 1948. The Construction of Reality in the Child. New York: Basic Books Inc. \n"
    "Piaget, J. 1952. The Origins of Intelligence in Children. New York: International Universities \n"
    "\n"
    "Press. \n"
    "Piattelli-Palmarini, M.P. 1980. The Debate between Jean Piaget and Noam Chomsky. London \n"
    "\n"
    "and Henley: Routledge & Kegan Paul. \n"
    "Ferguson, Snow, ed. 1977. \"Mother speech research: From input to interaction\". Talking to \n"
    "\n"
    "Children: Language Input Acquisition. Cambridge: Cambridge University Press. \n"
    " \n"
    "\n"
    "\n"
    "		2012-08-09T11:48:08+0200\n"
    "	Preflight Ticket Signature\n"
    "\n"
    "\n"
    "\n"
    "\n"
    "\n"
)


# =========================================================
# Pipeline cache
# =========================================================

PIPELINES = {}

def get_classifier(model_display_name):
    model_name = MODELS[model_display_name]

    if model_name not in PIPELINES:
        PIPELINES[model_name] = pipeline(
            "text-classification",
            model=model_name,
            tokenizer=model_name,
            return_all_scores=True
        )

    return PIPELINES[model_name]

# =========================================================
# Safe prediction (IMPORTANT FIX)
# =========================================================

def get_best_prediction(classifier, text):

    # HARD SAFETY: prevents tokenizer overflow in pipeline
    text = text[:2000]

    result = classifier(text)

    if isinstance(result, list) and isinstance(result[0], list):
        result = result[0]

    return max(result, key=lambda x: x["score"]), result

# =========================================================
# SAFE CHUNKING (ROBUST FIX)
# =========================================================

def safe_chunk_text(text, tokenizer, max_tokens=480):
    """
    True safe chunking for XLM-R (leaves room for special tokens)
    """

    sentences = text.split("\n")

    chunks = []
    current = []
    current_len = 0

    for sent in sentences:

        sent_tokens = tokenizer.encode(sent, add_special_tokens=False)
        sent_len = len(sent_tokens)

        # Case 1: single sentence too large → hard split
        if sent_len > max_tokens:
            for i in range(0, sent_len, max_tokens):
                part = tokenizer.decode(sent_tokens[i:i + max_tokens])
                chunks.append(part)
            continue

        # Case 2: overflow chunk
        if current_len + sent_len > max_tokens:
            chunks.append("\n".join(current))
            current = [sent]
            current_len = sent_len
        else:
            current.append(sent)
            current_len += sent_len

    if current:
        chunks.append("\n".join(current))

    return chunks

# =========================================================
# Empty line cleanup
# =========================================================

def normalize_empty_lines(lines):

    cleaned = []
    prev_empty = False

    for l in lines:
        empty = not l.strip()

        if empty and prev_empty:
            continue

        cleaned.append(l)
        prev_empty = empty

    return cleaned

# =========================================================
# CORE PIPELINE (FIXED)
# =========================================================

def process_text_input(text, noise_labels, selected_model):

    if not text.strip():
        return "", "", "", None

    classifier = get_classifier(selected_model)
    tokenizer = classifier.tokenizer

    chunks = safe_chunk_text(text, tokenizer)

    logs = []
    kept = []
    removed = []

    line_counter = 0

    for c_id, chunk in enumerate(chunks):

        lines = chunk.splitlines()

        for line in lines:

            line_counter += 1

            if not line.strip():
                kept.append("")
                continue

            pred, _ = get_best_prediction(classifier, line)

            logs.append(
                f"[Chunk {c_id}] Line {line_counter} | "
                f"{pred['label']} ({pred['score']:.4f})\n{line}\n"
            )

            if pred["label"] in noise_labels:
                removed.append(line)
            else:
                kept.append(line)

    kept = normalize_empty_lines(kept)
    filtered = "\n".join(kept)

    tmp = tempfile.NamedTemporaryFile(
        delete=False,
        suffix=".txt",
        mode="w",
        encoding="utf-8"
    )

    tmp.write(filtered)
    tmp.close()

    stats = (
        f"Chunks: {len(chunks)}\n"
        f"Total lines: {line_counter}\n"
        f"Removed: {len(removed)}\n"
        f"Remaining: {len(kept)}"
    )

    return "\n".join(logs), filtered, stats, tmp.name

# =========================================================
# TXT FILE
# =========================================================

def process_document_file(file, noise_labels, selected_model):

    if file is None:
        return "", "", "", None

    with open(file.name, "r", encoding="utf-8") as f:
        text = f.read()

    return process_text_input(text, noise_labels, selected_model)

# =========================================================
# PDF EXTRACTION (DIGITAL + OCR)
# =========================================================

def extract_text_from_pdf(pdf_file):

    text_parts = []

    # 1. try digital extraction
    try:
        reader = PdfReader(pdf_file.name)

        for page in reader.pages:
            t = page.extract_text()
            if t:
                text_parts.append(t)

    except:
        pass

    text = "\n".join(text_parts).strip()

    # 2. OCR fallback
    if not text:

        pages = convert_from_path(pdf_file.name, dpi=300)

        ocr = []
        for page in pages:
            ocr.append(pytesseract.image_to_string(page))

        text = "\n".join(ocr)

    return text

# =========================================================
# PDF PIPELINE
# =========================================================

def process_pdf_file(file, noise_labels, selected_model):

    if file is None:
        return "", "", "", None

    text = extract_text_from_pdf(file)

    return process_text_input(text, noise_labels, selected_model)

# =========================================================
# UI
# =========================================================

with gr.Blocks(title="Stable Academic Text Filter") as demo:

    gr.Markdown("""
    # Academic Text Noise Filter

    ✔ Filter text content types
    ✔ OCR + PDF support  
    ✔ Safe chunking (XLM-R compatible)  
    ✔ Robust long-document handling
    """)

    with gr.Tab("Text"):
        m = gr.Dropdown(list(MODELS.keys()), value=DEFAULT_MODEL)
        t = gr.Textbox(lines=15,
                       label="Input Text",
                       value=EXAMPLE_TEXT,)
        n = gr.CheckboxGroup(LABELS, label="Remove content type", value=DEFAULT_NOISE)

        btn = gr.Button("Process")

        o1 = gr.Textbox(lines=15,
                        label="Classification Result",)
        o2 = gr.Textbox(lines=15,
                        label="Modified Text",)
        o3 = gr.Textbox(lines=15,
                        label="Details",)
        o4 = gr.File()

        btn.click(process_text_input, [t, n, m], [o1, o2, o3, o4])

    with gr.Tab("TXT"):
        m = gr.Dropdown(list(MODELS.keys()), value=DEFAULT_MODEL)
        f = gr.File(file_types=[".txt"])
        n = gr.CheckboxGroup(LABELS, label="Remove content type", value=DEFAULT_NOISE)

        btn = gr.Button("Process")

        o1 = gr.Textbox(lines=15,
                        label="Classification Result",)
        o2 = gr.Textbox(lines=15,
                        label="Modified Text",)
        o3 = gr.Textbox(lines=15,
                        label="Details",)
        o4 = gr.File()

        btn.click(process_document_file, [f, n, m], [o1, o2, o3, o4])

    with gr.Tab("PDF"):
        m = gr.Dropdown(list(MODELS.keys()), value=DEFAULT_MODEL)
        f = gr.File(file_types=[".pdf"])
        n = gr.CheckboxGroup(LABELS, label="Remove content type", value=DEFAULT_NOISE)

        btn = gr.Button("Process PDF")

        o1 = gr.Textbox(lines=15)
        o2 = gr.Textbox(lines=15)
        o3 = gr.Textbox()
        o4 = gr.File()

        btn.click(process_pdf_file, [f, n, m], [o1, o2, o3, o4])

if __name__ == "__main__":
    demo.launch()