fine-tuned mpnet-energy 1 epochs

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+    "word_embedding_dimension": 768,
+    "pooling_mode_cls_token": true,
+    "pooling_mode_mean_tokens": false,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false,
+    "pooling_mode_weightedmean_tokens": false,
+    "pooling_mode_lasttoken": false,
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,863 @@
+---
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- dense
+- generated_from_trainer
+- dataset_size:53913
+- loss:MultipleNegativesRankingLoss
+base_model: Alibaba-NLP/gte-multilingual-base
+widget:
+- source_sentence: How does the monitoring system for well integrity function after
+    CO2 injection?
+  sentences:
+  - 'Drilling is a complex process and delivering a successful well requires identifying
+    proper technologies and utilizing them efficiently to save time & cost. Today
+    in Oil & Gas industry there is a huge focus on digital technologies to improve
+    Drilling Process efficiency and PDO decided to implement an innovative approach
+    of process optimization by implementing a unique project "electronically Delivering
+    the Limit (eDtL)".
+
+    The overall approach with eDtL project was to implement a platform which can provide
+    Drilling Operations team the technical limit for all Drilling Activities, which
+    is the theoretical minimum time required to perform an activity, based on available
+    knowledge and technology.
+
+    eDtL system utilizes rig sensors data transmitted in Real-Time from Drilling Rigs
+    to automatically detect the Rig Activity and focus on identifying the areas of
+    Drilling Performance Improvements and minimizing redundant tasks for rig and office
+    teams. The identified opportunities are communicated with rig team for implementation
+    and the performance is tracked again to highlight the improvements.
+
+    eDtL system also provides capability for continuous improvement of organizational
+    processes by introducing automation of redundant tasks. One of such improvement
+    was partial automation of Daily Drilling Report which was historically manually
+    recorded by rig team daily.'
+  - 'ADNOC has embarked on a major Carbon Capture and Storage (CCS) project where
+    large quantities of CO2 are injected into deep saline aquifers for permanent storage
+    instead of releasing into the atmosphere.
+
+    An advanced chemical tracer technology was deployed in the first CCS project in
+    the UAE for continuous CO2 monitoring to ensure permanent and safe CO2 storage.
+    In case of containment breach, the chemical tracer technology can confirm the
+    leakage and identify its source.
+
+    After CO2 injection for permanent storage, any containment breaching would be
+    detected in the shallow soil monitoring borehole. Few soil monitoring boreholes
+    were excavated across the field in which Capillary Adsorption Tubes (CAT) were
+    inserted for some time and replaced by another according to the sampling frequency
+    plan. The tube is sent to the lab for CO2 leak detection and reporting. The high
+    detection resolution is in the order of 0.1 parts per trillion (ppt). This has
+    a positive impact on the system economics because smaller quantities of chemical
+    tracer material are required.
+
+    The tracer injection monitoring system is ongoing in the first CO2 storage area
+    of Abu Dhabi. The monitoring includes soil monitoring which are shallow boreholes.
+    The soil monitoring boreholes were excavated close to the CO2 injection well to
+    ensure that there are no well integrity issues developed due to thermal effects
+    by CO2 injection. The soil monitoring boreholes to be verified by surface gas
+    CO2 monitors. Soil monitors were located around the radial storage area, to detect
+    CO2 leakage and to understand CO2 migration to the soil through the cap rock (in
+    case of leakage). The monitoring system for caprock and well integrity will provide:
+    Surface soil monitoring for cap rock integrity, integrity confirmation for legacy
+    wells, integrity confirmation of injection well in the post-injection monitoring
+    period, leakage quantification, leakage origin if multiple injectors. The monitoring
+    system can continue for up to 30 years of the operational period as well as the
+    full post-injection monitoring, measurement and verification horizon.
+
+    This paper presents a description of a sophisticated CO2 monitoring technology
+    that is being deployed in UAE''s first CCS project. CO2 tracer technology is considered
+    as one of the most accurate methods to detect CO2 leakage at surface. Its high-detection
+    resolution allows early leakage identification and early mitigation action. In
+    addition, it proves to be relatively low cost, operationally easy to execute,
+    and requires a small operational footprint.'
+  - 'Carbon Capture and Storage, as a solution to mitigate the increase in greenhouse
+    gases emissions in the atmosphere, is still bringing intensive worldwide R&D activities.
+    In particular, significant acceleration of in situ CCS experiments supports technical
+    developments as well as acceptability of this technology. Among the major risks
+    identified to this technology, wells are often considered to be the weakest spots
+    with respect to CO2 confinement in the geological reservoir. Therefore, long-term
+    well integrity performance assessment is one of the critical steps that must be
+    addressed before large scale CCS technology deployment is accepted as a safe solution
+    to reduce CO2 emissions.
+
+    A risk-based methodology associated with well integrity is proposed within CO2
+    geological storage. The main objectives of this approach are to identify and quantify
+    risks associated with CO2 leakages along wells over time (from tens to thousands
+    of years), to evaluate risks and to propose relevant actions to reduce unacceptable
+    risks. The methodological framework emphasized the use of the risk concept as
+    a relevant criterion to (i) evaluate the overall performance of well confinement
+    with respect to different stakes, (ii) include different levels of uncertainty
+    associated to the studied system, and (iii) provide a reliable decision making
+    support. For the quantification of risk, a coupled CO2 flow model (gas flow and
+    degradation processes) was used to identify possible leakage pathways along the
+    wellbore and quantify possible CO2 leakage towards sensitive targets (surface,
+    fresh water, any aquifers…) for different scenarios. This approach offers an operational
+    response to some of the challenges inherent to well integrity management over
+    well lifecycle.
+
+    This paper focuses on the application of the methodology to a synthetic case based
+    on an existing well. The practical outcomes and the added values will be presented:
+    (i) an objective and structured process, (ii) scenarios identification and quantification
+    of CO2 migration along the wellbore for each scenario, (iii) risk mapping, (iv)
+    and operational action plans for risk treatment of well integrity.'
+- source_sentence: How did the detailed pre-survey planning impact the success of
+    the offshore seismic acquisition campaign?
+  sentences:
+  - 'We showcase an innovative campaigning and business-focused approach to reservoir
+    monitoring of multiple fields using 4D (time-lapse) seismic. Benefits obtained
+    in terms of cost, speed and the quality of insights gained are discussed, in comparison
+    with a piecemeal approach. Challenges and lessons learned are described, with
+    a view to this approach becoming more widely adopted and allowing 4D monitoring
+    to be extended to smaller or more marginal fields.
+
+    An offshore seismic acquisition campaign was planned and successfully executed
+    for a sequence of four 4D monitor surveys for fields located within 250 km of
+    each other on the greater Northwest Shelf of Australia. The four monitors were
+    acquired in H1 2020 comprising (in this order): Pluto Gas Field M2 (second monitor),
+    Brunello Gas Field M1 (first monitor), Laverda Oil Field M1 and Cimatti Oil Field
+    M1.
+
+    Cost savings expected from campaigning were realised, despite three cyclones during
+    operations, with success largely attributed to detailed pre-survey planning. Also
+    important were the choice of vessel and planning for operational flexibility.
+    The baseline surveys were diverse and required careful planning to achieve repeatability
+    between vintages over each field, and to optimise the acquisition sequence – minimising
+    time required to reconfigure the streamer spreads between surveys. The Cimatti
+    baseline survey was acquired using a dual-vessel operation; modelling, combined
+    with now-standard steerable streamers, showed a single-vessel monitor survey was
+    feasible. These optimisations provided cost savings incremental to the principal
+    economy of sharing vessel mobilisation costs across the whole campaign.
+
+    Both processing and evaluation (ongoing at the time of writing) are essentially
+    separate per field, but follow a consistent approach. Processing is carried out
+    by more than one contractor to debottleneck this phase, with products, including
+    intermediate quality control (QC) volumes, delivered as pre-stack depth migrations.
+    While full evaluation of the monitor surveys to static and dynamic reservoir model
+    updates will continue beyond 2020, key initial reservoir insights are expected
+    to emerge within days of processing completion, with some even earlier from QC
+    volumes. Furthermore, concurrent 4D evaluations are expected to result in fruitful
+    exchanges of ideas and technologies between fields.'
+  - 'Advances in seismic acquisition, processing, computing hardware and theory continue
+    to enhance seismic-image quality. However, an investment decision on seismic projects
+    should be based not only technical criteria but a quantifiable expected value
+    above all currently available field data including well information. This presentation
+    will include a case history of a major carbonate oil field demonstrating how this
+    value was estimated before a major reprocessing project and how this value is
+    being achieved.
+
+    This field contains over 1000 wellbore penetrations. A 3D seismic survey was acquired
+    over the field during 2001–2002, but the reservoir development team believed that
+    these data to date had added limited value. The motivation for evaluating the
+    potential for further investment in seismic data was a multi-billion-dollar field-redevelopment
+    plan.
+
+    The Value of Information (VOI) exercise to justify a seismic project began with
+    an evaluation of technical issues that limited the use of existing seismic data.
+    Through a targeted fast track reprocessing effort it was determined that the existing
+    survey had been designed and acquired adequately, and that the deficiencies in
+    the dataset at the reservoir level are primarily caused by near-surface and overburden
+    effects. The first-order impact is that mapped seismic surfaces exhibit a "roughness"
+    primarily from the overlying "non-geologic" noise. There was concern that many
+    subtle faults interpreted at the reservoir level could be "non-geologic" artifacts
+    which resulted in reluctance to incorporate these into the reservoir model. Amplitude
+    balancing issues in the original data precluded quantitative assessments such
+    as porosity prediction. The targeted reprocessing also verified that existing
+    algorithms and traditional workflows alone were insufficient to resolve the technical
+    issues.
+
+    Working with the reservoir development team the key business drivers for reprocessing
+    were identified as follows:
+
+    Increase individual well productivity and recovery
+
+    Image and define new opportunities in current poor-data areas
+
+    Save on well cost by preventing re-drills
+
+    Improve overall field development plan
+
+    Specific expected value metrics and risks were assigned to the above objectives
+    and a VOI assessment was completed. It was estimated that successfully achieving
+    the above business objectives would result in a potential value at least 15 times
+    the cost of the reprocessing. This resulted in management approval of the full
+    field reprocessing.
+
+    Following completion of the seismic reprocessing, the project team objectively
+    assessed whether the technical criteria had been achieved and if the business
+    criteria will be achieved. In both cases the team determined that value metrics
+    will be met. The reprocessing has impacted drill-wells as well as field development
+    planning. In addition, the reprocessed seismic data will produce additional potential
+    value as a result of opportunities not recognized at the start of the project.'
+  - 'Sabiriyah Mauddud is a giant reservoir in NK under active water flood with about
+    200 producers and 32 injectors. The reservoir has no aquifer or insignificant
+    energy support and had been on production since 1960s. Water flood started in
+    the year 1997, initially with a pilot and later on expanded to the full field
+    in a phased manner. Initial development was on pattern flood concept with all
+    vertical injectors & producers which has now been replaced with Produce High-Inject
+    Low (PHIL) concept using horizontal wells. In light of the significance of this
+    reservoir for Kuwait''s production, all efforts are made to optimize the performance
+    of this reservoir. To achieve this objective, Pressure monitoring & performance
+    analysis is considered to be the backbone of all production as well as injection
+    activities.
+
+    This paper presents the methodology conceived and implemented to assess the reservoir
+    pressure performance and estimate the current reservoir pressure in different
+    segments/ blocks in an innovative way so as to maximize the value of "Water flooding"
+    in North Kuwait area along with the meeting of production aspirations using ESP
+    as artificial lift system in an optimized manner. Except the RFT data in newly
+    drilled wells, the availability of pressure data was limited during recent past,
+    making it necessary to integrate all the available information so as to build
+    a powerful tool to be used for water flood monitoring. All available information
+    – Repeat Formation Tester "RFT", Static Bottom Hole Pressure "SBHP" and Pump intake
+    pressure "PIP" under dynamic and static conditions, were collected & analyzed.
+    An initial study related to compartmentalization showed two main areas, north
+    and south, based on comprehensive analysis of all the pressure points. The analysis
+    also helped for identifying areas with good vertical connectivity and understanding
+    segments with vertical barriers matching with the geological description. In order
+    to have the latest pressure mapping, data were combined to have an integrated
+    imagery of the pressure distribution across the reservoir. During the exercise,
+    "Gaps" were identified which were filled in by the intake pressure live data as
+    well as shut in data to have a meaningful mimic of the reservoir pressure to help
+    the ongoing production as well as injection activities.
+
+    Based on the innovative approach as above, surveillance plan has been made to
+    further enhance the quality of the mapping. Several maps such as opportunity map;
+    PVT properties map; layer wise pressure maps etc. have been generated for ready-to-use
+    information to facilitate daily operations.
+
+    The objective of the paper is to share the innovative, simple, smart and very
+    useful approach adopted by North Kuwait to manage the giant Mauddud. This paper
+    presents the methodology conceived and implemented to assess the reservoir pressure
+    performance and estimate the current reservoir pressure in difference segments/
+    blocks in an innovative way so as to maximize the value of "Water flooding" in
+    North Kuwait area along with the meeting the production aspirations using ESP
+    as artificial lift system in an optimized manner.'
+- source_sentence: What are the primary recovery techniques used in oil and gas extraction?
+  sentences:
+  - The extraction of oil and gas involves various techniques to enhance recovery
+    rates. Primary recovery relies on the natural pressure of the reservoir, while
+    secondary recovery techniques such as water flooding and gas injection are employed
+    to increase output after primary methods become inefficient. Tertiary recovery
+    methods, also known as enhanced oil recovery (EOR), use thermal, gas, or chemical
+    injection to further improve extraction rates. Each method comes with its own
+    cost implications and efficiency rates, which can significantly affect the overall
+    economics of an oilfield development project.
+  - 'In this paper one of the areas of conflicts observed with the performance of
+    horizontal wells standoff with respect to development of thin oil rim reservoirs
+    is examined.
+
+    In a technical paper as part of the critical review of literature on the exploitation
+    of thin oil rim reservoirs with large gas cap and aquifer, this author had highlighted
+    the problem. As part of sensitives in horizontal well standoff, Cosmos and Fatoke
+    (2004) tested three positions; one-third, centre and two-third positions from
+    the GOC in a Niger Delta field. They concluded that the landing closest to the
+    GOC (one-third position) yielded lowest Oil compared to the centre and two-third
+    positions. Surprisingly the work done by Sai Garimella et al (2011) in a 60ft
+    Ghariff & Al Khlata shallow marine low permeability sandstone reservoirs in a
+    field in Oman showed a different result with the one-third position indicating
+    an optimum recovery from a horizontal well. Interestingly both authors positions
+    on the performance had support from other authors.
+
+    This study used a 3D reservoir model, investigated different horizontal well standoff
+    performances and applied permeability reduction to simulate different reservoir
+    quality. The objective was to see if the reservoir quality was a factor in the
+    different horizontal well standoff performance seen from different regions of
+    the world while noting their different depositional environments. Results from
+    the investigation is presented in this paper and shows a different trend from
+    both authors mentioned above.'
+  - The oil extraction process typically involves drilling a well into the earth's
+    crust where oil deposits are located. The well is often lined with casing to prevent
+    collapse and water intrusion. Once the well is drilled, various techniques such
+    as primary recovery, secondary recovery, and tertiary recovery can be employed.
+    Primary recovery uses natural reservoir pressure to extract oil, while secondary
+    recovery employs water or gas injection to maintain pressure. Tertiary recovery,
+    also known as enhanced oil recovery, uses techniques like thermal injection or
+    chemicals to further reduce the viscosity of oil and increase extraction rates.
+    Each of these methods has distinct implications on the yield and economic viability
+    of oil extraction operations.
+- source_sentence: What advantages do helicopters have over fixed-wing aircraft for
+    leak detection surveys?
+  sentences:
+  - The reservoir characteristics such as porosity and permeability are crucial for
+    evaluating the potential of oil and gas fields. Porosity refers to the void spaces
+    within rocks that can hold hydrocarbons, while permeability measures how easily
+    fluids can flow through rock formations. These two properties significantly influence
+    the extraction methods used and the overall productivity of a reservoir. Enhancing
+    permeability through hydraulic fracturing has become a common technique in unconventional
+    resource extraction, allowing for more efficient recovery of oil and gas from
+    low-permeability reservoirs.
+  - 'BP gas production operations in North America manages over 15,000 miles of onshore
+    pipelines that make up our vast, complex, and aging gas gathering networks. Surveying
+    these for leaks presents a huge resource challenge using current ground based
+    technology and, in turn, impacts the assurance of the safety and integrity of
+    these operations.
+
+    The Exploration and Production Technology Group evaluated new leak detection technologies
+    using laser, thermal imaging camera and a high speed gas sampling detector that
+    were deployed on aircraft and used global positioning systems coordinates to survey
+    gas gathering pipelines. Field trials on gas gathering systems in the North Texas,
+    Anadarko asset showed that the laser and gas sampling based leak detection systems
+    were the most accurate, but the video imaging from the thermal camera made a powerful
+    statement. Helicopters proved to be more suitable in leak detection surveys on
+    gas gathering pipelines than that of fixed-wing aircraft.
+
+    The aerial leak detection technologies produce a significant increase in efficiency
+    and productivity in managing the integrity of BP''s gas gathering systems. While
+    that improves business performance, perhaps more importantly is the fact that
+    small gas leaks can be easily found before they become big ones. That reduces
+    environmental damage and the potential for leaks to impact the public. The development
+    and implementation of aerial leak detection in BP is being recognized as an integrity
+    tool in providing a significantly improved integrity assurance to its gas gathering
+    operations.'
+  - 'One of prerequisite of any detection system is to get the requirement the risk
+    analysis that estimates mainly the safety and environmental impacts of a loss
+    of containment. From this prerequisite it is possible to consider a strategy for
+    an early detection of a loss of containment, and to choose a method or a technology.
+    Methods of detection belong to two main families:
+
+    External based Leak Detection System which used local leak sensors to generate
+    a leak alarm. The main External based Leak Detection Systems are acoustic emission
+    detectors, pressure detectors, fiber optic cable, vapor and / or liquid sensing
+    cables;
+
+    Internal based Leak Detection Systems which used normal field sensors (e.g. pressure
+    transmitters, flowmeters) for leak detection and leak localization. The main internal
+    Leak Detection Systems are:
+
+    ○
+
+    balancing systems (line balance, volume balance, compensated mass balance etc.);
+
+    ○
+
+    Real Time Transient Model;
+
+    ○
+
+    pressure/ flow monitoring;
+
+    ○
+
+    statistical analysis…
+
+    The main External based Leak Detection Systems was studied internally through
+    different evaluation and development programs and for some of them in operation.
+
+    The main findings were the followings:
+
+    The acoustic based detection is sensitive to external noises as well as some pipeline
+    fluid (multi-phase, critical flow, transit phase) and pipeline elements (e.g.
+    elbows, valves). This technology requires the management of high quantity of data,
+    a significant tuning period, and many sensors connected to the pipeline. Distributed
+    Acoustic Sensing (DAS) using the fiber optic cable media is currently used internally
+    to detect real time intrusion.
+
+    The pressure emission detectors may be insensitive and require accurate pressure
+    measurement. This technology is difficultly practical on short lines, gas or multi-phase
+    pipelines with transient phases.
+
+    The vapor / liquid sensing cable technology needs to be physically close to the
+    pipe to become wet in case of leakage. These sensitive cables should be replaced
+    or cleaned after a leak. This technology is ne suitable easily for long distance
+    application. Their retrievable capability with the implementation of pulling chamber
+    every few hundred meters needs to be carefully considered. In addition, this technology
+    is highly sensitive. This implies that false alarms may occurred in case of former
+    contamination (presence of hydrocarbon). This technology is also sensitive to
+    the soil disruption, fluid properties and is affected by the ageing (sensitive
+    polymer alteration). However, this technology is suitable for short distance and
+    for some leaks detection when there is no temperature variation between the fluid
+    and the soil.
+
+    The Fiber optic solution was highly considered for a leak detection through several
+    evaluation programs and, in particular two PIT (Projet d’Innovation Technologique)
+    Projects. These two PIT projects were performed between 2015 and 2019 and presented
+    to the following ADIPEC sessions
+
+    ○
+
+    (Baque, 2017) 2017 Abu Dhabi International Petroleum Exhibition & Conference SPE-188669-MS
+    Early Gas Detection
+
+    ○
+
+    (Baque, 2020) 2020 Abu Dhabi International Petroleum Exhibition & Conference SPE-203293-MS
+    Fiber Optic Liquid Leakage Detection
+
+    Note: Some of the paragraph parts of this manuscript are extracted from these
+    two SPE documents referred (Baque, 2017) and (Baque, 2020). Other evaluation and
+    development programs not presented previously are also presented in this manuscript.'
+- source_sentence: What occupational health hazards are anticipated with large construction
+    projects during the energy transition?
+  sentences:
+  - "institutionalized political structures to realize particular social objectives\
+    \ or serve particular\nconstituencies.  \n**Non-hazardous waste:** Waste, other\
+    \ than Hazardous waste, resulting from company\noperations, including process\
+    \ and oil field wastes disposed of, on site or off site, as well as\noffice, commercial\
+    \ or packaging related wastes [ENV-7].  \n**Normalization:** The ratio of a quantitative\
+    \ indicator output (e.g. emissions) to an\naggregated measure of another output\
+    \ (e.g. oil and gas production or refinery throughput)  \n[Module 1 _Reporting\
+    \ process_ ].  \n**Occupational illness:** An Employee or Contractor health condition\
+    \ or disorder requiring\nmedical treatment due to a workplace Incident, typically\
+    \ involving multiple exposures to\nhazardous substances or to physical agents.\
+    \ Examples include noise-induced hearing loss,\nrespiratory disease, and contact\
+    \ dermatitis [SHS-3].  \n**Occupational injury:** Harm of an Employee or Contractor\
+    \ resulting from a single\ninstantaneous workplace incident that results in medical\
+    \ treatment (beyond simple first aid),\nwork restrictions, days away from work\
+    \ (lost time) or a Fatality [SHS-3].  \n**Operating area:** An area where business\
+    \ activities take place with potential to interact with\nthe adjacent environment\
+    \ [ENV-4].  \n**Operation:** A generic term used to denote any kind of business\
+    \ activity involving productrelated processes, such as production, manufacturing\
+    \ and transport. Note: the term oil and\ngas operations used in the Guidance is\
+    \ intended to be broad and inclusive of other types of\nproduct, such as chemicals.\
+    \  \n**7.5**"
+  - "The broader work of the Directorate is carried out  \nby its four standing committees.\
+    \  \nSafety Committee: This committee’s objective is the  \ncore of the Directorate:\
+    \ to eliminate fatalities and  \ncatastrophic process safety events in our industry.\n\
+    In pursuit of this aim, the committee develops\nand promotes the adoption of recommended\n\
+    practices – a task it performs both on its own and\nwith partners and trade associations.\
+    \ The resulting\npublications lay a foundation for both safety\nand efficiency,\
+    \ and develops the motivated and\nempowered workforce needed to provide the world\n\
+    with clean, affordable energy.  \nHighlights of the committee’s 2023 activities\n\
+    include participation in events, the creation\nof expert groups, issuing of publications,\
+    \ and\nengagement in data reviews.  \n- Events: In 2023, in addition to the regular\n\
+    committee and subcommittee meetings,\nthe committee held diving workshops in\n\
+    Rio De Janeiro and Paris. These meetings\nchampioned local stakeholders and sought\n\
+    to improve local diving performance. It also\nhosted two Aviation Procurement\
+    \ Managers\nForums – one in London, the other in Houston  \n– to address industry\
+    \ contracting behaviours  \nand its impact on contractor resilience and\nsafety.\
+    \ In addition, the committee conducted a\nProcess Safety Workshop at the IOGP\
+    \ Summit\nin Indonesia. Finally, at this year’s Offshore\nEurope conference, IOGP\
+    \ Safety Director Steve\nNorton moderated a panel on learning from, and\nsharing,\
+    \ safety lessons.  \n- Expert Groups: The committee established\nthree expert\
+    \ groups in 2023: two to revise\nexisting Reports (365 on land transportation\n\
+    safety and 365-12 on in-vehicle monitoring), and\none to consider adoption and\
+    \ implementation of\nrecommended safety practices.  \n- Publications: The committee\
+    \ issued ten  \nguidance documents in 2023, covering critical\nareas such as diving,\
+    \ aviation, and process\nsafety; see page 34 for a full list of publications.\
+    \  \n- Data reviews: The committee published its\nannual compilations of safety\
+    \ performance data,\ncovering occupational, process, aviation, and\nland transportation\
+    \ safety. IOGP has collected\nsafety performance data from its Members\nsince\
+    \ 1985 and our database is the largest in\nthe upstream industry, providing companies\n\
+    with valuable information for benchmarking and\nperformance improvement.  \n17"
+  - "endotoxins and fungi. The authors recommended that\nongoing real–time measurement\
+    \ of these exposures be\ncarried out to identify boundary conditions, phases,\
+    \ and\nsettings with the highest pollutant release.  \n12 — Health in the energy\
+    \ transition  \nGood quality studies are needed on the health effects of\nrenewable\
+    \ energy sources. Such studies should include\npopulations and patients with well-characterized\
+    \ exposure,\nhigh-quality information on outcome, and assessment of\npotential\
+    \ confounders. While retrospective (e.g., case-control)\nstudies might produce\
+    \ useful results, prospective longitudinal\nstudies would provide the strongest\
+    \ evidence.  \nSeveral LCA studies have been conducted for the different\ntechnologies.\
+    \ These LCAs reported relative low levels of\nemissions during the lifecycle of\
+    \ renewable sources of\nenergy. Few of these studies included a comparison with\n\
+    fossil-based technologies. When more life cycle studies\nbecome available it would\
+    \ be important to include them\nin the literature review. While looking at the\
+    \ life cycle of a\ncertain technology, other health effects in the value chain\n\
+    could potentially be identified (reference: UNECE on Carbon\nNeutrality in the\
+    \ UNECE Region: Integrated Life-cycle\nAssessment of Electricity Sources).  \n\
+    As of December 2024, very few occupational and public\nhealth hazards specific\
+    \ to energy transition technologies\nhave been identified. The energy transition\
+    \ is in an early stage\nand will evolve quickly, and additional hazards unique\
+    \ to\nenergy transition activities may emerge; the specifics of this\nare, at\
+    \ this time, uncertain.  \nWhat is certain is that the energy transition will\
+    \ involve large\nconstruction projects whose risks (and effective methods to\n\
+    manage those risks) are well-known and understood. Existing\noccupational health\
+    \ approaches will be able to manage\nthese risks effectively, provided the correct\
+    \ assessments are\nconducted properly."
+datasets:
+- Sampath1987/offshore_energy_v1
+pipeline_tag: sentence-similarity
+library_name: sentence-transformers
+metrics:
+- cosine_accuracy
+model-index:
+- name: SentenceTransformer based on Alibaba-NLP/gte-multilingual-base
+  results:
+  - task:
+      type: triplet
+      name: Triplet
+    dataset:
+      name: ai job validation
+      type: ai-job-validation
+    metrics:
+    - type: cosine_accuracy
+      value: 0.9700252413749695
+      name: Cosine Accuracy
+---
+
+# SentenceTransformer based on Alibaba-NLP/gte-multilingual-base
+
+This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Alibaba-NLP/gte-multilingual-base](https://huggingface.co/Alibaba-NLP/gte-multilingual-base) on the [offshore_energy_v1](https://huggingface.co/datasets/Sampath1987/offshore_energy_v1) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+
+## Model Details
+
+### Model Description
+- **Model Type:** Sentence Transformer
+- **Base model:** [Alibaba-NLP/gte-multilingual-base](https://huggingface.co/Alibaba-NLP/gte-multilingual-base) <!-- at revision 9bbca17d9273fd0d03d5725c7a4b0f6b45142062 -->
+- **Maximum Sequence Length:** 8192 tokens
+- **Output Dimensionality:** 768 dimensions
+- **Similarity Function:** Cosine Similarity
+- **Training Dataset:**
+    - [offshore_energy_v1](https://huggingface.co/datasets/Sampath1987/offshore_energy_v1)
+<!-- - **Language:** Unknown -->
+<!-- - **License:** Unknown -->
+
+### Model Sources
+
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
+
+### Full Model Architecture
+
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'NewModel'})
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+  (2): Normalize()
+)
+```
+
+## Usage
+
+### Direct Usage (Sentence Transformers)
+
+First install the Sentence Transformers library:
+
+```bash
+pip install -U sentence-transformers
+```
+
+Then you can load this model and run inference.
+```python
+from sentence_transformers import SentenceTransformer
+
+# Download from the 🤗 Hub
+model = SentenceTransformer("Sampath1987/EnergyEmbed-v2-e3")
+# Run inference
+sentences = [
+    'What occupational health hazards are anticipated with large construction projects during the energy transition?',
+    'endotoxins and fungi. The authors recommended that\nongoing real–time measurement of these exposures be\ncarried out to identify boundary conditions, phases, and\nsettings with the highest pollutant release.  \n12 — Health in the energy transition  \nGood quality studies are needed on the health effects of\nrenewable energy sources. Such studies should include\npopulations and patients with well-characterized exposure,\nhigh-quality information on outcome, and assessment of\npotential confounders. While retrospective (e.g., case-control)\nstudies might produce useful results, prospective longitudinal\nstudies would provide the strongest evidence.  \nSeveral LCA studies have been conducted for the different\ntechnologies. These LCAs reported relative low levels of\nemissions during the lifecycle of renewable sources of\nenergy. Few of these studies included a comparison with\nfossil-based technologies. When more life cycle studies\nbecome available it would be important to include them\nin the literature review. While looking at the life cycle of a\ncertain technology, other health effects in the value chain\ncould potentially be identified (reference: UNECE on Carbon\nNeutrality in the UNECE Region: Integrated Life-cycle\nAssessment of Electricity Sources).  \nAs of December 2024, very few occupational and public\nhealth hazards specific to energy transition technologies\nhave been identified. The energy transition is in an early stage\nand will evolve quickly, and additional hazards unique to\nenergy transition activities may emerge; the specifics of this\nare, at this time, uncertain.  \nWhat is certain is that the energy transition will involve large\nconstruction projects whose risks (and effective methods to\nmanage those risks) are well-known and understood. Existing\noccupational health approaches will be able to manage\nthese risks effectively, provided the correct assessments are\nconducted properly.',
+    'institutionalized political structures to realize particular social objectives or serve particular\nconstituencies.  \n**Non-hazardous waste:** Waste, other than Hazardous waste, resulting from company\noperations, including process and oil field wastes disposed of, on site or off site, as well as\noffice, commercial or packaging related wastes [ENV-7].  \n**Normalization:** The ratio of a quantitative indicator output (e.g. emissions) to an\naggregated measure of another output (e.g. oil and gas production or refinery throughput)  \n[Module 1 _Reporting process_ ].  \n**Occupational illness:** An Employee or Contractor health condition or disorder requiring\nmedical treatment due to a workplace Incident, typically involving multiple exposures to\nhazardous substances or to physical agents. Examples include noise-induced hearing loss,\nrespiratory disease, and contact dermatitis [SHS-3].  \n**Occupational injury:** Harm of an Employee or Contractor resulting from a single\ninstantaneous workplace incident that results in medical treatment (beyond simple first aid),\nwork restrictions, days away from work (lost time) or a Fatality [SHS-3].  \n**Operating area:** An area where business activities take place with potential to interact with\nthe adjacent environment [ENV-4].  \n**Operation:** A generic term used to denote any kind of business activity involving productrelated processes, such as production, manufacturing and transport. Note: the term oil and\ngas operations used in the Guidance is intended to be broad and inclusive of other types of\nproduct, such as chemicals.  \n**7.5**',
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 768]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities)
+# tensor([[1.0000, 0.5463, 0.1943],
+#         [0.5463, 1.0000, 0.1698],
+#         [0.1943, 0.1698, 1.0000]])
+```
+
+<!--
+### Direct Usage (Transformers)
+
+<details><summary>Click to see the direct usage in Transformers</summary>
+
+</details>
+-->
+
+<!--
+### Downstream Usage (Sentence Transformers)
+
+You can finetune this model on your own dataset.
+
+<details><summary>Click to expand</summary>
+
+</details>
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+## Evaluation
+
+### Metrics
+
+#### Triplet
+
+* Dataset: `ai-job-validation`
+* Evaluated with [<code>TripletEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator)
+
+| Metric              | Value    |
+|:--------------------|:---------|
+| **cosine_accuracy** | **0.97** |
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Dataset
+
+#### offshore_energy_v1
+
+* Dataset: [offshore_energy_v1](https://huggingface.co/datasets/Sampath1987/offshore_energy_v1) at [4e9339c](https://huggingface.co/datasets/Sampath1987/offshore_energy_v1/tree/4e9339cce67dbff9d1e6ba25cfdcd9a4a7f529f7)
+* Size: 53,913 training samples
+* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | anchor                                                                             | positive                                                                             | negative                                                                              |
+  |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|
+  | type    | string                                                                             | string                                                                               | string                                                                                |
+  | details | <ul><li>min: 14 tokens</li><li>mean: 23.77 tokens</li><li>max: 42 tokens</li></ul> | <ul><li>min: 36 tokens</li><li>mean: 392.08 tokens</li><li>max: 961 tokens</li></ul> | <ul><li>min: 45 tokens</li><li>mean: 389.63 tokens</li><li>max: 1109 tokens</li></ul> |
+* Samples:
+  | anchor                                                                                                                             | positive                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | negative                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+  |:-----------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | <code>What statistical methods were employed to enhance the accuracy of comparisons in the field testing of shaped cutters?</code> | <code>As shaped polycrystalline diamond compact (PDC) cutter geometries become more prevalent across the industry, this paper statistically reviews field testing of novel shaped PDC cutters in a variety of challenging applications. Firstly, the paper identifies the improvement in efficiency when compared with conventional PDC cutter geometries. Secondly, it confirms the reliability and robustness of the aforementioned shaped cutter geometries.<br>After several years of field testing shaped PDC cutter geometries, the question of how they hold up against conventional cylinder-shaped cutters remains unanswered. This study looks at drill bits that have the same overall design; however, each bit has different shape configurations that are deployed in a range of hole sizes and drilling applications. Data was collected from more than 100 runs and included advanced dull evaluation techniques, data mining, and comparative analyses. During data collation and interpretation, several statistical methods we...</code>    | <code>This paper details the improvements to drilling performance and torsional response of fixed cutter bits when changing from a conventional 19-mm cutter diameter configuration to 25-mm cutter diameters for similar blade counts in two different hole sizes. Key performance metrics include rate of penetration (ROP), rerun-ability, torsional response, and ability to maintain tool-face control during directional drilling.<br>A high-performance drilling application was selected with several existing offset wells using a 12¼-in., five-bladed, 19-mm (519) drill bit design, and a concept bit developed using 25-mm diameter cutters while maintaining comparable ancillary features. This was tested in the same field on both vertical and S-shape sections using the same bent-housing motor assembly and drilling performance compared to the existing offsets. A 17½-in. hole size application that experiences high drillstring vibration was also selected, and a 25-mm cutter diameter drill bit was designed with co...</code>                               |
+  | <code>What are vapor recovery units (VRU) used for in oil and gas operations?</code>                                               | <code>## 4. Vapour recovery units  <br>Vapor recovery units (VRU) are used to prevent emissions by capturing the streams and<br>re-routing them either back to the process or for use as fuel. More details on the<br>components, installation, and operation of VRU are captured in the following sections.</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | <code>##### 3.1.2 Reduction and recovery of glycol dehydration flash gas  <br>Gas from the flash vessel will consist primarily of hydrocarbons and is continuously<br>produced. If installed, a flash vessel will typically remove 90% or more of the entrained<br>hydrocarbon gas and dissolved gases in the glycol leaving the contactor column.  <br>Glycol flash vessels typically operate at 3-7 barg [18], meaning there is generally a sufficient<br>pressure drop for the flash gas to commonly be routed to flare or a low-pressure fuel gas<br>system. If the composition of the flash gas prevents this, or there is no fuel gas system,<br>then a Vapour Recovery Unit (VRU) may be needed for recovery into other process units.  <br>Minimization of the flash gas itself is also possible by optimizing the glycol flowrate,<br>such as by adjusting the dry gas water temperature specification based on accurate site<br>conditions because the water dew point needed could vary seasonally or from site to<br>site by using more accurate ambient temperatur...</code> |
+  | <code>What challenges are posed by fractures and faults in the completion of MRC wells?</code>                                     | <code>The Maximum Reservoir Contact (MRC) concept was developed to improve well productivity and sustainability by maximizing the contact area with target reservoirs. MRC is a proven technology for the development of tight/non-economical reservoirs. Completion design for MRC wells plays a vital role in enhancing well deliverability, monitoring and accessibility.<br>MRC technology was put into application to appraise a tight and thin heterogeneous carbonate reservoir in a giant offshore field in Abu Dhabi. Different completion scenarios were simulated to select the best suited completion to achieve enhanced well deliverability, monitoring and accessibility.<br>Heavy casing design with liner and tie-back system was finalized to maximize accessibility and achieve proper isolation behind casing. A special pre-perforated liner was also designed to eliminate the pressure drop across the wellbore. The MRC drain was divided mainly into two sections, blank pipe and pre-perforated liner equipped with swell ...</code> | <code>The Clair field is the largest discovered oilfield on the UK continental shelf (UKCS) but has high reservoir uncertainty associated with a complex natural fracture network. The field area covers over 200 sq km with an estimated STOIIP of 7 billion barrels. The scale and complexity of the reservoir has led to a phased multi-platform development.<br>Phase 1 started production in 2005 with 20 wells drilled prior to an extended drill break. Five new wells (A21 to A25) were drilled and brought online during 2016/17 which increased platform production by c.70%. The new wells incorporated historic lessons to mitigate the risk of wellbore instability in the overburden and be robust to the dynamic uncertainties of the fractured reservoir. Many of the well outcomes and risk events were predicted and mitigated effectively, however the new wells still provided some surprises.<br>This paper presents a summary of the lessons from the historic Clair development wells which underpinned the recent drilling c...</code>                            |
+* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
+  ```json
+  {
+      "scale": 20.0,
+      "similarity_fct": "cos_sim",
+      "gather_across_devices": false
+  }
+  ```
+
+### Evaluation Dataset
+
+#### offshore_energy_v1
+
+* Dataset: [offshore_energy_v1](https://huggingface.co/datasets/Sampath1987/offshore_energy_v1) at [4e9339c](https://huggingface.co/datasets/Sampath1987/offshore_energy_v1/tree/4e9339cce67dbff9d1e6ba25cfdcd9a4a7f529f7)
+* Size: 6,739 evaluation samples
+* Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | anchor                                                                             | positive                                                                              | negative                                                                             |
+  |:--------|:-----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|
+  | type    | string                                                                             | string                                                                                | string                                                                               |
+  | details | <ul><li>min: 11 tokens</li><li>mean: 23.56 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>min: 55 tokens</li><li>mean: 386.01 tokens</li><li>max: 1082 tokens</li></ul> | <ul><li>min: 45 tokens</li><li>mean: 382.6 tokens</li><li>max: 1175 tokens</li></ul> |
+* Samples:
+  | anchor                                                                                                                              | positive                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | negative                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+  |:------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | <code>What is the importance of quantifying carbon emissions during cementing operations in decarbonization?</code>                 | <code>An important step in decarbonization is using an end-to-end approach to quantify carbon emissions during cementing operations. By careful analysis of the entire cementing operations process, it is then possible to measure and compare carbon emissions at various stages of the operation. Understanding and isolating the main drivers of the carbon emissions footprint enables making better choices and developing best alternatives with lower environmental impact.<br>The methodology considers the lifecycle assessment of cement from quarry extraction to well abandonment, and includes steps such as manufacturing of raw materials, transportation and logistics, and operations in the field. For these stages, careful quantification of emissions is performed based on the manufacturer's carbon emissions of cementing products, transportation (distance and means) to the bulk plant and rig site, and equipment-related emissions such as blending and pumping units. In some cases, when assessing the footprint ...</code> | <code>Objectives/Scope<br>There are many different views on the Energy Transition. What is agreed is that to achieve current climate change targets, the journey to deep decarbonisation must start now. Scope 3 emissions are clearly the major contributor to total emissions and must be actively reduced. However, if Oil and Gas extraction is to be continued, then operators must understand, measure, and reduce Scope 1 and 2 emissions. This paper examines the constituent parts of typical Scope 1 emissions for O&G assets and discusses a credible pathway and initial steps towards decarbonisation of operations.<br>Methods, Procedures, Process<br>Emissions from typical assets are investigated: data is examined to determine the overall and individual contributions of Scope 1 emissions. A three tiered approach to emissions savings is presented:<br>–<br>Reduce overall energy usage<br>–<br>Seek to Remove environmental losses<br>–<br>Replace energy supply with low carbon alternatives<br>A simple method, used to assess carbon emissions,...</code> |
+  | <code>What factors must engineers consider during the drilling design phase?</code>                                                 | <code>The drilling of oil and gas wells involves several stages including the exploration phase, drilling design, and perforation techniques. In the exploration phase, geologists use seismic surveys to identify potential drilling locations. During the drilling design phase, engineers must consider factors such as wellbore stability, fluid mechanics, and formation pressures. Once the well is drilled, perforation techniques are applied to enhance the flow of hydrocarbons into the wellbore. The effectiveness of these techniques can significantly impact production rates and overall project success.</code>                                                                                                                                                                                                                                                                                                                                                                                                                            | <code>The extraction of crude oil and natural gas is typically carried out through drilling. Drilling uses different techniques to reach the petroleum reservoirs located deep underground. One key method is rotary drilling, where a drill bit is rotated while cutting through the earth's layers to create a wellbore. Rotary drilling is favored for its efficiency in penetrating hard rock layers. Another method is directional drilling, which allows operators to drill at various angles to reach reservoirs that are not directly beneath the drilling platform. This technique increases the area covered by the well and can optimize production. In addition, hydraulic fracturing enhances recovery rates by injecting fluids under high pressure to create fractures in the rock, increasing the permeability and allowing oil and gas to flow more freely. Lastly, the safety and environmental impacts of drilling techniques are a growing concern, and advancements are continually being sought to mitigate these effect...</code>                               |
+  | <code>How does the 'Dissolved pore network' concept enhance matrix permeability in the modeling of carbonate oil reservoirs?</code> | <code>In this paper, we present a case study of using dual porosity dual permeability (DPDP) simulation for an offshore Abu Dhabi carbonate oil reservoir exhibiting complex flow behavior through matrix, fracture system and conductive faults. The main objective of the study is to present and explain the reservoir flow behaviors by constructing and using advanced reservoir geologic and simulation models. The results of the study will be utilized as part of the inputs for full field development plan.<br>Initially, an extensive work on the faults and fractures characterization was conducted to properly integrate this information into a dynamic model using DPDP modeling approach. However, the poor response of some wells or field sectors indicated the insufficiency of this concept to capture the full complexity of the reservoir system. Consequently, a new geological concept was proposed to represent the effect of enhanced matrix permeability related to facies dissolution process in the reservoir mode...</code> | <code>Integration of pressure-derived permeability thickness with other geological data plays a crucial role in estimating the apparent reservoir permeability, which is a key reservoir property required for reliable reservoir characterization as it governs fluid flow and greatly impacts decisions related to production, field development, and reservoir management. The geological model provides a representation of the subsurface reservoir, capturing the spatial distribution of lithology, porosity, permeability, and other geological properties. Analysis of pressure data provides valuable information on well condition, reservoir extent, and dynamic reservoir parameters. Integrating such data with the geological model is an enabler to better quantify and manage the uncertainty in the spatial 3D distribution of permeability away from well control.<br>This work proposes a methodology to build high-resolution geological models based on the available dynamic data, seismic data, and geologic interpretati...</code>                            |
+* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
+  ```json
+  {
+      "scale": 20.0,
+      "similarity_fct": "cos_sim",
+      "gather_across_devices": false
+  }
+  ```
+
+### Training Hyperparameters
+#### Non-Default Hyperparameters
+
+- `eval_strategy`: steps
+- `per_device_train_batch_size`: 16
+- `per_device_eval_batch_size`: 16
+- `learning_rate`: 2e-05
+- `warmup_ratio`: 0.1
+
+#### All Hyperparameters
+<details><summary>Click to expand</summary>
+
+- `overwrite_output_dir`: False
+- `do_predict`: False
+- `eval_strategy`: steps
+- `prediction_loss_only`: True
+- `per_device_train_batch_size`: 16
+- `per_device_eval_batch_size`: 16
+- `per_gpu_train_batch_size`: None
+- `per_gpu_eval_batch_size`: None
+- `gradient_accumulation_steps`: 1
+- `eval_accumulation_steps`: None
+- `torch_empty_cache_steps`: None
+- `learning_rate`: 2e-05
+- `weight_decay`: 0.0
+- `adam_beta1`: 0.9
+- `adam_beta2`: 0.999
+- `adam_epsilon`: 1e-08
+- `max_grad_norm`: 1.0
+- `num_train_epochs`: 3
+- `max_steps`: -1
+- `lr_scheduler_type`: linear
+- `lr_scheduler_kwargs`: {}
+- `warmup_ratio`: 0.1
+- `warmup_steps`: 0
+- `log_level`: passive
+- `log_level_replica`: warning
+- `log_on_each_node`: True
+- `logging_nan_inf_filter`: True
+- `save_safetensors`: True
+- `save_on_each_node`: False
+- `save_only_model`: False
+- `restore_callback_states_from_checkpoint`: False
+- `no_cuda`: False
+- `use_cpu`: False
+- `use_mps_device`: False
+- `seed`: 42
+- `data_seed`: None
+- `jit_mode_eval`: False
+- `use_ipex`: False
+- `bf16`: False
+- `fp16`: False
+- `fp16_opt_level`: O1
+- `half_precision_backend`: auto
+- `bf16_full_eval`: False
+- `fp16_full_eval`: False
+- `tf32`: None
+- `local_rank`: 0
+- `ddp_backend`: None
+- `tpu_num_cores`: None
+- `tpu_metrics_debug`: False
+- `debug`: []
+- `dataloader_drop_last`: False
+- `dataloader_num_workers`: 0
+- `dataloader_prefetch_factor`: None
+- `past_index`: -1
+- `disable_tqdm`: False
+- `remove_unused_columns`: True
+- `label_names`: None
+- `load_best_model_at_end`: False
+- `ignore_data_skip`: False
+- `fsdp`: []
+- `fsdp_min_num_params`: 0
+- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
+- `fsdp_transformer_layer_cls_to_wrap`: None
+- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
+- `deepspeed`: None
+- `label_smoothing_factor`: 0.0
+- `optim`: adamw_torch
+- `optim_args`: None
+- `adafactor`: False
+- `group_by_length`: False
+- `length_column_name`: length
+- `ddp_find_unused_parameters`: None
+- `ddp_bucket_cap_mb`: None
+- `ddp_broadcast_buffers`: False
+- `dataloader_pin_memory`: True
+- `dataloader_persistent_workers`: False
+- `skip_memory_metrics`: True
+- `use_legacy_prediction_loop`: False
+- `push_to_hub`: False
+- `resume_from_checkpoint`: None
+- `hub_model_id`: None
+- `hub_strategy`: every_save
+- `hub_private_repo`: None
+- `hub_always_push`: False
+- `hub_revision`: None
+- `gradient_checkpointing`: False
+- `gradient_checkpointing_kwargs`: None
+- `include_inputs_for_metrics`: False
+- `include_for_metrics`: []
+- `eval_do_concat_batches`: True
+- `fp16_backend`: auto
+- `push_to_hub_model_id`: None
+- `push_to_hub_organization`: None
+- `mp_parameters`: 
+- `auto_find_batch_size`: False
+- `full_determinism`: False
+- `torchdynamo`: None
+- `ray_scope`: last
+- `ddp_timeout`: 1800
+- `torch_compile`: False
+- `torch_compile_backend`: None
+- `torch_compile_mode`: None
+- `include_tokens_per_second`: False
+- `include_num_input_tokens_seen`: False
+- `neftune_noise_alpha`: None
+- `optim_target_modules`: None
+- `batch_eval_metrics`: False
+- `eval_on_start`: False
+- `use_liger_kernel`: False
+- `liger_kernel_config`: None
+- `eval_use_gather_object`: False
+- `average_tokens_across_devices`: False
+- `prompts`: None
+- `batch_sampler`: batch_sampler
+- `multi_dataset_batch_sampler`: proportional
+- `router_mapping`: {}
+- `learning_rate_mapping`: {}
+
+</details>
+
+### Training Logs
+| Epoch  | Step  | Training Loss | Validation Loss | ai-job-validation_cosine_accuracy |
+|:------:|:-----:|:-------------:|:---------------:|:---------------------------------:|
+| 0.2967 | 1000  | -             | 0.1458          | 0.9605                            |
+| 0.5935 | 2000  | -             | 0.1217          | 0.9665                            |
+| 0.8902 | 3000  | -             | 0.1095          | 0.9711                            |
+| 1.1869 | 4000  | -             | 0.1131          | 0.9682                            |
+| 1.4837 | 5000  | 0.1672        | 0.1107          | 0.9687                            |
+| 1.7804 | 6000  | -             | 0.1030          | 0.9709                            |
+| 2.0772 | 7000  | -             | 0.1081          | 0.9693                            |
+| 2.3739 | 8000  | -             | 0.1091          | 0.9691                            |
+| 2.6706 | 9000  | -             | 0.1098          | 0.9691                            |
+| 2.9674 | 10000 | 0.0678        | 0.1065          | 0.9700                            |
+
+
+### Framework Versions
+- Python: 3.10.12
+- Sentence Transformers: 5.1.0
+- Transformers: 4.53.3
+- PyTorch: 2.8.0+cu128
+- Accelerate: 1.9.0
+- Datasets: 4.0.0
+- Tokenizers: 0.21.2
+
+## Citation
+
+### BibTeX
+
+#### Sentence Transformers
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084",
+}
+```
+
+#### MultipleNegativesRankingLoss
+```bibtex
+@misc{henderson2017efficient,
+    title={Efficient Natural Language Response Suggestion for Smart Reply},
+    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
+    year={2017},
+    eprint={1705.00652},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

config.json

@@ -0,0 +1,49 @@
+{
+  "architectures": [
+    "NewModel"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration.NewConfig",
+    "AutoModel": "modeling.NewModel",
+    "AutoModelForMaskedLM": "Alibaba-NLP/new-impl--modeling.NewForMaskedLM",
+    "AutoModelForMultipleChoice": "Alibaba-NLP/new-impl--modeling.NewForMultipleChoice",
+    "AutoModelForQuestionAnswering": "Alibaba-NLP/new-impl--modeling.NewForQuestionAnswering",
+    "AutoModelForSequenceClassification": "Alibaba-NLP/new-impl--modeling.NewForSequenceClassification",
+    "AutoModelForTokenClassification": "Alibaba-NLP/new-impl--modeling.NewForTokenClassification"
+  },
+  "classifier_dropout": 0.0,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "layer_norm_type": "layer_norm",
+  "logn_attention_clip1": false,
+  "logn_attention_scale": false,
+  "max_position_embeddings": 8192,
+  "model_type": "new",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pack_qkv": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "rope",
+  "rope_scaling": {
+    "factor": 8.0,
+    "type": "ntk"
+  },
+  "rope_theta": 20000,
+  "torch_dtype": "float32",
+  "transformers_version": "4.53.3",
+  "type_vocab_size": 1,
+  "unpad_inputs": false,
+  "use_memory_efficient_attention": false,
+  "vocab_size": 250048
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "SentenceTransformer",
+  "__version__": {
+    "sentence_transformers": "5.1.0",
+    "transformers": "4.53.3",
+    "pytorch": "2.8.0+cu128"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

configuration.py

@@ -0,0 +1,145 @@
+# coding=utf-8
+# Copyright 2024 The GTE Team Authors and Alibaba Group.
+# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" NEW model configuration"""
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class NewConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`NewModel`] or a [`TFNewModel`]. It is used to
+    instantiate a NEW model according to the specified arguments, defining the model architecture. Instantiating a
+    configuration with the defaults will yield a similar configuration to that of the NEW
+    [izhx/new-base-en](https://huggingface.co/izhx/new-base-en) architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 30522):
+            Vocabulary size of the NEW model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`NewModel`] or [`TFNewModel`].
+        hidden_size (`int`, *optional*, defaults to 768):
+            Dimensionality of the encoder layers and the pooler layer.
+        num_hidden_layers (`int`, *optional*, defaults to 12):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 12):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        intermediate_size (`int`, *optional*, defaults to 3072):
+            Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
+        hidden_act (`str` or `Callable`, *optional*, defaults to `"gelu"`):
+            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
+            `"relu"`, `"silu"` and `"gelu_new"` are supported.
+        hidden_dropout_prob (`float`, *optional*, defaults to 0.1):
+            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
+        attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1):
+            The dropout ratio for the attention probabilities.
+        max_position_embeddings (`int`, *optional*, defaults to 512):
+            The maximum sequence length that this model might ever be used with. Typically set this to something large
+            just in case (e.g., 512 or 1024 or 2048).
+        type_vocab_size (`int`, *optional*, defaults to 2):
+            The vocabulary size of the `token_type_ids` passed when calling [`NewModel`] or [`TFNewModel`].
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        layer_norm_eps (`float`, *optional*, defaults to 1e-12):
+            The epsilon used by the layer normalization layers.
+        position_embedding_type (`str`, *optional*, defaults to `"rope"`):
+            Type of position embedding. Choose one of `"absolute"`, `"rope"`.
+        rope_theta (`float`, *optional*, defaults to 10000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
+            strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
+            `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
+            `max_position_embeddings` to the expected new maximum. See the following thread for more information on how
+            these scaling strategies behave:
+            https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This is an
+            experimental feature, subject to breaking API changes in future versions.
+        classifier_dropout (`float`, *optional*):
+            The dropout ratio for the classification head.
+
+    Examples:
+
+    ```python
+    >>> from transformers import NewConfig, NewModel
+
+    >>> # Initializing a NEW izhx/new-base-en style configuration
+    >>> configuration = NewConfig()
+
+    >>> # Initializing a model (with random weights) from the izhx/new-base-en style configuration
+    >>> model = NewModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "new"
+
+    def __init__(
+        self,
+        vocab_size=30528,
+        hidden_size=768,
+        num_hidden_layers=12,
+        num_attention_heads=12,
+        intermediate_size=3072,
+        hidden_act="gelu",
+        hidden_dropout_prob=0.1,
+        attention_probs_dropout_prob=0.0,
+        max_position_embeddings=2048,
+        type_vocab_size=1,
+        initializer_range=0.02,
+        layer_norm_type='layer_norm',
+        layer_norm_eps=1e-12,
+        # pad_token_id=0,
+        position_embedding_type="rope",
+        rope_theta=10000.0,
+        rope_scaling=None,
+        classifier_dropout=None,
+        pack_qkv=True,
+        unpad_inputs=False,
+        use_memory_efficient_attention=False,
+        logn_attention_scale=False,
+        logn_attention_clip1=False,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.hidden_act = hidden_act
+        self.intermediate_size = intermediate_size
+        self.hidden_dropout_prob = hidden_dropout_prob
+        self.attention_probs_dropout_prob = attention_probs_dropout_prob
+        self.max_position_embeddings = max_position_embeddings
+        self.type_vocab_size = type_vocab_size
+        self.initializer_range = initializer_range
+        self.layer_norm_type = layer_norm_type
+        self.layer_norm_eps = layer_norm_eps
+        self.position_embedding_type = position_embedding_type
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.classifier_dropout = classifier_dropout
+
+        self.pack_qkv = pack_qkv
+        self.unpad_inputs = unpad_inputs
+        self.use_memory_efficient_attention = use_memory_efficient_attention
+        self.logn_attention_scale = logn_attention_scale
+        self.logn_attention_clip1 = logn_attention_clip1

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f813720e8b536a661d3048b868e8405cc63a902c23edf8ef7f5311253a7cce79
+size 1221487872

modeling.py

@@ -0,0 +1,1418 @@
+# coding=utf-8
+# Copyright 2024 The GTE Team Authors and Alibaba Group.
+# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch NEW model."""
+
+import math
+from dataclasses import dataclass
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.utils.checkpoint
+from torch import nn
+
+from transformers.activations import ACT2FN
+from transformers.modeling_outputs import (
+    BaseModelOutput,
+    BaseModelOutputWithPooling,
+    MaskedLMOutput,
+    MultipleChoiceModelOutput,
+    QuestionAnsweringModelOutput,
+    SequenceClassifierOutput,
+    ModelOutput,
+)
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import logging
+
+try:
+    import xformers.ops as xops
+except ImportError as e:
+    xops = None
+
+from .configuration import NewConfig
+
+
+logger = logging.get_logger(__name__)
+
+
+# Adapted from https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/bert_padding.py
+# Which was adapted from https://github.com/mlcommons/training_results_v1.1/blob/main/NVIDIA/benchmarks/bert/implementations/pytorch/padding.py
+class IndexFirstAxis(torch.autograd.Function):
+    @staticmethod
+    def forward(ctx, input, indices):
+        ctx.save_for_backward(indices)
+        assert input.ndim >= 2
+        ctx.first_axis_dim, other_shape = input.shape[0], input.shape[1:]
+        second_dim = other_shape.numel()
+        # TD [2022-03-04] For some reason torch.gather is a bit faster than indexing.
+        # return input[indices]
+        # return torch.gather(
+        #     rearrange(input, "b ... -> b (...)"), 0, repeat(indices, "z -> z d", d=second_dim)
+        # ).reshape(-1, *other_shape)
+        return torch.gather(
+            input.view(ctx.first_axis_dim, second_dim),
+            0,
+            indices.unsqueeze(-1).expand(indices.size(0), second_dim)
+        ).reshape(-1, *other_shape)
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        (indices,) = ctx.saved_tensors
+        assert grad_output.ndim >= 2
+        other_shape = grad_output.shape[1:]
+        # grad_output = rearrange(grad_output, "b ... -> b (...)")
+        grad_output = grad_output.view(grad_output.size(0), other_shape.numel())
+        grad_input = torch.zeros(
+            [ctx.first_axis_dim, grad_output.shape[1]],
+            device=grad_output.device,
+            dtype=grad_output.dtype,
+        )
+        # TD [2022-03-04] For some reason torch.scatter is a bit faster than indexing.
+        # grad_input[indices] = grad_output
+        # grad_input.scatter_(0, repeat(indices, "z -> z d", d=grad_output.shape[1]), grad_output)
+        grad_input.scatter_(
+            0, indices.unsqueeze(-1).expand(indices.size(0), grad_output.size(1)), grad_output
+        )
+        return grad_input.reshape(ctx.first_axis_dim, *other_shape), None
+
+
+index_first_axis = IndexFirstAxis.apply
+
+
+def unpad_input(hidden_states, attention_mask=None, indices=None):
+    """
+    Arguments:
+        hidden_states: (batch, seqlen, ...)
+        attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid.
+        indices: (total_nnz), the indices of non-masked tokens from the flattened input sequence.
+    Return:
+        hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask.
+    """
+    if indices is None:
+        assert attention_mask is not None
+        indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
+
+    # TD [2022-03-04] We don't want to index with a bool mask, because Pytorch will expand the
+    # bool mask, then call nonzero to get the indices, then index with those. The indices is @dim
+    # times larger than it needs to be, wasting memory. It's faster and more memory-efficient to
+    # index with integer indices. Moreover, torch's index is a bit slower than it needs to be,
+    # so we write custom forward and backward to make it a bit faster.
+    hidden_states = hidden_states.view(-1, *hidden_states.shape[2:])
+    return index_first_axis(hidden_states, indices)
+
+
+class IndexPutFirstAxis(torch.autograd.Function):
+    @staticmethod
+    def forward(
+        ctx,
+        values: torch.Tensor,
+        indices: torch.Tensor,
+        first_axis_dim
+    ) -> torch.Tensor:
+        ctx.save_for_backward(indices)
+        assert indices.ndim == 1
+        assert values.ndim >= 2
+        output = torch.zeros(
+            first_axis_dim, *values.shape[1:], device=values.device, dtype=values.dtype
+        )
+        output[indices] = values
+        return output
+
+    @staticmethod
+    def backward(ctx, grad_output: torch.Tensor) -> Tuple[torch.Tensor, None, None]:
+        indices, = ctx.saved_tensors
+        grad_values = grad_output[indices]
+        return grad_values, None, None
+
+
+index_put_first_axis = IndexPutFirstAxis.apply
+
+
+def pad_input(inputs: torch.Tensor, indices: torch.Tensor, batch: int, seqlen: int) -> torch.Tensor:
+    """Add padding to sequences.
+
+    Arguments:
+        inputs: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask.
+        indices: (total_nnz), `indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()`
+        batch: int batch_size
+        seqlen: int max sequence length
+
+    Returns:
+        inputs: (batch, seqlen, ...)
+    """
+    output = index_put_first_axis(inputs, indices, batch * seqlen)
+    return output.view(batch, seqlen, *inputs.shape[1:])
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q, k, cos, sin):
+    """Applies Rotary Position Embedding to the query and key tensors.
+
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        sin (`torch.Tensor`): The sine part of the rotary embedding.
+    Returns:
+        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
+    """
+    cos, sin = cos.to(q.dtype), sin.to(q.dtype)
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    return q_embed, k_embed
+
+
+class RotaryEmbedding(torch.nn.Module):
+    def __init__(self, dim, max_position_embeddings=512, base=10000.0, device=None):
+        super().__init__()
+
+        self.dim = dim
+        self.max_position_embeddings = max_position_embeddings
+        self.base = base
+        inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+        # Build here to make `torch.jit.trace` work.
+        self._set_cos_sin_cache(
+            seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype()
+        )
+
+    def _set_cos_sin_cache(self, seq_len, device, dtype):
+        self.max_seq_len_cached = seq_len
+        t = torch.arange(self.max_seq_len_cached, device=device, dtype=torch.float32)
+
+        freqs = torch.einsum("i,j->ij", t, self.inv_freq)
+        # Different from paper, but it uses a different permutation in order to obtain the same calculation
+        emb = torch.cat((freqs, freqs), dim=-1)
+        self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
+        self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
+
+    def forward(self, x, seq_len=None):
+        # x: [bs, num_attention_heads, seq_len, head_size]
+        if seq_len > self.max_seq_len_cached:
+            self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=x.dtype)
+
+        return (
+            self.cos_cached[:seq_len, ...].to(dtype=x.dtype),
+            self.sin_cached[:seq_len, ...].to(dtype=x.dtype),
+        )
+
+
+class NTKScalingRotaryEmbedding(RotaryEmbedding):
+    """RotaryEmbedding extended with fixed and mixed NTK scaling. https://kexue.fm/archives/9706 """
+
+    def __init__(self, dim, max_position_embeddings=512, base=10000, device=None, scaling_factor=1.0, mixed_b=None):
+        self.scaling_factor = scaling_factor
+        self.mixed_b = mixed_b
+        super().__init__(dim, max_position_embeddings, base, device)
+        max_position_embeddings = max_position_embeddings * self.scaling_factor
+        self._set_cos_sin_cache(max_position_embeddings, self.inv_freq.device, torch.get_default_dtype())
+
+    def _set_cos_sin_cache(self, seq_len, device, dtype):
+        self.max_seq_len_cached = seq_len
+
+        if seq_len > self.max_position_embeddings:
+            base = self.base * (self.scaling_factor if self.mixed_b is None else 1)
+            inv_freq = 1.0 / (base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
+
+            if self.mixed_b is None:
+                inv_freq = inv_freq / self.scaling_factor ** (2 / self.dim)  # (6)
+            else:
+                a = torch.tensor(self.scaling_factor).log() / (self.dim / 2) ** self.mixed_b  # (13)
+                lambda_1_m = (a * torch.arange(1, self.dim // 2 + 1).float().to(device) ** self.mixed_b).exp()  # (12)
+                inv_freq = inv_freq / lambda_1_m  # (10)
+
+            self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+        t = torch.arange(self.max_seq_len_cached, device=device, dtype=torch.float32)
+
+        freqs = torch.einsum("i,j->ij", t, self.inv_freq)
+        # Different from paper, but it uses a different permutation in order to obtain the same calculation
+        emb = torch.cat((freqs, freqs), dim=-1)
+        self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
+        self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        RMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+
+LAYER_NORM = {
+    'layer_norm': nn.LayerNorm,
+    'rms_norm': RMSNorm
+}
+
+
+class NewEmbeddings(nn.Module):
+    """
+    Embedding and Unpadding.
+    """
+
+    def __init__(self, config: NewConfig):
+        super().__init__()
+        self.padding_idx = config.pad_token_id
+        self.word_embeddings = nn.Embedding(
+            config.vocab_size, config.hidden_size, padding_idx=self.padding_idx
+        )
+
+        self.position_embedding_type = config.position_embedding_type
+        if self.position_embedding_type == 'absolute':
+            self.position_embeddings = nn.Embedding(
+                config.max_position_embeddings, config.hidden_size, padding_idx=self.padding_idx
+            )
+        elif self.position_embedding_type == 'rope':
+            self._init_rope(config)
+        else:
+            raise ValueError
+
+        self.type_vocab_size = config.type_vocab_size
+        if self.type_vocab_size > 0:
+            self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)
+
+        # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
+        # any TensorFlow checkpoint file
+        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        # position_ids is contiguous in memory and excluded when serialized
+        self.register_buffer(
+            "position_ids", torch.arange(config.max_position_embeddings), persistent=False
+        )
+
+    def _init_rope(self, config):
+        kwargs = dict(
+            dim=int(config.hidden_size / config.num_attention_heads),
+            max_position_embeddings=config.max_position_embeddings,
+            base=config.rope_theta
+        )
+        if config.rope_scaling is None:
+            self.rotary_emb = RotaryEmbedding(**kwargs)
+        else:
+            kwargs.update(scaling_factor=config.rope_scaling["factor"])
+            scaling_type = config.rope_scaling["type"]
+            if scaling_type == 'ntk':
+                kwargs.update(mixed_b=config.rope_scaling.get('mixed_b', None))
+                self.rotary_emb = NTKScalingRotaryEmbedding(**kwargs)
+            # elif scaling_type == "linear":
+            #     self.rotary_emb = LinearScalingRotaryEmbedding(**kwargs)
+            # elif scaling_type == "dynamic":
+            #     self.rotary_emb = DynamicNTKScalingRotaryEmbedding(**kwargs)
+            else:
+                raise ValueError(f"Unknown RoPE scaling type {scaling_type}")
+
+    def forward(
+        self,
+        unpad_inputs: bool,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        length: Optional[List[int]] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+    ) -> Tuple[torch.Tensor, torch.Tensor, Optional[Tuple], Optional[List[int]]]:
+        """
+        """
+        if inputs_embeds is None:
+            device, input_shape = input_ids.device, input_ids.shape
+        else:
+            device, input_shape = inputs_embeds.device, inputs_embeds.shape[:2]
+        batch_size, seq_length = input_shape
+
+        # Set attention_mask if it's None
+        if attention_mask is None:
+            attention_mask = torch.ones(input_shape, device=device)
+            if length is not None:
+                for i, l in enumerate(length):
+                    attention_mask[i, l:] = 0
+
+        # Set attention_mask_bool for unpadding
+        if unpad_inputs:
+            attention_mask_bool = attention_mask.bool()
+            if length is None:
+                length = attention_mask.sum(-1).tolist()
+
+        # Get word embeddings
+        if inputs_embeds is None:
+            if unpad_inputs:
+                input_ids = input_ids[attention_mask_bool].unsqueeze(0)
+            inputs_embeds = self.word_embeddings(input_ids)
+        else:
+            if unpad_inputs:
+                inputs_embeds = inputs_embeds[attention_mask_bool].unsqueeze(0)
+        embeddings = inputs_embeds
+
+        # Set and unpad position_ids
+        if position_ids is None:
+            if seq_length > self.position_ids.size(0):
+                self.register_buffer(
+                    "position_ids", torch.arange(seq_length, device=embeddings.device), persistent=False
+                )
+            if unpad_inputs:
+                # [1, cumsum_seq_len]
+                position_ids = torch.cat([self.position_ids[:l] for l in length]).unsqueeze(0)
+            else:
+                # [bs, seq_len]
+                position_ids = self.position_ids[:seq_length].expand(batch_size, -1)
+        elif unpad_inputs:
+            position_ids = position_ids[attention_mask_bool].unsqueeze(0)  # [1, cumsum_seq_len]
+
+        # Compute rotary embedding
+        if self.position_embedding_type == 'rope':
+            rope_cos, rope_sin = self.rotary_emb(inputs_embeds, seq_len=seq_length)
+            rope_cos = rope_cos[position_ids].unsqueeze(2)  # [bs, seq_len, 1, dim]
+            rope_sin = rope_sin[position_ids].unsqueeze(2)  # [bs, seq_len, 1, dim]
+            rope_embeds = rope_cos, rope_sin
+        else:
+            rope_embeds = None
+
+        if self.type_vocab_size > 0:
+            if token_type_ids is None:
+                token_type_ids = position_ids.mul(0)
+            else:
+                if self.type_vocab_size < 2:
+                    token_type_ids.mul_(0)
+                if unpad_inputs:
+                    token_type_ids = token_type_ids[attention_mask_bool].unsqueeze(0)
+
+            token_type_embeddings = self.token_type_embeddings(token_type_ids)
+            embeddings = embeddings + token_type_embeddings
+
+        # BERT position
+        if self.position_embedding_type == "absolute":
+            position_embeddings = self.position_embeddings(position_ids)
+            embeddings = embeddings + position_embeddings
+
+        embeddings = self.LayerNorm(embeddings)
+        embeddings = self.dropout(embeddings)
+
+        return embeddings, attention_mask, rope_embeds, length
+
+
+class NewAttention(nn.Module):
+    def __init__(self, config: NewConfig, pack_qkv=None, use_memory_efficient_attention=None):
+        super().__init__()
+        self.config = config
+        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
+            raise ValueError(
+                f"The hidden size ({config.hidden_size}) is not a multiple of the number of attention "
+                f"heads ({config.num_attention_heads})"
+            )
+
+        self.hidden_size = config.hidden_size
+        self.num_attention_heads = config.num_attention_heads
+        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
+        self.all_head_size = self.num_attention_heads * self.attention_head_size
+
+        if pack_qkv is None:
+            pack_qkv = config.pack_qkv
+        self.pack_qkv = pack_qkv
+
+        if self.pack_qkv:
+            self.qkv_proj = nn.Linear(config.hidden_size, self.all_head_size * 3, bias=True)
+        else:
+            self.q_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+            self.k_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+            self.v_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+
+        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
+        self.o_proj = nn.Linear(config.hidden_size, config.hidden_size, bias=True)
+
+        if use_memory_efficient_attention is None:
+            use_memory_efficient_attention = self.config.use_memory_efficient_attention
+        self.use_memory_efficient_attention = use_memory_efficient_attention
+        self.memory_efficient_attention = None if xops is None else xops.memory_efficient_attention
+        if self.use_memory_efficient_attention:
+            assert self.memory_efficient_attention is not None, 'please install xformers'
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_bias: torch.FloatTensor,
+        rope_embeds: Optional[Tuple[torch.FloatTensor, torch.FloatTensor]] = None,
+        padding_inputs: Optional[Tuple] = None,  # indices, batch, seqlen
+        attention_scale: Optional[torch.FloatTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        output_attentions: Optional[bool] = False,
+        qkv_inputs: Optional[Tuple] = None,  # For RetroMAE
+    ) -> Tuple[torch.Tensor, ...]:
+        shape_hd = (self.num_attention_heads, self.attention_head_size)
+        # qkv
+        if self.pack_qkv and qkv_inputs is None:
+            qkv_pack = self.qkv_proj(hidden_states).split(self.all_head_size, dim=-1)
+        else:
+            if qkv_inputs is None:
+                qkv_inputs = (hidden_states, hidden_states, hidden_states)
+            qkv_pack = [
+                getattr(self, n + '_proj')(s) for s, n in zip(qkv_inputs, 'qkv')
+            ]
+        query_states, key_states, value_states = [t.view(t.shape[:-1] + shape_hd) for t in qkv_pack]
+
+        if self.config.position_embedding_type == 'rope':
+            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, *rope_embeds)
+
+        dtype = query_states.dtype
+
+        if self.config.logn_attention_scale and attention_scale is not None:
+            # https://kexue.fm/archives/8823
+            query_states = query_states * attention_scale.to(dtype)
+
+        if padding_inputs is not None:
+            query_states = pad_input(query_states.squeeze(), *padding_inputs)
+            key_states = pad_input(key_states.squeeze(), *padding_inputs)
+            value_states = pad_input(value_states.squeeze(), *padding_inputs)
+
+        if self.use_memory_efficient_attention:
+            assert self.memory_efficient_attention is not None, "xformers is not loaded"
+            assert output_attentions is False, "memory_efficient_attention do not output attentions"
+            assert head_mask is None, "Not support yet"
+            attention_probs = None
+            if torch.is_tensor(attention_bias):
+                attention_bias = attention_bias.to(dtype)
+            context_layer = self.memory_efficient_attention(
+                query_states,
+                key_states,
+                value_states,
+                attn_bias=attention_bias,
+                p=self.dropout.p
+            )
+        else:
+            if output_attentions and isinstance(self, NewSdpaAttention):
+                raise RuntimeError("SDPA do not output attentions")
+            context_layer, attention_probs = self._attention(
+                query_states, key_states, value_states, attention_bias, head_mask
+            )
+
+        if padding_inputs is not None:
+            context_layer = unpad_input(context_layer, indices=padding_inputs[0])
+
+        new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
+        context_layer = context_layer.view(new_context_layer_shape)
+
+        # output proj
+        attn_output = self.o_proj(context_layer)
+
+        # add attentions if we output them
+        outputs = (attn_output, attention_probs) if output_attentions else (attn_output,)
+        return outputs
+
+    def _attention(self, query_states, key_states, value_states, attention_bias, head_mask):
+        """
+        Args:
+            q/k/v: (B, L, n_head, head_dim),
+        Returns:
+            attn_output: (B L, n_head, head_dim)
+        """
+        query_states = query_states.transpose(1, 2)
+        key_states = key_states.transpose(1, 2)
+        value_states = value_states.transpose(1, 2)
+        # Take the dot product between "query" and "key" to get the raw attention scores.
+        attention_scores = torch.matmul(query_states, key_states.transpose(-1, -2))
+
+        attention_scores = attention_scores / math.sqrt(self.attention_head_size)
+        if attention_bias is not None:
+            # Apply the attention mask is (precomputed for all layers in BertModel forward() function)
+            attention_scores = attention_scores + attention_bias
+
+        # Normalize the attention scores to probabilities.
+        attention_probs = nn.functional.softmax(attention_scores, dim=-1)
+
+        # This is actually dropping out entire tokens to attend to, which might
+        # seem a bit unusual, but is taken from the original Transformer paper.
+        if self.dropout.p > 0:
+            attention_probs = self.dropout(attention_probs)
+
+        # Mask heads if we want to
+        if head_mask is not None:
+            attention_probs = attention_probs * head_mask
+
+        context_layer = torch.matmul(attention_probs, value_states)
+
+        context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
+        return context_layer, attention_probs
+
+
+class NewSdpaAttention(NewAttention):
+    """
+    New attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from
+    `NewAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to
+    SDPA API.
+    """
+    def __init__(self, config: NewConfig, **kwargs):
+        super().__init__(config, **kwargs)
+        # torch.backends.cuda.enable_mem_efficient_sdp(False)
+        # logger.warning(
+        #     "Disable memory efficient attention kernel for `NewSdpaAttention`, you can set "
+        #     "`use_memory_efficient_attention=True` if it expected to use."
+        # )
+
+    def _attention(self, query_states, key_states, value_states, attention_bias, head_mask):
+        attn_output = torch.nn.functional.scaled_dot_product_attention(
+            query_states.transpose(1, 2),
+            key_states.transpose(1, 2),
+            value_states.transpose(1, 2),
+            attn_mask=attention_bias,
+            dropout_p=self.dropout.p if self.training else 0.0,
+        )
+        attn_output = attn_output.permute(0, 2, 1, 3).contiguous()
+        return attn_output, None
+
+
+NEW_ATTENTION_CLASSES = {
+    "eager": NewAttention,
+    # "flash_attention_2": ,  # TODO
+    "sdpa": NewSdpaAttention,
+}
+
+
+class NewGatedMLP(nn.Module):
+    """
+    GLU Variants Improve Transformer.
+    """
+
+    def __init__(self, config: NewConfig):
+        super().__init__()
+        self.intermediate_size = config.intermediate_size
+        self.up_gate_proj = nn.Linear(config.hidden_size, self.intermediate_size * 2, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, config.hidden_size, bias=True)
+        self.act_fn = ACT2FN[config.hidden_act]
+        if config.hidden_dropout_prob > 0:
+            self.hidden_dropout = nn.Dropout(config.hidden_dropout_prob)
+        else:
+            self.hidden_dropout = None
+
+    def forward(self, hidden_states):
+        up_gate = self.up_gate_proj(hidden_states)
+        up_states, gate = torch.split(up_gate, self.intermediate_size, dim=-1)
+        gate = self.act_fn(gate)
+        gated_states = gate * up_states
+        if self.hidden_dropout is not None:
+            gated_states = self.hidden_dropout(gated_states)
+        down_states = self.down_proj(gated_states)
+        return down_states
+
+
+class NewLayer(nn.Module):
+    def __init__(
+        self,
+        config: NewConfig,
+        pack_qkv=None,
+        use_memory_efficient_attention=None,
+        attn_implementation=None
+    ):
+        super().__init__()
+        if attn_implementation is None:
+            attn_implementation = config._attn_implementation
+        if use_memory_efficient_attention is None:
+            use_memory_efficient_attention = config.use_memory_efficient_attention
+        if use_memory_efficient_attention:
+            if attn_implementation != 'eager':
+                logger.warning_once(f"Override {attn_implementation=} to 'eager' as {use_memory_efficient_attention=}")
+                attn_implementation = 'eager'  # Since it will be SDPA by default for torch>=2.1.1
+        self.attention = NEW_ATTENTION_CLASSES[attn_implementation](
+            config, pack_qkv=pack_qkv, use_memory_efficient_attention=use_memory_efficient_attention
+        )
+        self.mlp = NewGatedMLP(config)
+
+        ln_class = LAYER_NORM[config.layer_norm_type]
+        self.attn_ln = ln_class(config.hidden_size, eps=config.layer_norm_eps)
+        self.mlp_ln = ln_class(config.hidden_size, eps=config.layer_norm_eps)
+
+        if config.hidden_dropout_prob > 0:
+            self.hidden_dropout = nn.Dropout(config.hidden_dropout_prob)
+        else:
+            self.hidden_dropout = None
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_bias: torch.FloatTensor,
+        rope_embeds: Optional[Tuple[torch.FloatTensor, torch.FloatTensor]] = None,
+        padding_inputs: Optional[Tuple] = None,  # indices, batch, seqlen
+        attention_scale: Optional[torch.FloatTensor] = None,
+        subset_indices: Optional[torch.LongTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        output_attentions: Optional[bool] = False,
+        qkv_inputs: Optional[Tuple] = None,  # For RetroMAE
+    ) -> Tuple[torch.Tensor, ...]:
+        # Multi head self attention
+        residual = hidden_states if qkv_inputs is None else qkv_inputs[0]
+        attention_outputs = self.attention(
+            hidden_states,
+            attention_bias,
+            rope_embeds,
+            padding_inputs,
+            attention_scale,
+            head_mask,
+            output_attentions=output_attentions,
+            qkv_inputs=qkv_inputs,
+        )
+        hidden_states = attention_outputs[0]
+        if self.hidden_dropout is not None:
+            hidden_states = self.hidden_dropout(hidden_states)
+        hidden_states = residual + hidden_states
+
+        # In pretraining, after the attention of last layer, we only need the masked tokens.
+        if subset_indices is not None:
+            hidden_states = hidden_states[subset_indices]
+
+        hidden_states = self.attn_ln(hidden_states)
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.mlp(hidden_states)
+        if self.hidden_dropout is not None:
+            hidden_states = self.hidden_dropout(hidden_states)
+        hidden_states = residual + hidden_states
+        hidden_states = self.mlp_ln(hidden_states)
+
+        # add self attentions if we output attention weights
+        outputs = (hidden_states,) + attention_outputs[1:]
+        return outputs
+
+
+class NewEncoder(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.layer = nn.ModuleList([NewLayer(config) for _ in range(config.num_hidden_layers)])
+        self.gradient_checkpointing = False
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_bias: Optional[torch.FloatTensor] = None,
+        rope_embeds: Optional[Tuple[torch.FloatTensor, torch.FloatTensor]] = None,
+        padding_inputs: Optional[Tuple] = None,  # indices, batch, seqlen
+        attention_scale: Optional[torch.FloatTensor] = None,
+        subset_indices: Optional[torch.LongTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        output_attentions: Optional[bool] = False,
+        output_hidden_states: Optional[bool] = False,
+        return_dict: Optional[bool] = True,
+    ) -> Union[Tuple[torch.Tensor], BaseModelOutput]:
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attentions = () if output_attentions else None
+
+        for i, layer_module in enumerate(self.layer):
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (hidden_states,)
+
+            if i >= len(self.layer) - 1:
+                layer_subset_indices = subset_indices
+            else:
+                layer_subset_indices = None
+
+            layer_head_mask = head_mask[i] if head_mask is not None else None
+
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = self._gradient_checkpointing_func(
+                    layer_module.__call__,
+                    hidden_states,
+                    attention_bias,
+                    rope_embeds,
+                    padding_inputs,
+                    attention_scale,
+                    layer_subset_indices,
+                    layer_head_mask,
+                )
+            else:
+                layer_outputs = layer_module(
+                    hidden_states,
+                    attention_bias,
+                    rope_embeds,
+                    padding_inputs,
+                    attention_scale,
+                    layer_subset_indices,
+                    layer_head_mask,
+                    output_attentions,
+                )
+
+            hidden_states = layer_outputs[0]
+            if output_attentions:
+                all_self_attentions = all_self_attentions + (layer_outputs[1],)
+
+        if output_hidden_states:
+            all_hidden_states = all_hidden_states + (hidden_states,)
+
+        if not return_dict:
+            return tuple(
+                v
+                for v in [
+                    hidden_states,
+                    all_hidden_states,
+                    all_self_attentions,
+                ]
+                if v is not None
+            )
+        return BaseModelOutput(
+            last_hidden_state=hidden_states,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attentions,
+        )
+
+
+# Copied from transformers.models.bert.modeling_bert.BertPooler with Bert->New
+class NewPooler(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+        self.activation = nn.Tanh()
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        # We "pool" the model by simply taking the hidden state corresponding
+        # to the first token.
+        first_token_tensor = hidden_states[:, 0]
+        pooled_output = self.dense(first_token_tensor)
+        pooled_output = self.activation(pooled_output)
+        return pooled_output
+
+
+class NewPreTrainedModel(PreTrainedModel):
+    """
+    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
+    models.
+    """
+
+    config_class = NewConfig
+    base_model_prefix = "new"
+    supports_gradient_checkpointing = True
+    _supports_sdpa = True
+
+    def _init_weights(self, module):
+        """Initialize the weights"""
+        if isinstance(module, nn.Linear):
+            # Slightly different from the TF version which uses truncated_normal for initialization
+            # cf https://github.com/pytorch/pytorch/pull/5617
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+        elif isinstance(module, nn.LayerNorm):
+            module.bias.data.zero_()
+            module.weight.data.fill_(1.0)
+
+
+class NewModel(NewPreTrainedModel):
+    """
+    The bare New Model transformer outputting raw hidden-states without any specific head on top.
+    """
+
+    def __init__(self, config: NewConfig, add_pooling_layer=False):
+        super().__init__(config)
+        self.config = config
+
+        self.embeddings = NewEmbeddings(config)
+        self.encoder = NewEncoder(config)
+
+        self.pooler = NewPooler(config) if add_pooling_layer else None
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embeddings.word_embeddings
+
+    def set_input_embeddings(self, value):
+        self.embeddings.word_embeddings = value
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        length: Optional[List[int]] = None,
+        subset_indices: Optional[torch.LongTensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], BaseModelOutputWithPooling]:
+        r"""
+        length  (`list` of length `batch_size`, *optional*):
+            If is `None`, return padded `last_hidden_state`.
+        subset_indices  ():
+            pass
+        unpad_inputs  (`bool`, *optional*):
+            pass
+        """
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        unpad_inputs = unpad_inputs if unpad_inputs is not None else self.config.unpad_inputs
+        output_padded = length is None
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
+        elif input_ids is not None:
+            self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
+            input_shape = input_ids.size()
+        elif inputs_embeds is not None:
+            input_shape = inputs_embeds.size()[:-1]
+        else:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        # TODO: not used
+        # # Prepare head mask if needed
+        # # 1.0 in head_mask indicate we keep the head
+        # # attention_probs has shape bsz x n_heads x N x N
+        # # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
+        # # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
+        # head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
+
+        # Get embeddings, may unpad them
+        (embedding_output, attention_mask, rope_embeds, length) = self.embeddings(
+            unpad_inputs,
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            length=length,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds
+        )
+
+        batch_size, seq_length = input_shape
+        if unpad_inputs and self.config.use_memory_efficient_attention:
+            attention_bias = xops.fmha.attn_bias.BlockDiagonalMask.from_seqlens(length)
+        else:
+            # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
+            # ourselves in which case we just need to make it broadcastable to all heads.
+            attention_bias = self.get_extended_attention_mask(attention_mask, input_shape)
+            if self.config.use_memory_efficient_attention:
+                # Invalid shape for attention bias: torch.Size([48, 1, 1, 512]) (expected (48, 12, 512, 512))
+                attention_bias = attention_bias.expand(-1, self.config.num_attention_heads, seq_length, -1)
+
+        padding_inputs = None
+        if unpad_inputs and (output_padded or not self.config.use_memory_efficient_attention):
+            indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
+            if not self.config.use_memory_efficient_attention:
+                padding_inputs = (indices, *input_shape)
+
+        attention_scale = None
+        if self.config.logn_attention_scale:
+            logger.warning_once("TODO: logn_attention_scale")
+        #     # attention scale log_512(input_len)
+        #     attention_scale = attention_mask.sum(1).log() / torch.tensor(self.config.max_position_embeddings).log()
+        #     # inference-time logn scale need clip 1
+        #     if self.config.logn_attention_clip1:
+        #         attention_scale.clip_(1)
+        #     attention_scale = attention_scale[:, None, None, None]
+        # else:
+        #     attention_scale = None
+
+        encoder_outputs = self.encoder(
+            embedding_output,
+            attention_bias=attention_bias,
+            rope_embeds=rope_embeds,
+            padding_inputs=padding_inputs,
+            attention_scale=attention_scale,
+            subset_indices=subset_indices,
+            head_mask=head_mask,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        sequence_output = encoder_outputs[0]
+        if unpad_inputs and output_padded:
+            sequence_output = pad_input(
+                sequence_output.squeeze(), indices, batch_size, seq_length
+            )
+
+        pooled_output = self.pooler(sequence_output) if self.pooler is not None else None
+
+        if not return_dict:
+            return (sequence_output, pooled_output) + encoder_outputs[1:]
+
+        return BaseModelOutputWithPooling(
+            last_hidden_state=sequence_output,
+            pooler_output=pooled_output,
+            hidden_states=encoder_outputs.hidden_states,
+            attentions=encoder_outputs.attentions,
+        )
+
+
+class NewLMPredictionHead(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+        self.transform_act_fn = ACT2FN[config.hidden_act]
+        self.norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+
+        # The output weights are the same as the input embeddings, but there is
+        # an output-only bias for each token.
+        self.decoder = nn.Linear(config.hidden_size, config.vocab_size)
+
+    def forward(self, hidden_states):
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.transform_act_fn(hidden_states)
+        hidden_states = self.norm(hidden_states)
+        hidden_states = self.decoder(hidden_states)
+        return hidden_states
+
+
+class NewForMaskedLM(NewPreTrainedModel):
+    _tied_weights_keys = ["lm_head.decoder.bias", "lm_head.decoder.weight"]
+
+    def __init__(self, config: NewConfig):
+        super().__init__(config)
+        self.new = NewModel(config, add_pooling_layer=False)
+        self.lm_head = NewLMPredictionHead(config)
+        self.loss_fct = nn.CrossEntropyLoss()
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_output_embeddings(self):
+        return self.lm_head.decoder
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head.decoder = new_embeddings
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], MaskedLMOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
+            config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
+            loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
+        """
+
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if labels is None or not self.new.config.unpad_inputs:
+            length = None
+            subset_indices = None
+        else:
+            length = attention_mask.sum(-1).tolist()
+            labels = labels[attention_mask.bool()].unsqueeze(0)
+            subset_indices = labels > -100
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            length=length,
+            subset_indices=subset_indices,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        sequence_output = outputs[0]
+        prediction_scores = self.lm_head(sequence_output)
+
+        masked_lm_loss = None
+        if labels is not None:
+            if subset_indices is None:
+                mask = attention_mask.bool()
+                prediction_scores = prediction_scores[mask]
+                labels = labels[mask]
+            else:
+                labels = labels[subset_indices]
+            masked_lm_loss = self.loss_fct(prediction_scores, labels)
+
+        if not return_dict:
+            output = (prediction_scores,) + outputs[2:]
+            return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output
+
+        return MaskedLMOutput(
+            loss=masked_lm_loss,
+            logits=prediction_scores,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+class NewForSequenceClassification(NewPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+        self.config = config
+
+        self.new = NewModel(config, add_pooling_layer=True)
+        classifier_dropout = (
+            config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
+        )
+        self.dropout = nn.Dropout(classifier_dropout)
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], SequenceClassifierOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
+            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
+            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        pooled_output = outputs[1]
+
+        pooled_output = self.dropout(pooled_output)
+        logits = self.classifier(pooled_output)
+
+        loss = None
+        if labels is not None:
+            if self.config.problem_type is None:
+                if self.num_labels == 1:
+                    self.config.problem_type = "regression"
+                elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
+                    self.config.problem_type = "single_label_classification"
+                else:
+                    self.config.problem_type = "multi_label_classification"
+
+            if self.config.problem_type == "regression":
+                loss_fct = nn.MSELoss()
+                if self.num_labels == 1:
+                    loss = loss_fct(logits.squeeze(), labels.squeeze())
+                else:
+                    loss = loss_fct(logits, labels)
+            elif self.config.problem_type == "single_label_classification":
+                loss_fct = nn.CrossEntropyLoss()
+                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+            elif self.config.problem_type == "multi_label_classification":
+                loss_fct = nn.BCEWithLogitsLoss()
+                loss = loss_fct(logits, labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return SequenceClassifierOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+class NewForMultipleChoice(NewPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.new = NewModel(config, add_pooling_layer=True)
+        classifier_dropout = (
+            config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
+        )
+        self.dropout = nn.Dropout(classifier_dropout)
+        self.classifier = nn.Linear(config.hidden_size, 1)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], MultipleChoiceModelOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the multiple choice classification loss. Indices should be in `[0, ...,
+            num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See
+            `input_ids` above)
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        num_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1]
+
+        input_ids = input_ids.view(-1, input_ids.size(-1)) if input_ids is not None else None
+        attention_mask = attention_mask.view(-1, attention_mask.size(-1)) if attention_mask is not None else None
+        token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1)) if token_type_ids is not None else None
+        position_ids = position_ids.view(-1, position_ids.size(-1)) if position_ids is not None else None
+        inputs_embeds = (
+            inputs_embeds.view(-1, inputs_embeds.size(-2), inputs_embeds.size(-1))
+            if inputs_embeds is not None
+            else None
+        )
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        pooled_output = outputs[1]
+
+        pooled_output = self.dropout(pooled_output)
+        logits = self.classifier(pooled_output)
+        reshaped_logits = logits.view(-1, num_choices)
+
+        loss = None
+        if labels is not None:
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(reshaped_logits, labels)
+
+        if not return_dict:
+            output = (reshaped_logits,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return MultipleChoiceModelOutput(
+            loss=loss,
+            logits=reshaped_logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+@dataclass
+class NewTokenClassifierOutput(ModelOutput):
+    loss: Optional[torch.FloatTensor] = None
+    logits: torch.FloatTensor = None
+    last_hidden_state: torch.FloatTensor = None
+    hidden_states: Optional[Tuple[torch.FloatTensor, ...]] = None
+    attentions: Optional[Tuple[torch.FloatTensor, ...]] = None
+
+
+class NewForTokenClassification(NewPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+
+        self.new = NewModel(config, add_pooling_layer=False)
+        classifier_dropout = (
+            config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
+        )
+        self.dropout = nn.Dropout(classifier_dropout)
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], NewTokenClassifierOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        sequence_output = outputs[0]
+
+        sequence_output = self.dropout(sequence_output)
+        logits = self.classifier(sequence_output)
+
+        loss = None
+        if labels is not None:
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+
+        if not return_dict:
+            output = (logits,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return NewTokenClassifierOutput(
+            loss=loss,
+            logits=logits,
+            last_hidden_state=sequence_output,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+class NewForQuestionAnswering(NewPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+
+        self.new = NewModel(config, add_pooling_layer=False)
+        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        start_positions: Optional[torch.Tensor] = None,
+        end_positions: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], QuestionAnsweringModelOutput]:
+        r"""
+        start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for position (index) of the start of the labelled span for computing the token classification loss.
+            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
+            are not taken into account for computing the loss.
+        end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for position (index) of the end of the labelled span for computing the token classification loss.
+            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
+            are not taken into account for computing the loss.
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        sequence_output = outputs[0]
+
+        logits = self.qa_outputs(sequence_output)
+        start_logits, end_logits = logits.split(1, dim=-1)
+        start_logits = start_logits.squeeze(-1).contiguous()
+        end_logits = end_logits.squeeze(-1).contiguous()
+
+        total_loss = None
+        if start_positions is not None and end_positions is not None:
+            # If we are on multi-GPU, split add a dimension
+            if len(start_positions.size()) > 1:
+                start_positions = start_positions.squeeze(-1)
+            if len(end_positions.size()) > 1:
+                end_positions = end_positions.squeeze(-1)
+            # sometimes the start/end positions are outside our model inputs, we ignore these terms
+            ignored_index = start_logits.size(1)
+            start_positions = start_positions.clamp(0, ignored_index)
+            end_positions = end_positions.clamp(0, ignored_index)
+
+            loss_fct = nn.CrossEntropyLoss(ignore_index=ignored_index)
+            start_loss = loss_fct(start_logits, start_positions)
+            end_loss = loss_fct(end_logits, end_positions)
+            total_loss = (start_loss + end_loss) / 2
+
+        if not return_dict:
+            output = (start_logits, end_logits) + outputs[2:]
+            return ((total_loss,) + output) if total_loss is not None else output
+
+        return QuestionAnsweringModelOutput(
+            loss=total_loss,
+            start_logits=start_logits,
+            end_logits=end_logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 8192,
+    "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa7a6ad87a7ce8fe196787355f6af7d03aee94d19c54a5eb1392ed18c8ef451a
+size 17082988

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_max_length": 8192,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "XLMRobertaTokenizerFast",
+  "unk_token": "<unk>"
+}