Microwave regeneration and thermal and oxidative stability of imidazolium cyanopyrrolide ionic liquid for direct air capture of carbon dioxide.

Abstract: Understanding the oxidative and thermal degradation of CO2 sorbents is essential for assessing long-term sorbent stability in direct air capture (DAC). The potential degradation pathway of imidazolium cyanopyrrolide, an ionic liquid (IL) functionalized for superior CO2 capacity and selectivity, was evaluated under accelerated degradation conditions to understand the secondary reactions that can occur during repetitive absorption-desorption thermal-swing cycles. The combined analysis from various spectroscopy, chromatography, and thermal gravimetric measurements indicated that radicalic and SN2 mechanisms in degradation are encouraged by the nucleophilicity of the anion. Thickening of the liquid and gas evolution are accompanied by 50% reduction in CO2 capacity after a 7-day exposure to O2 under 80 °C. In order to prevent long exposure to conventional thermal heating, we demonstrate microwave (MW) regeneration of the CO2-reactive IL where dielectric heating at 80 and 100 °C rapidly desorbs CO2 and regenerates the IL without any measurable degradation.


Advancements and Challenges in Direct Air Capture Technologies: Energy Intensity, Novel Methods, Economics, and Location Strategies

Abstract: Direct air capture (DAC) technology is increasingly recognized as a key tool in the pursuit of climate neutrality, enabling the removal of carbon dioxide directly from the atmosphere. Despite its potential, DAC remains in the early stages of development, with most installations limited to pilot or demonstration units. The main barriers to its widespread implementation include high energy demands and significant capture costs. This literature review addresses the most critical research directions related to the development of this technology, focusing on its challenges and prospects for deployment. Particular attention is given to studies aimed at developing new, cost-effective, and efficient sorbents that could significantly reduce the energy intensity and costs of the process. Alternative technologies, such as electrochemical and membrane-based processes, show promise but require further research to overcome limitations, such as sensitivity to oxygen presence or insufficient membrane selectivity. The economic feasibility of DAC remains uncertain, with current estimates subject to significant uncertainty. Governmental and regulatory support will be crucial for the technology’s success. Furthermore, the location of DAC installations should consider factors such as energy availability, options for carbon dioxide storage or utilization, and climatic conditions, which significantly affect process efficiency. This review highlights the necessity for continued research to overcome existing barriers and fully harness the potential of DAC technology.


The role of direct air capture in achieving climate-neutral aviation

Abstract: Growing demand for air travel and limited scalable solutions pose significant challenges to the mitigation of aviation’s climate change impact. Direct air capture (DAC) may gain prominence due to its versatile applications for either carbon removal (direct air carbon capture and storage, DACCS) or synthetic fuel production (direct air carbon capture and utilization, DACCU). Through a comprehensive and time-dynamic techno-economic assessment, we explore the conditions for synthetic fuels from DACCU to become cost-competitive with an emit-and-remove strategy based on DACCS under 2050 CO2 and climate neutrality targets. We find that synthetic fuels could achieve climate neutrality at lower cost than an emit-and-remove strategy due to their ability to cost-effectively mitigate contrails. Under demand reductions, contrail avoidance, and CO2 neutrality targets the cost advantage of synthetic fuels weakens or disappears. Low electricity cost (€0.02 kWh-1) and high fossil kerosene prices (€0.9 l-1) can favor synthetic fuels’ cost-competitiveness even under these conditions. Strategic interventions, such as optimal siting and the elimination of fossil fuel subsidies, can thus favor a shift away from fossil-reliant aviation. This study examines Direct Air Capture’s role in climate-neutral aviation via synthetic fuels or carbon removal. It finds Direct Air Capture-based fuels can cost-effectively mitigate CO₂ and non-CO₂ effects if cheap renewable energy is used.


Carbonation of MgO Single Crystals: Implications for Direct Air Capture of CO2.

Abstract: Direct air capture (DAC) may be feasible to remove carbon dioxide (CO2) from the atmosphere at the gigaton scale, holding promise to become a major contributor to climate change mitigation. Mineral looping using magnesium oxide (MgO) is potentially an economical, efficient, and sustainable pathway to gigaton-scale DAC. The hydroxylation and carbonation of MgO determine the efficiency of the looping process, but their rates and mechanisms remain uncertain. In this work, MgO single crystals were reacted in air or CO2 at varying humidities and characterized by X-ray scattering, microscopy, and vibrational spectroscopy. Results show that the hydroxylation formed a brucite (Mg(OH)2)-like layer immediately after crystal cleaving. Concurrently, the carbonation formed hydrated magnesium carbonate phases, including barringtonite (MgCO3·2H2O) and nesquehonite (MgCO3·2H2O), in the layer. Rapid initial growth of the layer is also manifested in short-range bending/warping of nanocrystallites, resulting in multiple orientations of the same phases on the surface. The layer growth slowed down over time, indicating surface passivation. The formation of barringtonite and nesquehonite with 1:1 CO3/Mg ratio indicates an efficient carbonation when compared to other magnesium carbonate phases of lower ratio. Our results are essential for understanding surface passivation mechanisms and tackling the passivation issue of mineral looping DAC technology.


Perspective on distributed direct air capture: what, why, and how?

Abstract: Direct air capture (DAC) is widely considered as a critical negative emission technology to not only mitigate but reverse global climate change. While commercially expanding, its efficiency is limited by energy-intensive sorbent regeneration. Here, we highlight distributed DAC as a complement to centralized systems, analyzing the regeneration energy demands and carbon footprints of various sorbents. A comprehensive evaluation of distributed DAC’s impact is crucial for maximizing its potential.


Research needs targeting direct air capture of carbon dioxide: Material & process performance characteristics under realistic environmental conditions


Environmental, economic and social trade-offs of membrane-based direct air capture technologies integrated with CO2 conversion using life cycle assessment

Abstract: Technological solutions to address climate change are coalescing around renewable energy deployment. Yet, the deployment of renewables alone may not be sufficient to restrict temperature increases in line with Paris Agreement targets. The emergence of carbon negative technologies to bridge this shortfall is needed and expected to overcome this gap. Among these technologies, direct air capture (DAC) is being deployed at multiple scales using various approaches. This study investigates membrane-based DAC integrated with subsequent carbon dioxide to carbon fuel conversion, i.e. , DAC with utilization (DAC-U). The DAC-U evaluation is undertaken holistically, beginning with a cradle-to-grave life cycle assessment, followed by economic feasibility scenario analysis and social acceptability analysis to establish acceptable deployment pricing and necessary policy interventions. This study reveals that, although the DAC-U represents a carbon negative capable technology with positive lifestyle and environmental outcomes, high capital costs present a significant barrier to deployment. To overcome this barrier, a robust policy regime including subsidies or fuel credits may be necessary. Further technological innovation and efficiency gains will also close this gap, meaning that the membrane-based DAC-U concept may play a role in achieving carbon neutrality goals in the near future.


Economically viable geological CO2 storage from direct air capture has critical threshold of 70% CO2 concentration

Abstract: Direct Air Capture (DAC), a key component of Carbon Capture and Storage (CCS), has been widely studied. However, its large-scale deployment is hindered by the high energy cost of purifying captured CO2. Using impure CO2 can reduce energy consumption and overall costs, but it also lowers storage efficiency. This work employs molecular dynamics simulations to examine storage efficiency by analyzing the impurity systems’ density across a wide temperature and pressure range. The results indicate a strong similarity between the density changes at the macroscopic level and the Van der Waals interaction changes at the molecular level. Additionally, the Normalized Storage Efficiency caused by Impurities (NSEI) is proposed, which can be used for storage potential and cost evaluation. A detailed NSEI analysis suggests that CO2 concentration should reach at least 70% to achieve economically viable storage. This finding provides practical guidance for DAC capture system design and impurity CCS project planning. Le Zhang and colleagues propose that a 70% concentration should be met for economically viable geological storage of carbon dioxide from Direct Air Capture (DAC). This finding provides practical guidance for DAC system design and impurity carbon capture and storage project planning.


Integrated direct air CO2 capture and utilization via in-situ catalytic conversion to fuels and chemicals using dual functional materials: Recent progresses and perspectives


Carbon Capture from Natural Gas Flue Emissions and Air via (Bi)Carbonate Formation in a Cyclodextrin-Based Metal-Organic Framework.

Abstract: Carbon capture and utilization or sequestration (CCUS) from industrial point sources and direct air capture (DAC) are both necessary to curb the rising atmospheric levels of CO2. Amine scrubbers, the current leading carbon capture technology, suffer from poor oxidative and thermal stability, limiting their long-term cycling stability under oxygen-rich streams such as air and the emissions from natural gas combined cycle (NGCC) power plants. Herein, we demonstrate that the hydroxide-based cyclodextrin metal-organic framework (CD-MOF) Rb2CO3 CD-MOF ST possesses high CO2 capacities from dry dilute streams at low temperatures and humid streams at elevated temperatures. Additionally, it displays good thermal, oxidative, and cycling stabilities and selective CO2 capture under mixed gas conditions in dynamic breakthrough experiments. Unexpectedly, under dry, hot conditions, a shift in the CO2 adsorption mechanism─from reversibly formed bicarbonate to irreversibly formed carbonate─is observed, as supported by gas sorption and spectroscopic studies coupled with density functional theory (DFT) calculations. This mechanistic switch, akin to urea formation in amine-functionalized sorbents, has not been previously reported in a hydroxide-based material and sheds new light on the interplay between bicarbonate and carbonate species during CO2 capture. Our findings provide valuable insight for the design of next-generation materials containing oxygen-based nucleophiles for carbon capture applications.

Journal of the American Chemical Society


Neo-Carbon Food concept: A pilot-scale hybrid biological–inorganic system with direct air capture of carbon dioxide

Abstract: Abstract The pace at which the human population is growing raises serious concerns related to food security while at the same time conventional agriculture-based food production is becoming a major cause of environmental pollution and greenhouse gas emissions. Numerous solutions have been proposed to boost food production among which edible microbial biomass is considered a promising alternative to conventional sources of food and feed with lower environmental footprint. This work introduces the Neo-Carbon Food concept that is a pilot-scale hybrid biological–inorganic process suitable for the production of microbial biomass. The concept includes integrated hydrogen production by water electrolysis, direct air capture (DAC) of carbon dioxide, and its subsequent assimilation by autotrophic hydrogen-oxidizing bacteria (HOB). The hydrogen production with in situ electrolysis achieved specific energy consumption just below 100 kWh/kgH2 while the specific energy consumption of DAC was around 20 kWh/kgCO2.


Cost Analysis of Direct Air Capture and Sequestration Coupled to Low-Carbon Thermal Energy in the U.S.

Abstract: Negative emissions technologies will play an important role in preventing 2°C warming by 2100. The next decade is critical for technological innovation and deployment to meet mid-century carbon removal goals of 10 - 20 GtCO2 /yr. Direct air capture (DAC) is positioned to play a critical role in carbon removal, yet remains under paced in deployment efforts, mainly due to high costs. This study outlines a roadmap for DAC cost reductions through the exploitation of low-temperature heat, recent US policy drivers, and logical, regional end-use opportunities in the US. Specifically, two scenarios are identified that allow for the production of compressed high-purity CO2 for costs  $300/tCO2, net delivered with an opportunity to scale to 19 MtCO2/yr. These scenarios use thermal energy from geothermal and nuclear power plants to produce steam and transport the purified CO2 via trucks to the nearest opportunity for direct use or subsurface permanent storage. While some utilization pathways result in the re-emission of CO2 and cannot be considered true carbon removal, they would provide economic incentive to deploying DAC plants at scale by mid-century. In addition, the federal tax credit 45Q was applied for qualifying facilities, (i.e., producing  100 ktCO2/yr).


Carbon Purchase Agreements, Dactories, and Supply-Chain Innovation: What Will It Take to Scale-Up Modular Direct Air Capture Technology to a Gigatonne Scale

Abstract: Natural and engineered carbon dioxide removal have become regular features of climate models which limit warming to 1.5°C or even 2°C above pre-industrial levels. This gives rise to an assumption that solutions, for example direct air capture (DAC)—involving the direct removal of carbon dioxide from ambient air—can be commercialised and deployed at the necessary speed and scale to have a material impact, in the order of gigatonnes, by mid-century. Modular, solid-sorbent DAC on a gigatonne scale will require the mass mobilisation of supply chains to manufacture millions of modular DAC units−20 million of the present state of the art 50 tonne/year modules to deliver 1 gigatonne per year, as well as the large-scale production of novel chemical sorbents. To achieve a climate relevant DAC industry will demand innovative procurement models, for example carbon purchase agreements (CPAs), and dedicated DAC manufacturing facilities or dactories. In addition, insight is offered through the work of DAC start-up Carbon Infinity into the industry supply-chain position, adopting lessons from computing, and energy technologies. In particular, we look at approaches to drive demand and scale-up DAC module production, and opportunities presented in the development of an integrated DAC manufacturing industry.


Membrane Separation Technology in Direct Air Capture

Abstract: Direct air capture (DAC) is an emerging negative CO2 emission technology that aims to introduce a feasible method for CO2 capture from the atmosphere. Unlike carbon capture from point sources, which deals with flue gas at high CO2 concentrations, carbon capture directly from the atmosphere has proved difficult due to the low CO2 concentration in ambient air. Current DAC technologies mainly consider sorbent-based systems; however, membrane technology can be considered a promising DAC approach since it provides several advantages, e.g., lower energy and operational costs, less environmental footprint, and more potential for small-scale ubiquitous installations. Several recent advancements in validating the feasibility of highly permeable gas separation membrane fabrication and system design show that membrane-based direct air capture (m-DAC) could be a complementary approach to sorbent-based DAC, e.g., as part of a hybrid system design that incorporates other DAC technologies (e.g., solvent or sorbent-based DAC). In this article, the ongoing research and DAC application attempts via membrane separation have been reviewed. The reported membrane materials that could potentially be used for m-DAC are summarized. In addition, the future direction of m-DAC development is discussed, which could provide perspective and encourage new researchers’ further work in the field of m-DAC.


Solar-Powered Direct Air Capture: Techno-Economic and Environmental Assessment

Abstract: Direct air capture (DAC) of CO2 has gained attention as a sustainable carbon source. One of the most promising technologies currently available is liquid solvent DAC (L-DAC), but the significant fraction of fossil CO2 in the output stream hinders its utilization in carbon-neutral fuels and chemicals. Fossil CO2 is generated and captured during the combustion of fuels to calcine carbonates, which is difficult to decarbonize due to the high temperatures required. Solar thermal energy can provide green high-temperature heat, but it flourishes in arid regions where environmental conditions are typically unfavorable for L-DAC. This study proposes a solar-powered L-DAC approach and develops a model to assess the influence of the location and plant capacity on capture costs. The performed life cycle assessment enables the comparison of technologies based on net CO2 removal, demonstrating that solar-powered L-DAC is not only more environmentally friendly but also more cost-effective than conventional L-DAC.


How Low-Carbon Heat Requirements for Direct Air Capture of CO2 Can Enable the Expansion of Firm Low-Carbon Electricity Generation Resources

Abstract: A rapid build-out of direct air capture (DAC), deployed in order to mitigate climate change, will require significant amounts of both low-carbon thermal and electrical energy. Firm low-carbon power resources, including nuclear, geothermal, or natural gas with carbon capture, which also will become more highly valued as variable renewable energy penetration increases, would be able to provide both heat and electricity for DAC. In this study, we examined the techno-economic synergy between a hypothetical DAC plant in the year 2030 and a nuclear small modular reactor, and determined two avenues for which this relationship could benefit the nuclear plant. First, we demonstrated that, under certain assumptions, selling a portion of its energy to a DAC facility allows the nuclear plant to take in 21% less revenue from selling electricity to wholesale markets than its projected levelized cost, and still break even. Second, after estimating a potential revenue stream, we showed that an integration with DAC allows for the nuclear plant's capital costs to be up to 35% higher than what would be required if only selling electricity to wholesale markets. This could enable the nuclear plant to operate economically even in the face of variable and decreasing wholesale electricity prices, and also could offer developers more financial certainty when planning a new project. Ultimately, this study shows that the need for low-carbon energy for DAC plants might incentivize the development of advanced nuclear plants and firm low-carbon resources more broadly.


Polymer Sorbent Design for the Direct Air Capture of CO2

Abstract: Anthropogenic activities have resulted in enormous increases in atmospheric CO2 concentrations particularly since the onset of the Industrial Revolution, which have potential links with increased global temperatures, rising sea levels, increased prevalence, and severity of natural disasters, among other consequences. To enable a carbon-neutral and sustainable society, various technologies have been developed for CO2 capture from industrial process streams as well as directly from air. Here, direct air capture (DAC) represents an essential need for reducing CO2 concentration in the atmosphere to mitigate the negative consequences of greenhouse effects, involving systems that can reversibly adsorb and release CO2, in which polymers have played an integral role. This work provides insights into the development of polymer sorbents for DAC of CO2, specifically from the perspective of material design principles. We discuss how physical properties and chemical identities of amine-containing polymers can impact their ability to uptake CO2, as well as be efficiently regenerated. Additionally, polymers which use ionic interactions to react with CO2 molecules, such as poly(ionic liquids), are also common DAC sorbent materials. Finally, a perspective is provided on the future research and technology opportunities of developing polymer-derived sorbents for DAC.


Assessing Impacts of Atmospheric Conditions on Efficiency and Siting of Large-Scale Direct Air Capture Facilities

Abstract: The cost and efficiency of direct air capture (DAC) of carbon dioxide (CO2) will be decisive in determining whether this technology can play a large role in decarbonization. To probe the role of meteorological conditions on DAC we examine, at 1 × 1° resolution for the continental United States (U.S.), the impacts of temperature, humidity, atmospheric pressure, and CO2 concentration for a representative amine-based adsorption process. Spatial and temporal variations in atmospheric pressure and CO2 concentration lead to strong variations in the CO2 available in ambient air across the U.S. The specific DAC process that we examine is described by a process model that accounts for both temperature and humidity. A process that assumes the same operating choices at all locations in the continental U.S. shows strong variations in performance, with the most influential variables being the H2O gas phase volume fraction and temperature, both of which are negatively correlated with DAC productivity for the specific process that we consider. The process also shows a moderate positive correlation of ambient CO2 with productivity and recovery. We show that optimizing the DAC process at seven representative locations to reflect temporal and spatial variations in ambient conditions significantly improves the process performance and, more importantly, would lead to different choices in the sites for the best performance than models based on a single set of process conditions. Our work provides a framework for assessing spatial variations in DAC performance that could be applied to any DAC process and indicates that these variations will have important implications in optimizing and siting DAC facilities.


Assessment of the Potential of Electrochemical Steps in Direct Air Capture through Techno-Economic Analysis

Abstract: Direct air capture (DAC) technologies are proposed to reduce the atmospheric CO2 concentration to mitigate climate change and simultaneously provide carbon as a feedstock independent of fossil resources. The currently high energy demand and cost of DAC technologies are challenging and could limit the significance of DAC processes. The present work estimates the potential energy demand and the levelized cost of capture (LCOC) of liquid solvent absorption and solid adsorption DAC processes in the long term. A consistent framework is applied to compare nonelectrochemical to electrochemical DAC processes and estimate the LCOC depending on the electricity price. We determine the equivalent cell voltage needed for the electrochemical steps to achieve comparable or lower energy demand than nonelectrochemical processes. The capital expenses (CapEx) of the electrochemical steps are estimated using analogies to processes that are similar in function. The results are calculated for a range of initial data of CapEx and energy demand to include uncertainties in the data.


Integrating direct air capture with small modular nuclear reactors: understanding performance, cost, and potential

Abstract: Direct air capture (DAC) is a key component in the transition to net-zero society. However, its giga-tonne deployment faces daunting challenges in terms of availability of both financial resources and, most of all, large quantities of low-carbon energy. Within this context, small modular nuclear reactors (SMRs) might potentially facilitate the deployment of DAC. In the present study, we present a detailed thermodynamic analysis of integrating an SMR with solid sorbent DAC. We propose different integration designs and find that coupling the SMR with DAC significantly increases the use of thermal energy produced in the nuclear reactor: from 32% in a stand-alone SMR to 76%–85% in the SMR-DAC system. Moreover, we find that a 50–MW SMR module equipped with DAC could remove around 0.3 MtCO2 every year, while still producing electricity at 24%–42% of the rated power output. Performing a techno-economic analysis of the system, we estimate a net removal cost of around 250 €/tCO2. When benchmarking it to other low-carbon energy supply solutions, we find that the SMR-DAC system is potentially more cost-effective than a DAC powered by high-temperature heat pumps or dedicated geothermal systems. Finally, we evaluate the potential of future deployment of SMR-DAC in China, Europe, India, South Africa and the USA, finding that it could enable up to around 96 MtCO2/year by 2035 if SMRs prove to be cost-competitive. The impact of regional differences on the removal cost is also assessed.


Probing Structural Transformations and Degradation Mechanisms by Direct Observation in SIFSIX-3-Ni for Direct Air Capture.

Abstract: Despite global efforts to reduce carbon dioxide (CO2) emissions, continued industrialization threatens to exacerbate climate change. This work investigates methods to capture CO2, with a focus on the SIFSIX-3-Ni metal-organic framework (MOF) as a direct air capture (DAC) sorbent. SIFSIX-3-Ni exhibits promising CO2 adsorption properties but suffers from degradation processes under accelerated aging, which are akin to column regeneration conditions. Herein, we have grown the largest SIFSIX-3-Ni single crystals to date, facilitating single crystal X-ray diffraction analyses that enabled direct observation of the H2O and CO2 dynamics through adsorption and desorption. In addition, a novel space group (I4/mcm) for the SIFSIX-3-Ni is identified, which provided insights into structural transitions within the framework and elucidated water's role in degrading CO2 uptake performance as the material ages. In situ X-ray scattering methods revealed long-range and local structural transformations associated with CO2 adsorption in the framework pores as well as a temperature-dependent desorption mechanism. Pair distribution function analysis revealed a partial decomposition to form nonporous single-layer nanosheets of edge-sharing nickel oxide octahedra upon aging. The formation of these nanosheets is irreversible and reduces the amount of active material for the CO2 sorption. These findings provide crucial insights for the development of efficient and stable DAC sorbents, effectively reducing greenhouse gases, and suggest avenues for enhancing MOF stability under practical DAC conditions.


Carbon dioxide direct air capture for effective climate change mitigation based on renewable electricity: a new type of energy system sector coupling

Abstract: Pathways for achieving the 1.5–2 °C global temperature moderation target imply a massive scaling of carbon dioxide (CO 2 ) removal technologies, in particular in the 2040s and onwards. CO 2 direct air capture (DAC) is among the most promising negative emission technologies (NETs). The energy demands for low-temperature solid-sorbent DAC are mainly heat at around 100 °C and electricity, which lead to sustainably operated DAC systems based on low-cost renewable electricity and heat pumps for the heat supply. This analysis is carried out for the case of the Maghreb region, which enjoys abundantly available low-cost renewable energy resources. The energy transition results for the Maghreb region lead to a solar photovoltaic (PV)-dominated energy supply with some wind energy contribution. DAC systems will need the same energy supply structure. The research investigates the levelised cost of CO 2 DAC (LCOD) in high spatial resolution and is based on full hourly modelling for the Maghreb region. The key results are LCOD of about 55 €/t CO2 in 2050 with a further cost reduction potential of up to 50%. The area demand is considered and concluded to be negligible. Major conclusions for CO 2 removal as a new energy sector are drawn. Key options for a global climate change mitigation strategy are first an energy transition towards renewable energy and second NETs for achieving the targets of the Paris Agreement.

Mitigation and Adaptation Strategies for Global Change


Life cycle carbon efficiency of Direct Air Capture systems with strong hydroxide sorbents

Abstract: Abstract Direct Air Capture (DAC) of carbon dioxide (CO2) from ambient air has the potential to combat climate change. DAC systems capture CO2 using a sorbent material and compress it for storage. In this study, we calculated the life cycle carbon efficiency (Ec) of a DAC system which equals the net amount of carbon stored per amount of carbon captured from capture to geological storage. We included greenhouse gas (GHG) emissions during construction of the necessary facilities as well as GHG emissions from energy, water and chemicals needed during operations. The system we analysed includes a hydroxide solution as sorbent material and utilizes the pelletized variant of the Kraft process to regenerate the sorbent and separate the CO2. To our knowledge, the potential climate benefit over the full life cycle of this DAC system has not been fully investigated up to now. Using the baseline scenario, we obtained a positive Ec of 62%. For the optimistic and pessimistic scenarios, we found a Ec of 93% and 10%, respectively. We conclude that this type of DAC system may be a feasible option to help keep limit global temperature increases to well below 2 °C.

International Journal of Greenhouse Gas Control


Facile Synthesis of Self-supported Solid Amine Sorbents for Direct Air Capture.

Abstract: Conventional usage of tetraethylenepentamine (TEPA) via being supported on porous solid materials for carbon capture is susceptible to oxidative degradation during regeneration cycles. This study reports a novel method to synthesize a TEPA based solid polymer for efficient CO2 removal via direct air capture (DAC). The polymer was obtained through epoxy-amine crosslinking reaction, leading to the transformation of liquid TEPA to a self-supported solid polymer. The synthesis was conducted under ambient conditions via a one-pot process with no waste products, which is aligned with green synthesis. The performance of the solid amine was evaluated in DAC under realistic conditions and compared with TEPA supported on SiO2 and zeolite 13X prepared through the conventional method. The solid TEPA amine exhibited a high CO2 uptake of 6.2 wt.% comparable to the conventional counterparts. More importantly, the solid TEPA amine demonstrated high resistance to oxidation during the accelerated ageing process at 80 °C in air for 24 h, whereas the two supported TEPA samples experienced severe degradation, with zeolite 13X supported TEPA incurring a reduction of 86.5% in CO2 capturing capacity after the ageing. This work sheds light on the novel usage of TEPA as an efficient solid amine for practical DAC operation.


Scalable and Highly Porous Membrane Adsorbents for Direct Air Capture of CO2.

Abstract: Direct air capture (DAC) of CO2 is a carbon-negative technology to mitigate carbon emissions, and it requires low-cost sorbents with high CO2 sorption capacity that can be easily manufactured on a large scale. In this work, we develop highly porous membrane adsorbents comprising branched polyethylenimine (PEI) impregnated in low-cost, porous Solupor supports. The effect of the PEI molecular mass and loading on the physical properties of the adsorbents is evaluated, including porosity, degradation temperature, glass transition temperature, and CO2 permeance. CO2 capture from simulated air containing 400 ppm of CO2 in these sorbents is thoroughly investigated as a function of temperature and relative humidity (RH). Polymer dynamics was examined using differential scanning calorimetry (DSC) and broadband dielectric spectroscopy (BDS), showing that CO2 sorption is limited by its diffusion in these PEI-based sorbents. A membrane adsorbent containing 48 mass% PEI (800 Da) with a porosity of 72% exhibits a CO2 sorption capacity of 1.2 mmol/g at 25 °C and RH of 30%, comparable to the state-of-the-art adsorbents. Multicycles of sorption and desorption were performed to determine their regenerability, stability, and potential for practical applications.


The performance of solvent-based direct air capture across geospatial and temporal climate regimes

Abstract: Liquid-solvent direct air capture (DAC) is a prominent approach for carbon dioxide removal but knowing where to site these systems is challenging because it requires considering a multitude of interrelated geospatial factors. Two of the most pressing factors are: (1) how should DAC be powered to provide the greatest net removal of CO2 and (2) how does weather impact its performance?.To investigate these questions, this study develops a process-level model of a liquid-solvent DAC system and couples it to a 20-year dataset of temperature and humidity conditions at a ~9km resolution across the contiguous US.We find that the amount of CO2 sequestered could be 30% to 50% greater than the amount of CO2 removed from the atmosphere if natural gas is burned on site to power DAC, but that the optimal way to power DAC is independent of capture rate (i.e., weather), depending solely on the upstream GHG intensity of electricity and natural gas. Regardless of how it is powered, air temperature and humidity conditions can change the performance of DAC by up to ~3x and can also vary substantially across weather years. Across the continuous US, we find that southern states (e.g., Gulf Coast) are preferrable locations for a variety of reasons, including higher and less variable air temperature and relative humidity. Lastly, we also find the performance of liquid-solvent DAC calculated with monthly means is within 2% of the estimated performance calculated with hourly data for more than a third of the country, including in the states with weather most favorable for liquid-solvent DAC.


Physicochemical Properties, Equilibrium Adsorption Performance, Manufacturability, and Stability of TIFSIX-3-Ni for Direct Air Capture of CO2

Abstract: The use of adsorbents for direct air capture (DAC) of CO2 is regarded as a promising and essential carbon dioxide removal technology to help meet the goals outlined by the 2015 Paris Agreement. A class of adsorbents that has gained significant attention for this application is ultramicroporous metal organic frameworks (MOFs). However, the necessary data needed to facilitate process scale evaluation of these materials is not currently available. Here, we investigate TIFSIX-3-Ni, a previously reported ultramicroporous MOF for DAC, and measure several physicochemical and equilibrium adsorption properties. We report its crystal structure, textural properties, thermal stability, specific heat capacity, CO2, N2, and H2O equilibrium adsorption isotherms at multiple temperatures, and Ar and O2 isotherms at a single temperature. For CO2, N2, and H2O, we also report isotherm model fitting parameters and calculate heats of adsorption. We assess the manufacturability and process stability of TIFSIX-3-Ni by investigating the impact of batch reproducibility, binderless pelletization, humidity, and adsorption–desorption cycling (50 cycles) on its crystal structure, textural properties, and CO2 adsorption. For pelletized TIFSIX-3-Ni, we also report its skeletal, pellet, and bed density, total pore volume, and pellet porosity. Overall, our data enable initial process modeling and optimization studies to evaluate TIFSIX-3-Ni for DAC at the process scale. They also highlight the possibility to pelletize TIFSIX-3-Ni and the limited stability of the MOF under humid and oxidative conditions as well as upon multiple adsorption–desorption cycles.


Optimization of a combined power plant CO2 capture and direct air capture concept for flexible power plant operation

Abstract: Deployment of carbon capture and storage (CCS) equipped fossil fuel power plants on the supply-side and direct air capture (DAC) technologies on the demand side can address the dual challenge...


How Would Ideal Sorbents Improve the Technical and Economic Performance of Adsorption-Based Direct Air Capture?

Abstract: : Current climate policies and national pledges are insufficient to achieve the necessary reduction in CO 2 emissions, underscoring the urgency of carbon dioxide removal (CDR) from the atmosphere. Direct air capture (DAC) is an engineered CDR process with substantial removal potential but also one with clear technical limits and opportunities for enhancement. One such improvement is the optimization of the material, enabling the separation. To investigate this from a quantitative perspective, this study evaluates DAC performance when replacing existing sorbents with theoretical ones designed to optimize the process. In order to do so, the adsorption isotherm parameters of CO 2 for different isotherm models were optimized along with process design variables to minimize energy and maximize productivity. Combining equilibrium and rate-based models, our analysis offers insights from thermodynamic, reactor, and economic perspectives. The results show that optimal sorbents have the potential to significantly improve the DAC performance obtained with existing sorbents, for example, Lewatit VP-OC-1065 and MIL-101, especially in terms of energy consumption and costs. This requires that some critical sorbent characteristics are met: moderate cost (e.g., <30$/kg), long lifetime (e.g., >2 years), and a linear driving force in line with other gas separation processes (e.g., k LDF > 0.001 s − 1 ). Furthermore, our findings highlight that optimizing productivity and energy consumption enables the identification of minimum-cost configurations for DAC. This work underscores the importance of tailored sorbent design in advancing DAC toward cost-effective, efficient CO 2 removal solutions.


Combining LLM Code Generation with Formal Specifications and Reactive Program Synthesis

Abstract: In the past few years, Large Language Models (LLMs) have exploded in usefulness and popularity for code generation tasks. However, LLMs still struggle with accuracy and are unsuitable for high-risk applications without additional oversight and verification. In particular, they perform poorly at generating code for highly complex systems, especially with unusual or out-of-sample logic. For such systems, verifying the code generated by the LLM may take longer than writing it by hand. We introduce a solution that divides the code generation into two parts; one to be handled by an LLM and one to be handled by formal methods-based program synthesis. We develop a benchmark to test our solution and show that our method allows the pipeline to solve problems previously intractable for LLM code generation.


Unveiling the Power of Neural Program Synthesis in High Level Code Generation

Abstract: In this research, we delve into a transformative exploration of the realm of code generation, unveiling three fundamental modules that constitute the backbone of our study: the Text-to-Code Generation Model, Code Autocompletion Model, and the formidable Neural Program Synthesis Model. At the forefront of our investigation, we underscore the paramount importance of Neural Program Synthesis—a dynamic force adept at crafting custom programs from input-output examples. Our exploration delves into the intricate landscape of natural language-to-code generation, offering profound insights and invaluable contributions to the field. The ability to automatically generate code from natural language instructions holds immense significance in streamlining software development processes, reducing manual effort, and enhancing productivity. Furthermore, this research addresses a pressing need within the software engineering community for efficient methods of code generation, particularly in domains where human-program interaction is prevalent. Through rigorous experimentation and meticulous analysis, we shed light on methodologies and promising avenues for future research in this burgeoning domain, paving the way for advancements in automated software development and programming assistance tools.

2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT)


Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

Abstract: Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.


MapCoder: Multi-Agent Code Generation for Competitive Problem Solving

Abstract: Code synthesis, which requires a deep understanding of complex natural language problem descriptions, generation of code instructions for complex algorithms and data structures, and the successful execution of comprehensive unit tests, presents a significant challenge. While large language models (LLMs) demonstrate impressive proficiency in natural language processing, their performance in code generation tasks remains limited. In this paper, we introduce a new approach to code generation tasks leveraging multi-agent prompting that uniquely replicates the full cycle of program synthesis as observed in human developers. Our framework, MapCoder, consists of four LLM agents specifically designed to emulate the stages of this cycle: recalling relevant examples, planning, code generation, and debugging. After conducting thorough experiments, with multiple LLM ablations and analyses across eight challenging competitive problem-solving and program synthesis benchmarks, MapCoder showcases remarkable code generation capabilities, achieving new state-of-the-art results (pass@1) on HumanEval (93.9%), MBPP (83.1%), APPS (22.0%), CodeContests (28.5%), and xCodeEval (45.3%). Moreover, our method consistently delivers superior performance across various programming languages and varying problem difficulties. We open-source our framework at https://github.com/Md-Ashraful-Pramanik/MapCoder.

Annual Meeting of the Association for Computational Linguistics


AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing

Abstract: Recent advancements in automatic code generation using large language models (LLMs) have brought us closer to fully automated secure software development. However, existing approaches often rely on a single agent for code generation, which struggles to produce secure, vulnerability-free code. Traditional program synthesis with LLMs has primarily focused on functional correctness, often neglecting critical dynamic security implications that happen during runtime. To address these challenges, we propose AutoSafeCoder, a multi-agent framework that leverages LLM-driven agents for code generation, vulnerability analysis, and security enhancement through continuous collaboration. The framework consists of three agents: a Coding Agent responsible for code generation, a Static Analyzer Agent identifying vulnerabilities, and a Fuzzing Agent performing dynamic testing using a mutation-based fuzzing approach to detect runtime errors. Our contribution focuses on ensuring the safety of multi-agent code generation by integrating dynamic and static testing in an iterative process during code generation by LLM that improves security. Experiments using the SecurityEval dataset demonstrate a 13% reduction in code vulnerabilities compared to baseline LLMs, with no compromise in functionality.


Guiding Enumerative Program Synthesis with Large Language Models

Abstract: Pre-trained Large Language Models (LLMs) are beginning to dominate the discourse around automatic code generation with natural language specifications. In contrast, the best-performing synthesizers in the domain of formal synthesis with precise logical specifications are still based on enumerative algorithms. In this paper, we evaluate the abilities of LLMs to solve formal synthesis benchmarks by carefully crafting a library of prompts for the domain. When one-shot synthesis fails, we propose a novel enumerative synthesis algorithm, which integrates calls to an LLM into a weighted probabilistic search. This allows the synthesizer to provide the LLM with information about the progress of the enumerator, and the LLM to provide the enumerator with syntactic guidance in an iterative loop. We evaluate our techniques on benchmarks from the Syntax-Guided Synthesis (SyGuS) competition. We find that GPT-3.5 as a stand-alone tool for formal synthesis is easily outperformed by state-of-the-art formal synthesis algorithms, but our approach integrating the LLM into an enumerative synthesis algorithm shows significant performance gains over both the LLM and the enumerative synthesizer alone and the winning SyGuS competition tool.

International Conference on Computer Aided Verification


A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

Abstract: Pre-trained large language models (LLMs) have recently achieved better generalization and sample efficiency in autonomous web automation. However, the performance on real-world websites has still suffered from (1) open domainness, (2) limited context length, and (3) lack of inductive bias on HTML. We introduce WebAgent, an LLM-driven agent that learns from self-experience to complete tasks on real websites following natural language instructions. WebAgent plans ahead by decomposing instructions into canonical sub-instructions, summarizes long HTML documents into task-relevant snippets, and acts on websites via Python programs generated from those. We design WebAgent with Flan-U-PaLM, for grounded code generation, and HTML-T5, new pre-trained LLMs for long HTML documents using local and global attention mechanisms and a mixture of long-span denoising objectives, for planning and summarization. We empirically demonstrate that our modular recipe improves the success on real websites by over 50%, and that HTML-T5 is the best model to solve various HTML understanding tasks; achieving 18.7% higher success rate than the prior method on MiniWoB web automation benchmark, and SoTA performance on Mind2Web, an offline task planning evaluation.

International Conference on Learning Representations


A Systematic Literature Review of 10 years of Research on Program Synthesis and Natural Language Processing


Execution-based Code Generation using Deep Reinforcement Learning

Abstract: The utilization of programming language (PL) models, pre-trained on large-scale code corpora, as a means of automating software engineering processes has demonstrated considerable potential in streamlining various code generation tasks such as code completion, code translation, and program synthesis. However, current approaches mainly rely on supervised fine-tuning objectives borrowed from text generation, neglecting unique sequence-level characteristics of code, including but not limited to compilability as well as syntactic and functional correctness. To address this limitation, we propose PPOCoder, a new framework for code generation that synergistically combines pre-trained PL models with Proximal Policy Optimization (PPO) which is a widely used deep reinforcement learning technique. By utilizing non-differentiable feedback from code execution and structure alignment, PPOCoder seamlessly integrates external code-specific knowledge into the model optimization process. It's important to note that PPOCoder is a task-agnostic and model-agnostic framework that can be used across different code generation tasks and PLs. Extensive experiments on three code generation tasks demonstrate the effectiveness of our proposed approach compared to SOTA methods, achieving significant improvements in compilation success rates and functional correctness across different PLs.


E-code: Mastering Efficient Code Generation through Pretrained Models and Expert Encoder Group

Abstract: Context: With the waning of Moore's Law, the software industry is placing increasing importance on finding alternative solutions for continuous performance enhancement. The significance and research results of software performance optimization have been on the rise in recent years, especially with the advancement propelled by Large Language Models(LLMs). However, traditional strategies for rectifying performance flaws have shown significant limitations at the competitive code efficiency optimization level, and research on this topic is surprisingly scarce. Objective: This study aims to address the research gap in this domain, offering practical solutions to the various challenges encountered. Specifically, we have overcome the constraints of traditional performance error rectification strategies and developed a Language Model (LM) tailored for the competitive code efficiency optimization realm. Method: We introduced E-code, an advanced program synthesis LM. Inspired by the recent success of expert LMs, we designed an innovative structure called the Expert Encoder Group. This structure employs multiple expert encoders to extract features tailored for different input types. We assessed the performance of E-code against other leading models on a competitive dataset and conducted in-depth ablation experiments. Results: Upon systematic evaluation, E-code achieved a 54.98% improvement in code efficiency, significantly outperforming other advanced models. In the ablation experiments, we further validated the significance of the expert encoder group and other components within E-code. Conclusion: The research findings indicate that the expert encoder group can effectively handle various inputs in efficiency optimization tasks, significantly enhancing the model's performance.


Neuro-Symbolic Program Synthesis for Multi-Hop Natural Language Navigation

Abstract: Solving navigation problems from natural language descriptions is essential for advancing humanrobot interaction and enhancing the usability of autonomous systems. Symbolic approaches to path planning excel in well defined environments but cannot cope with the ambiguity of natural language inputs. On the other hand, neural solutions centered on large language models (LLMs) can parse free-form natural language but lack the reasoning capabilities for solving complex multihop path planning problems. In this paper, we propose a neuro-symbolic framework based on program synthesis for multi-hop natural language navigation called NSPS. The framework uses an LLM to parse the problem definition in natural language, a graph of the environment, and an API of a graph library. Next, the code generation capabilities of the LLM are used to synthesize a program for path planning and verification. The path planning program is executed to generate a solution path that is checked by the verification program. A selfcorrection loop is used to fix both syntax and value errors. The framework is evaluated using 600 multihop navigation tasks with 1 to 10 hops. Compared with neural approaches, the NSPS framework improves the success rate and path efficiency by an average of 64.3% and 19.4% across all tasks, respectively.

International Conference on Applied Algorithms

2024 International Conference on Assured Autonomy (ICAA)


Diffusion On Syntax Trees For Program Synthesis

Abstract: Large language models generate code one token at a time. Their autoregressive generation process lacks the feedback of observing the program's output. Training LLMs to suggest edits directly can be challenging due to the scarcity of rich edit data. To address these problems, we propose neural diffusion models that operate on syntax trees of any context-free grammar. Similar to image diffusion models, our method also inverts ``noise'' applied to syntax trees. Rather than generating code sequentially, we iteratively edit it while preserving syntactic validity, which makes it easy to combine this neural model with search. We apply our approach to inverse graphics tasks, where our model learns to convert images into programs that produce those images. Combined with search, our model is able to write graphics programs, see the execution result, and debug them to meet the required specifications. We additionally show how our system can write graphics programs for hand-drawn sketches.


Improving Code Generation by Training with Natural Language Feedback

Abstract: The potential for pre-trained large language models (LLMs) to use natural language feedback at inference time has been an exciting recent development. We build upon this observation by formalizing an algorithm for learning from natural language feedback at training time instead, which we call Imitation learning from Language Feedback (ILF). ILF requires only a small amount of human-written feedback during training and does not require the same feedback at test time, making it both user-friendly and sample-efficient. We further show that ILF can be seen as a form of minimizing the KL divergence to the ground truth distribution and demonstrate a proof-of-concept on a neural program synthesis task. We use ILF to improve a Codegen-Mono 6.1B model's pass@1 rate by 38% relative (and 10% absolute) on the Mostly Basic Python Problems (MBPP) benchmark, outperforming both fine-tuning on MBPP and fine-tuning on repaired programs written by humans. Overall, our results suggest that learning from human-written natural language feedback is both more effective and sample-efficient than training exclusively on demonstrations for improving an LLM's performance on code generation tasks.


Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation

Abstract: For a complicated algorithm, its implementation by a human programmer usually starts with outlining a rough control flow followed by iterative enrichments, eventually yielding carefully generated syntactic structures and variables in a hierarchy. However, state-of-the-art large language models generate codes in a single pass, without intermediate warm-ups to reflect the structured thought process of"outline-then-detail". Inspired by the recent success of chain-of-thought prompting, we propose ChainCoder, a program synthesis language model that generates Python code progressively, i.e. from coarse to fine in multiple passes. We first decompose source code into layout frame components and accessory components via abstract syntax tree parsing to construct a hierarchical representation. We then reform our prediction target into a multi-pass objective, each pass generates a subsequence, which is concatenated in the hierarchy. Finally, a tailored transformer architecture is leveraged to jointly encode the natural language descriptions and syntactically aligned I/O data samples. Extensive evaluations show that ChainCoder outperforms state-of-the-arts, demonstrating that our progressive generation eases the reasoning procedure and guides the language model to generate higher-quality solutions. Our codes are available at: https://github.com/VITA-Group/ChainCoder.

International Conference on Machine Learning


B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis

Abstract: Program synthesis aims to create accurate, executable programs from problem specifications, specifically from natural language descriptions in our context. Recent studies have leveraged the power of reinforcement learning (RL) in conjunction with large language models (LLMs), significantly enhancing code generation capabilities. The application of RL focuses on directly optimizing for functional correctness, offering an advantage over conventional supervised methods. Despite policy-based RL methods dominating the literature on RL for program synthesis, the nature of program synthesis tasks hints at a natural alignment with value-based methods. This stems from the rich collection of off-policy programs, including those developed by human programmers and also historical samples, coupled with the straightforward verification of generated programs through automated unit testing, meaning rewards are easy to obtain. Diverging from the dominant use of policy-based algorithms, our work explores the feasibility of value-based approaches, leading to the development of our $\mathcal{B}$-Coder (pronounced Bellman coder). Yet, training value-based methods presents challenges due to the enormous search space inherent to program synthesis. To this end, we introduce an initialization protocol for RL agents utilizing pre-trained LMs and a conservative Bellman operator to reduce training complexities. Moreover, we demonstrate how to leverage the learned value functions as a dual strategy to post-process generated programs. Our empirical evaluations demonstrated $\mathcal{B}$-Coder's capability in achieving state-of-the-art performance when compared to policy-based methods. Remarkably, this achievement is reached with minimal reward engineering effort, highlighting the effectiveness of value-based RL, independent of reward designs.


Knowledge-Driven Robot Program Synthesis from Human VR Demonstrations

Abstract: Aging societies, labor shortages and increasing wage costs call for assistance robots capable of autonomously performing a wide array of real-world tasks. Such open-ended robotic manipulation requires not only powerful knowledge representations and reasoning (KR&R) algorithms, but also methods for humans to instruct robots what tasks to perform and how to perform them. In this paper, we present a system for automatically generating executable robot control programs from human task demonstrations in virtual reality (VR). We leverage common-sense knowledge and game engine-based physics to semantically interpret human VR demonstrations, as well as an expressive and general task representation and automatic path planning and code generation, embedded into a state-of-the-art cognitive architecture. We demonstrate our approach in the context of force-sensitive fetch-and-place for a robotic shopping assistant. The source code is available at https://github.com/ease-crc/vr-program-synthesis.

International Conference on Principles of Knowledge Representation and Reasoning


Program Synthesis for Cyber-Resilience

Abstract: Architectural tactics enable stakeholders to achieve cyber-resilience requirements. They permit systems to react, resist, detect, and recover from cyber incidents. This paper presents an approach to generate source code for architectural tactics typically used in safety and mission-critical systems. Our approach extensively relies on the use of the Event-B formal method and the EventB2Java code generation plugin of the Rodin platform. It leverages the modeling of architectural tactics in the Event-B formal language and uses a set of EventB2Java transformation rules to generate certified code implementations for the said tactics. Since resilience requirements are statements about a system over time, and because of the fact that the Event-B language does not provide (native) support for the writing of temporal specifications, we have implemented a novel Linear Temporal Logic (LTL) extension for Event-B. We support several architectural tactics for availability, performance, and security. The generated code is certified in the following sense: discharging proof obligations in Rodin - the platform we use for writing the Event-B models - attests to the soundness of the architectural tactics modelled in Event-B, and the soundness of the translation encoded by the EventB2Java tool attests to the code correctness. Finally, we demonstrate the usability of our resilience validation approach with the aid of an Autonomous Vehicle System. It further helped us increase our confidence in the soundness of our Event-B LTL extension.

IEEE Transactions on Software Engineering


Function-constrained Program Synthesis

Abstract: This work introduces (1) a technique that allows large language models (LLMs) to leverage user-provided code when solving programming tasks and (2) a method to iteratively generate modular sub-functions that can aid future code generation attempts when the initial code generated by the LLM is inadequate. Generating computer programs in general-purpose programming languages like Python poses a challenge for LLMs when instructed to use code provided in the prompt. Code-specific LLMs (e.g., GitHub Copilot, CodeLlama2) can generate code completions in real-time by drawing on all code available in a development environment. However, restricting code-specific LLMs to use only in-context code is not straightforward, as the model is not explicitly instructed to use the user-provided code and users cannot highlight precisely which snippets of code the model should incorporate into its context. Moreover, current systems lack effective recovery methods, forcing users to iteratively re-prompt the model with modified prompts until a sufficient solution is reached. Our method differs from traditional LLM-powered code-generation by constraining code-generation to an explicit function set and enabling recovery from failed attempts through automatically generated sub-functions. When the LLM cannot produce working code, we generate modular sub-functions to aid subsequent attempts at generating functional code. A by-product of our method is a library of reusable sub-functions that can solve related tasks, imitating a software team where efficiency scales with experience. We also introduce a new"half-shot"evaluation paradigm that provides tighter estimates of LLMs' coding abilities compared to traditional zero-shot evaluation. Our proposed evaluation method encourages models to output solutions in a structured format, decreasing syntax errors that can be mistaken for poor coding ability.


LogoMotion: Visually-Grounded Code Synthesis for Creating and Editing Animation

Abstract: Creating animation takes time, effort, and technical expertise. To help novices with animation, we present LogoMotion, an AI code generation approach that helps users create semantically meaningful animation for logos. LogoMotion automatically generates animation code with a method called visually-grounded code synthesis and program repair. This method performs visual analysis, instantiates a design concept, and conducts visual checking to generate animation code. LogoMotion provides novices with code-connected AI editing widgets that help them edit the motion, grouping, and timing of their animation. In a comparison study on 276 animations, LogoMotion was found to produce more content-aware animation than an industry-leading tool. In a user evaluation (n=16) comparing against a prompt-only baseline, these code-connected widgets helped users edit animations with control, iteration, and creative expression.

International Conference on Human Factors in Computing Systems

Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems


Automated C/C++ Program Repair for High-Level Synthesis via Large Language Models

Abstract: In High-Level Synthesis (HLS), converting a regular C/C++ program into its HLS-compatible counterpart (HLS-C) still requires tremendous manual effort. Various program scripts have been introduced to automate this process. But the resulting codes usually contain many issues that should be manually repaired by developers. Since Large Language Models (LLMs) have the ability to automate code generation, they can also be used for automated program repair in HLS. However, due to the limited training of LLMs considering hardware and software simultaneously, hallucinations may occur during program repair using LLMs, leading to compilation failures. Besides, using LLMs for iterative repair also incurs a high cost. To address these challenges, we propose an LLM-driven program repair framework that takes regular $\mathrm{C} / \mathrm{C}++$ code as input and automatically generates its corresponding HLS-C code for synthesis while minimizing human repair effort. To mitigate the hallucinations in LLMs and enhance the prompt quality, a Retrieval-Augmented Generation (RAG) paradigm is introduced to guide the LLMs toward correct repair. In addition, we use LLMs to create a static bit width optimization program to identify the optimized bit widths for variables. Moreover, LLM-driven HLS optimization strategies are introduced to add/tune pragmas in HLS-C programs for circuit optimization. Experimental results demonstrate that the proposed LLM-driven automated framework can achieve much higher repair pass rates in 24 real-world applications compared with the traditional scripts and the direct application of LLMs for program repair. The codes are open-sourced at this link: https://github.com/code-source1/catapult.

2024 ACM/IEEE 6th Symposium on Machine Learning for CAD (MLCAD)


Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

Abstract: Software engineers mainly write code by editing existing programs. In contrast, language models (LMs) autoregressively synthesize programs in a single pass. One explanation for this is the scarcity of sequential edit data. While high-quality instruction data for code synthesis is scarce, edit data for synthesis is even scarcer. To fill this gap, we develop a synthetic data generation algorithm called LintSeq. This algorithm refactors programs into sequences of synthetic edits by using a linter to procedurally sample across interdependent lines of source code. Synthetic edits sampled with LintSeq reflect the syntax and semantics of their programming language. To test the algorithm, we use it to refactor a dataset of instruction + program pairs into instruction + program-diff-sequence tuples. Then, we fine-tune a series of smaller LMs ranging from 2.6B to 14B parameters on both the re-factored and original versions of this dataset. We perform comprehensive evaluations comparing edit sequence code LMs against baselines on HumanEval, MBPP(+), CodeContests, DS-1000, and BigCodeBench. We show that models fine-tuned to iteratively synthesize code match or outperform baselines on pass@1, and exhibit better scaling across higher pass@k as a function of total test-time FLOPs. Finally, we also pretrain our own tiny LMs for code understanding. We show that fine-tuning these models to synthesize code edit-by-edit results in strong performance on HumanEval and MBPP(+) compared to existing code language models of similar scale such as CodeT5+, AlphaCode, and Codex.


Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming

Abstract: While there has been a lot of research recently on robots in household environments, at the present time, most robots in existence can be found on shop floors, and most interactions between humans and robots happen there. ``Collaborative robots''(cobots) designed to work alongside humans on assembly lines traditionally require expert programming, limiting ability to make changes, or manual guidance, limiting expressivity of the resulting programs. To address these limitations, we explore using Large Language Models (LLMs), and in particular, their abilities of doing in-context learning, for conversational code generation. As a first step, we define RATS, the ``Repetitive Assembly Task'', a 2D building task designed to lay the foundation for simulating industry assembly scenarios. In this task, a `programmer'instructs a cobot, using natural language, on how a certain assembly is to be built; that is, the programmer induces a program, through natural language. We create a dataset that pairs target structures with various example instructions (human-authored, template-based, and model-generated) and example code. With this, we systematically evaluate the capabilities of state-of-the-art LLMs for synthesising this kind of code, given in-context examples. Evaluating in a simulated environment, we find that LLMs are capable of generating accurate `first order code'(instruction sequences), but have problems producing `higher-order code'(abstractions such as functions, or use of loops).


Learning to Reason via Program Generation, Emulation, and Search

Abstract: Program synthesis with language models (LMs) has unlocked a large set of reasoning abilities; code-tuned LMs have proven adept at generating programs that solve a wide variety of algorithmic symbolic manipulation tasks (e.g. word concatenation). However, not all reasoning tasks are easily expressible as code, e.g. tasks involving commonsense reasoning, moral decision-making, and sarcasm understanding. Our goal is to extend an LM's program synthesis skills to such tasks and evaluate the results via pseudo-programs, namely Python programs where some leaf function calls are left undefined. To that end, we propose, Code Generation and Emulated EXecution (CoGEX). CoGEX works by (1) training LMs to generate pseudo-programs, (2) teaching them to emulate their generated program's execution, including those leaf functions, allowing the LM's knowledge to fill in the execution gaps; and (3) using them to search over many programs to find an optimal one. To adapt the CoGEX model to a new task, we introduce a method for performing program search to find a single program whose pseudo-execution yields optimal performance when applied to all the instances of a given dataset. We show that our approach yields large improvements compared to standard in-context learning approaches on a battery of tasks, both algorithmic and soft reasoning. This result thus demonstrates that code synthesis can be applied to a much broader class of problems than previously considered. Our released dataset, fine-tuned models, and implementation can be found at \url{https://github.com/nweir127/CoGEX}.


Reinforcement Learning and Data-Generation for Syntax-Guided Synthesis

Abstract: Program synthesis is the task of automatically generating code based on a specification. In Syntax-Guided Synthesis (SyGuS) this specification is a combination of a syntactic template and a logical formula, and the result is guaranteed to satisfy both. We present a reinforcement-learning guided algorithm for SyGuS which uses Monte-Carlo Tree Search (MCTS) to search the space of candidate solutions. Our algorithm learns policy and value functions which, combined with the upper confidence bound for trees, allow it to balance exploration and exploitation. A common challenge in applying machine learning approaches to syntax-guided synthesis is the scarcity of training data. To address this, we present a method for automatically generating training data for SyGuS based on anti-unification of existing first-order satisfiability problems, which we use to train our MCTS policy. We implement and evaluate this setup and demonstrate that learned policy and value improve the synthesis performance over a baseline by over 26 percentage points in the training and testing sets. Our tool outperforms state-of-the-art tool cvc5 on the training set and performs comparably in terms of the total number of problems solved on the testing set (solving 23% of the benchmarks on which cvc5 fails). We make our data set publicly available, to enable further application of machine learning methods to the SyGuS problem.

AAAI Conference on Artificial Intelligence


From Code to Play: Benchmarking Program Search for Games Using Large Language Models

Abstract: Large language models (LLMs) have shown impressive capabilities in generating program code, opening exciting opportunities for applying program synthesis to games. In this work, we explore the potential of LLMs to directly synthesize usable code for a wide range of gaming applications, focusing on two programming languages, Python and Java. We use an evolutionary hill-climbing algorithm, where the mutations and seeds of the initial programs are controlled by LLMs. For Python, the framework covers various game-related tasks, including five miniature versions of Atari games, ten levels of Baba is You, an environment inspired by Asteroids, and a maze generation task. For Java, the framework contains 12 games from the TAG tabletop games framework. Across 29 tasks, we evaluated 12 language models for Python and 8 for Java. Our findings suggest that the performance of LLMs depends more on the task than on model size. While larger models generate more executable programs, these do not always result in higher-quality solutions but are much more expensive. No model has a clear advantage, although on any specific task, one model may be better. Trying many models on a problem and using the best results across them is more reliable than using just one.


Do Programs Dream of Electromagnetic Signals? Towards GAN-based Code-to-Signal Synthesis

Abstract: The analysis of EM signals has become a popular approach as a means for achieving non-intrusive and external anomaly detection, particularly in the realm of embedded devices. Previous efforts have shown huge potential in terms of discriminative precision between those signals that correspond to normal versus anomalous operations. However, virtually all research in the area neglects the challenge of gathering high-quality signals for the purpose of fingerprinting all possible execution paths of a program. Today, such efforts are mainly human-driven and because there may be hundreds of alternative paths even in simple programs, EM-based anomaly detection approaches exhibit poor scalability. In this work, we introduce a set of methods for the generation of EM signals directly from code. To the best of our knowledge, this is the first effort that aims to address code-to-signal synthesis for purposes of side-channel analysis. The proposed framework is based on Generative Adversarial Networks (GAN) that receive the corresponding assembly (ASM) code of a program as input and in turn produce artificial signals which bear near-identical morphological characteristics as the corresponding real EM signals. Although the proposed method pertains across a wide range of applications we have evaluated our approach specifically for the task of training anomaly detection models. We experimentally showed that synthetic signals can train models that can then detect even stealthy code injection attacks i.e., very challenging minimal code alterations with great accuracy. More specifically, synthetically trained anomaly detection models achieve a near-perfect AUC score of 0.993 for the detection of four malicious instructions and 0.981 for the detection of two malicious instructions. Interestingly, applying rudimentary feature engineering i.e., considering the maximum amplitude of the signal per cycle as opposed to the entire raw signal leads to perfect detection accuracy.

MILCOM 2023 - 2023 IEEE Military Communications Conference (MILCOM)


Learning-Based Automatic Synthesis of Software Code and Configuration

Abstract: Increasing demands in software industry and scarcity of software engineers motivates researchers and practitioners to automate the process of software generation and configuration. Large scale automatic software generation and configuration is a very complex and challenging task. In this proposal, we set out to investigate this problem by breaking down automatic software generation and configuration into two different tasks. In first task, we propose to synthesize software automatically with input output specifications. This task is further broken down into two sub-tasks. The first sub-task is about synthesizing programs with a genetic algorithm which is driven by a neural network based fitness function trained with program traces and specifications. For the second sub-task, we formulate program synthesis as a continuous optimization problem and synthesize programs with covariance matrix adaption evolutionary strategy (a state-of-the-art continuous optimization method). Finally, for the second task, we propose to synthesize configurations of large scale software from different input files (e.g. software manuals, configurations files, online blogs, etc.) using a sequence-to-sequence deep learning mechanism.


The Program Testing Ability of Large Language Models for Code

Abstract: Recent development of large language models (LLMs) for code like CodeX and CodeT5+ shows promise in achieving code intelligence. Their ability of synthesizing program targeting a pre-defined algorithmic coding task has been intensively tested and verified on datasets including HumanEval and MBPP. Yet, evaluation of these LLMs from more perspectives (than just program synthesis) is also anticipated, considering their broad scope of applications. In this paper, we explore their ability of automatic test cases generation. We show intriguing observations and reveal how the quality of their generated test cases can be improved. Following recent work which uses generated test cases to enhance program synthesis, we further leverage our findings in improving the quality of the synthesized programs and show +11.77% and +4.22% higher code pass rates on HumanEval+ comparing with the GPT-3.5-turbo baseline and the recent state-of-the-art, respectively. Our code is publicly available at https://github.com/asdasxzxcq/TestCaseGen.

Conference on Empirical Methods in Natural Language Processing


High-Level Synthesis of the OpenMP runtime to improve the generation of parallel accelerators

Abstract: Field Programmable Gate Arrays (FPGAs) are reconfigurable hardware components that can be programmed using a hardware specification language such as Verilog or VHDL.Writing code to program an FPGA requires a set of different skills than software programming, for example, related to the use of signals and registers instead of variables and the explicit description of parallelism. High-Level Synthesis (HLS) tools can help by automatically translating software specifications into hardware descriptions.

Proceedings of the 20th ACM International Conference on Computing Frontiers


Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search

Abstract: In this work we consider Code World Models, world models generated by a Large Language Model (LLM) in the form of Python code for model-based Reinforcement Learning (RL). Calling code instead of LLMs for planning has potential to be more precise, reliable, interpretable, and extremely efficient. However, writing appropriate Code World Models requires the ability to understand complex instructions, to generate exact code with non-trivial logic and to self-debug a long program with feedback from unit tests and environment trajectories. To address these challenges, we propose Generate, Improve and Fix with Monte Carlo Tree Search (GIF-MCTS), a new code generation strategy for LLMs. To test our approach in an offline RL setting, we introduce the Code World Models Benchmark (CWMB), a suite of program synthesis and planning tasks comprised of 18 diverse RL environments paired with corresponding textual descriptions and curated trajectories. GIF-MCTS surpasses all baselines on the CWMB and two other benchmarks, and we show that the Code World Models synthesized with it can be successfully used for planning, resulting in model-based RL agents with greatly improved sample efficiency and inference speed.


Mitigating Gender Bias in Code Large Language Models via Model Editing

Abstract: In recent years, with the maturation of large language model (LLM) technology and the emergence of high-quality programming code datasets, researchers have become increasingly confident in addressing the challenges of program synthesis automatically. However, since most of the training samples for LLMs are unscreened, it is inevitable that LLMs' performance may not align with real-world scenarios, leading to the presence of social bias. To evaluate and quantify the gender bias in code LLMs, we propose a dataset named CodeGenBias (Gender Bias in the Code Generation) and an evaluation metric called FB-Score (Factual Bias Score) based on the actual gender distribution of correlative professions. With the help of CodeGenBias and FB-Score, we evaluate and analyze the gender bias in eight mainstream Code LLMs. Previous work has demonstrated that model editing methods that perform well in knowledge editing have the potential to mitigate social bias in LLMs. Therefore, we develop a model editing approach named MG-Editing (Multi-Granularity model Editing), which includes the locating and editing phases. Our model editing method MG-Editing can be applied at five different levels of model parameter granularity: full parameters level, layer level, module level, row level, and neuron level. Extensive experiments not only demonstrate that our MG-Editing can effectively mitigate the gender bias in code LLMs while maintaining their general code generation capabilities, but also showcase its excellent generalization. At the same time, the experimental results show that, considering both the gender bias of the model and its general code generation capability, MG-Editing is most effective when applied at the row and neuron levels of granularity.


Constrained Decoding for Fill-in-the-Middle Code Language Models via Efficient Left and Right Quotienting of Context-Sensitive Grammars

Abstract: Large Language Models are powerful tools for program synthesis and advanced auto-completion, but come with no guarantee that their output code is syntactically correct. This paper contributes an incremental parser that allows early rejection of syntactically incorrect code, as well as efficient detection of complete programs for fill-in-the-middle (FIM) tasks. We extend the Earley parsing algorithm to allow for left and right quotients of context-free grammars, and develop methods to handle quotienting of several context-sensitive features present in the grammars of many common programming languages. The result of these contributions is an efficient, general, and well-grounded method for left and right quotient parsing. To validate our theoretical contributions -- and the effectiveness of certain design decisions -- we evaluate our method on the particularly difficult case of FIM completion for Python 3, with syntax-correctness constraints. Our results demonstrate that constrained generation can significantly reduce the incidence of syntax errors in recommended code.


LILO: Learning Interpretable Libraries by Compressing and Documenting Code

Abstract: While large language models (LLMs) now excel at code generation, a key aspect of software development is the art of refactoring: consolidating code into libraries of reusable and readable programs. In this paper, we introduce LILO, a neurosymbolic framework that iteratively synthesizes, compresses, and documents code to build libraries tailored to particular problem domains. LILO combines LLM-guided program synthesis with recent algorithmic advances in automated refactoring from Stitch: a symbolic compression system that efficiently identifies optimal lambda abstractions across large code corpora. To make these abstractions interpretable, we introduce an auto-documentation (AutoDoc) procedure that infers natural language names and docstrings based on contextual examples of usage. In addition to improving human readability, we find that AutoDoc boosts performance by helping LILO's synthesizer to interpret and deploy learned abstractions. We evaluate LILO on three inductive program synthesis benchmarks for string editing, scene reasoning, and graphics composition. Compared to existing neural and symbolic methods - including the state-of-the-art library learning algorithm DreamCoder - LILO solves more complex tasks and learns richer libraries that are grounded in linguistic knowledge.


Generation of Compiler Backends from Formal Models of Hardware

Abstract: Compilers convert between representations -- usually, from higher-level, human writable code to lower-level, machine-readable code. A compiler backend is the portion of the compiler containing optimizations and code generation routines for a specific hardware target. In this dissertation, I advocate for a specific way of building compiler backends: namely, by automatically generating them from explicit, formal models of hardware using automated reasoning algorithms. I describe how automatically generating compilers from formal models of hardware leads to increased optimization ability, stronger correctness guarantees, and reduced development time for compiler backends. As evidence, I present two case studies: first, Glenside, which uses equality saturation to increase the 3LA compiler's ability to offload operations to machine learning accelerators, and second, Lakeroad, a technology mapper for FPGAs which uses program synthesis and semantics extracted from Verilog to map hardware designs to complex, programmable hardware primitives.


Controlling Large Language Models to Generate Secure and Vulnerable Code


UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program Testing

Abstract: The remarkable capability of large language models (LLMs) in generating high-quality code has drawn increasing attention in the software testing community. However, existing code LLMs often demonstrate unsatisfactory capabilities in generating accurate, complete tests since they were trained on code snippets collected without differentiating between code for testing and for other purposes. In this paper, we present a large-scale dataset, UniTSyn, which can enhance LLMs for Unit Test Synthesis. Associating tests with the tested functions is crucial for LLMs to infer the expected behavior and the logic paths to be verified. By leveraging Language Server Protocol, UniTSyn achieves the challenging goal of collecting focal-test pairs without per-project execution setups or per-language heuristics, which tend to be fragile and difficult to scale. Containing 2.7 million focal-test pairs across five mainstream programming languages, it can enhance the test generation ability of LLMs. Our experiments demonstrate that, by building an autoregressive LLM based on UniTSyn, we can achieve significant benefits in learning and understanding unit test representations, resulting in improved generation accuracy and code coverage across all the evaluated programming languages.

Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis


Matching Linear Algebra and Tensor Code to Specialized Hardware Accelerators

Abstract: Dedicated tensor accelerators demonstrate the importance of linear algebra in modern applications. Such accelerators have the potential for impressive performance gains, but require programmers to rewrite code using vendor APIs - a barrier to wider scale adoption. Recent work overcomes this by matching and replacing patterns within code, but such approaches are fragile and fail to cope with the diversity of real-world codes. We develop ATC, a compiler that uses program synthesis to map regions of code to specific APIs. The mapping space that ATC explores is combinatorially large, requiring the development of program classification, dynamic analysis, variable constraint generation and lexical distance matching techniques to make it tractable. We apply ATC to real-world tensor and linear algebra codes and evaluate them against four state-of-the-art approaches. We accelerate between 2.6x and 7x more programs, leading to over an order of magnitude performance improvement.

Proceedings of the 32nd ACM SIGPLAN International Conference on Compiler Construction


Synthesis of Mathematical programs from Natural Language Specifications

Abstract: Several decision problems that are encountered in various business domains can be modeled as mathematical programs, i.e. optimization problems. The process of conducting such modeling often requires the involvement of experts trained in operations research and advanced algorithms. Surprisingly, despite the significant advances in the methods for program and code synthesis, AutoML, learning to optimize etc., there has been little or no attention paid to automating the task of synthesizing mathematical programs. We imagine a scenario where the specifications for modeling, i.e. the objective and constraints are expressed in an unstructured form in natural language (NL) and the mathematical program has to be synthesized from such an NL specification. In this work we evaluate the efficacy of employing CodeT5 with data augmentation and post-processing of beams. We utilize GPT-3 with back translation for generation of synthetic examples. Further we apply rules of linear programming to score beams and correct beams based on common error patterns. We observe that with these enhancements CodeT5 base gives an execution accuracy of 0.73 which is significantly better than zero-shot execution accuracy of 0.41 by ChatGPT and 0.36 by Codex.


WikiCoder: Learning to Write Knowledge-Powered Code

Abstract: We tackle the problem of automatic generation of computer programs from a few pairs of input-output examples. The starting point of this work is the observation that in many applications a solution program must use external knowledge not present in the examples: we call such programs knowledge-powered since they can refer to information collected from a knowledge graph such as Wikipedia. This paper makes a first step towards knowledge-powered program synthesis. We present WikiCoder, a system building upon state of the art machine-learned program synthesizers and integrating knowledge graphs. We evaluate it to show its wide applicability over different domains and discuss its limitations. WikiCoder solves tasks that no program synthesizers were able to solve before thanks to the use of knowledge graphs, while integrating with recent developments in the field to operate at scale.


Template-based Neural Program Repair

Abstract: In recent years, template-based and NMT-based automated program repair methods have been widely studied and achieved promising results. However, there are still disadvantages in both methods. The template-based methods cannot fix the bugs whose types are beyond the capabilities of the templates and only use the syntax information to guide the patch synthesis, while the NMT-based methods intend to generate the small range of fixed code for better performance and may suffer from the OOV (Out-of-vocabulary) problem. To solve these problems, we propose a novel template-based neural program repair approach called TENURE to combine the template-based and NMT- based methods. First, we build two large-scale datasets for 35 fix templates from template-based method and one special fix template (single-line code generation) from NMT-based method, respectively. Second, the encoder-decoder models are adopted to learn deep semantic features for generating patch intermediate representations (IRs) for different templates. The optimized copy mechanism is also used to alleviate the OOV problem. Third, based on the combined patch IRs for different templates, three tools are developed to recover real patches from the patch IRs, replace the unknown tokens, and filter the patch candidates with compilation errors by leveraging the project-specific information. On Defects4J-vl.2, TENURE can fix 79 bugs and 52 bugs with perfect and Ochiai fault localization, respectively. It is able to repair 50 and 32 bugs as well on Defects4J-v2.0. Compared with the existing template-based and NMT-based studies, TENURE achieves the best performance in all experiments.

2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)


RetypeR: Integrated Retrieval-based Automatic Program Repair for Python Type Errors

Abstract: Python is a widely popular dynamic programming language. While Python's dynamic type system facilitates the development of Python programs, it also introduces type errors at run-time which are often challenging to repair. Large language models (LLMs) have recently demonstrated impressive performance in code understanding and generation, and have been applied to patch synthesis. However, the ability of LLMs to repair type errors is often limited by their finite input context window, which may not fully encompass the repair information necessary for fixing a specific type error, such as valuable context code and fix patterns. In this paper, we introduce RetypeR, an integrated retrieval-based framework to repair Python type errors automatically. RetypeR comprises two stages: external retrieval and internal retrieval. The external retrieval stage retrieves applicable fix patterns from external code repositories containing historical error fixes via an abstract syntax tree (AST)-based retrieval strategy. In the internal retrieval stage, a novel fix ingredient retriever is designed to extract valuable repair ingredients from local context within source files based on both lexical and semantic similarity. To integrate the information retrieved from the aforementioned external and internal stages, RetypeR constructs context-aware repair prompts for LLMs to guide them in generating correct patches for type errors. We evaluate RetypeR on two real-world benchmarks, TypeBugs and BugsInPy. The experimental results demonstrate that RetypeR achieves state-of-the-art performance, improving the best baseline by 12.73% and 11.54% on TypeBugs and BugsInPy benchmarks, respectively.

2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)


Agentless: Demystifying LLM-based Software Engineering Agents

Abstract: Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless -- an agentless approach to automatically solve software development problems. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents! Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patch or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite-S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the current overlooked potential of a simple, interpretable technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction.