new

Get trending papers in your email inbox!

Subscribe

Daily Papers

byAK and the research community

Sep 2

Can Open-Source LLMs Compete with Commercial Models? Exploring the Few-Shot Performance of Current GPT Models in Biomedical Tasks

Commercial large language models (LLMs), like OpenAI's GPT-4 powering ChatGPT and Anthropic's Claude 3 Opus, have dominated natural language processing (NLP) benchmarks across different domains. New competing Open-Source alternatives like Mixtral 8x7B or Llama 3 have emerged and seem to be closing the gap while often offering higher throughput and being less costly to use. Open-Source LLMs can also be self-hosted, which makes them interesting for enterprise and clinical use cases where sensitive data should not be processed by third parties. We participated in the 12th BioASQ challenge, which is a retrieval augmented generation (RAG) setting, and explored the performance of current GPT models Claude 3 Opus, GPT-3.5-turbo and Mixtral 8x7b with in-context learning (zero-shot, few-shot) and QLoRa fine-tuning. We also explored how additional relevant knowledge from Wikipedia added to the context-window of the LLM might improve their performance. Mixtral 8x7b was competitive in the 10-shot setting, both with and without fine-tuning, but failed to produce usable results in the zero-shot setting. QLoRa fine-tuning and Wikipedia context did not lead to measurable performance gains. Our results indicate that the performance gap between commercial and open-source models in RAG setups exists mainly in the zero-shot setting and can be closed by simply collecting few-shot examples for domain-specific use cases. The code needed to rerun these experiments is available through GitHub.

  • 2 authors
·
Jul 18, 2024

The Benefits of Vulnerability Discovery and Bug Bounty Programs: Case Studies of Chromium and Firefox

Recently, bug-bounty programs have gained popularity and become a significant part of the security culture of many organizations. Bug-bounty programs enable organizations to enhance their security posture by harnessing the diverse expertise of crowds of external security experts (i.e., bug hunters). Nonetheless, quantifying the benefits of bug-bounty programs remains elusive, which presents a significant challenge for managing them. Previous studies focused on measuring their benefits in terms of the number of vulnerabilities reported or based on the properties of the reported vulnerabilities, such as severity or exploitability. However, beyond these inherent properties, the value of a report also depends on the probability that the vulnerability would be discovered by a threat actor before an internal expert could discover and patch it. In this paper, we present a data-driven study of the Chromium and Firefox vulnerability-reward programs. First, we estimate the difficulty of discovering a vulnerability using the probability of rediscovery as a novel metric. Our findings show that vulnerability discovery and patching provide clear benefits by making it difficult for threat actors to find vulnerabilities; however, we also identify opportunities for improvement, such as incentivizing bug hunters to focus more on development releases. Second, we compare the types of vulnerabilities that are discovered internally vs. externally and those that are exploited by threat actors. We observe significant differences between vulnerabilities found by external bug hunters, internal security teams, and external threat actors, which indicates that bug-bounty programs provide an important benefit by complementing the expertise of internal teams, but also that external hunters should be incentivized more to focus on the types of vulnerabilities that are likely to be exploited by threat actors.

  • 5 authors
·
Feb 23, 2023

Removing the Watermark Is Not Enough: Forensic Stealth in Generative-AI Watermark Removal

Watermarks for AI-generated images are meant to support downstream decisions about provenance, manipulation, and trust. In the settings that motivate watermark removal, therefore, success means more than causing the watermark test to fail. A successful remover must also preserve the utility of the image and make the output forensically indistinguishable from clean content, so that defeating the verifier restores deniability rather than merely replacing one detection signal with another. We show that current watermark removal attacks fail this stronger objective. Across six state-of-the-art removers spanning four attack families, independent forensic detectors distinguish removal-processed outputs from clean images at over 98% true-positive rate under a 1% false-positive budget. Thus, current removers often replace the watermark with a different detectable signal. Using UnMarker (IEEE S&P 2025) as a detailed case study, we show that this signal persists under common post-processing, exhibits a characteristic two-regime spectral deformation, and yields a three-way tension among removal success, image quality, and forensic stealth. These results show that existing removal benchmarks are incomplete: they reward verifier evasion and utility preservation while omitting forensic stealth. A workable watermark remover must satisfy all three conditions at once: watermark evasion, utility preservation, and forensic indistinguishability from clean content.

  • 2 authors
·
May 8

Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract Scene Descriptions

What constitutes the "vibe" of a particular scene? What should one find in "a busy, dirty city street", "an idyllic countryside", or "a crime scene in an abandoned living room"? The translation from abstract scene descriptions to stylized scene elements cannot be done with any generality by extant systems trained on rigid and limited indoor datasets. In this paper, we propose to leverage the knowledge captured by foundation models to accomplish this translation. We present a system that can serve as a tool to generate stylized assets for 3D scenes described by a short phrase, without the need to enumerate the objects to be found within the scene or give instructions on their appearance. Additionally, it is robust to open-world concepts in a way that traditional methods trained on limited data are not, affording more creative freedom to the 3D artist. Our system demonstrates this using a foundation model "team" composed of a large language model, a vision-language model and several image diffusion models, which communicate using an interpretable and user-editable intermediate representation, thus allowing for more versatile and controllable stylized asset generation for 3D artists. We introduce novel metrics for this task, and show through human evaluations that in 91% of the cases, our system outputs are judged more faithful to the semantics of the input scene description than the baseline, thus highlighting the potential of this approach to radically accelerate the 3D content creation process for 3D artists.

  • 4 authors
·
Jun 9, 2023

Mask-to-Height: A YOLOv11-Based Architecture for Joint Building Instance Segmentation and Height Classification from Satellite Imagery

Accurate building instance segmentation and height classification are critical for urban planning, 3D city modeling, and infrastructure monitoring. This paper presents a detailed analysis of YOLOv11, the recent advancement in the YOLO series of deep learning models, focusing on its application to joint building extraction and discrete height classification from satellite imagery. YOLOv11 builds on the strengths of earlier YOLO models by introducing a more efficient architecture that better combines features at different scales, improves object localization accuracy, and enhances performance in complex urban scenes. Using the DFC2023 Track 2 dataset -- which includes over 125,000 annotated buildings across 12 cities -- we evaluate YOLOv11's performance using metrics such as precision, recall, F1 score, and mean average precision (mAP). Our findings demonstrate that YOLOv11 achieves strong instance segmentation performance with 60.4\% mAP@50 and 38.3\% mAP@50--95 while maintaining robust classification accuracy across five predefined height tiers. The model excels in handling occlusions, complex building shapes, and class imbalance, particularly for rare high-rise structures. Comparative analysis confirms that YOLOv11 outperforms earlier multitask frameworks in both detection accuracy and inference speed, making it well-suited for real-time, large-scale urban mapping. This research highlights YOLOv11's potential to advance semantic urban reconstruction through streamlined categorical height modeling, offering actionable insights for future developments in remote sensing and geospatial intelligence.

  • 4 authors
·
Oct 31, 2025 1

Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages

Extracting clinical information from medical transcripts in low-resource languages remains a significant challenge in healthcare natural language processing (NLP). This study evaluates a two-step pipeline combining Aya-expanse-8B as a Persian-to-English translation model with five open-source small language models (SLMs) -- Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-3-1B-it -- for binary extraction of 13 clinical features from 1,221 anonymized Persian transcripts collected at a cancer palliative care call center. Using a few-shot prompting strategy without fine-tuning, models were assessed on macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity to account for class imbalance. Qwen2.5-7B-Instruct achieved the highest overall performance (median macro-F1: 0.899; MCC: 0.797), while Gemma-3-1B-it showed the weakest results. Larger models (7B--8B parameters) consistently outperformed smaller counterparts in sensitivity and MCC. A bilingual analysis of Aya-expanse-8B revealed that translating Persian transcripts to English improved sensitivity, reduced missing outputs, and boosted metrics robust to class imbalance, though at the cost of slightly lower specificity and precision. Feature-level results showed reliable extraction of physiological symptoms across most models, whereas psychological complaints, administrative requests, and complex somatic features remained challenging. These findings establish a practical, privacy-preserving blueprint for deploying open-source SLMs in multilingual clinical NLP settings with limited infrastructure and annotation resources, and highlight the importance of jointly optimizing model scale and input language strategy for sensitive healthcare applications.

  • 7 authors
·
Feb 24 2

The Coding Limits of Robust Watermarking for Generative Models

We study a basic question about cryptographic watermarking for generative models: how reliable can a watermark remain when an adversary is allowed to corrupt the encoded signal? To address this question, we introduce a minimal coding abstraction that we call a zero-bit tamper-detection code. This is a secret-key procedure that samples a pseudorandom codeword and, given a candidate word, decides whether it should be treated as unmarked content or as the result of tampering with a valid codeword. It captures the two core requirements of robust watermarking: soundness and tamper detection. Within this abstraction we prove a sharp unconditional limit on robustness to independent symbol corruption. For an alphabet of size q, there is a critical corruption rate of 1-1/q such that no scheme with soundness, even relaxed to allow a fixed constant false positive probability on random content, can reliably detect tampering once an adversary can change more than this fraction of symbols. In particular, in the binary case no cryptographic watermark can remain robust if more than half of the encoded bits are modified. We also show that this threshold is tight by giving simple information-theoretic constructions that achieve soundness and tamper detection for all strictly smaller corruption rates. We then test experimentally whether this limit appears in practice by looking at the recent watermarking for images of Gunn, Zhao, and Song (ICLR 2025). We show that a simple crop and resize operation reliably flipped about half of the latent signs and consistently prevented belief-propagation decoding from recovering the codeword, erasing the watermark while leaving the image visually intact.

  • 5 authors
·
May 1

Roadmap: 2D Materials for Quantum Technologies

Two-dimensional (2D) materials have emerged as a versatile and powerful platform for quantum technologies, offering atomic-scale control, strong quantum confinement, and seamless integration into heterogeneous device architectures. Their reduced dimensionality enables unique quantum phenomena, including optically addressable spin defects, tunable single-photon emitters, low-dimensional magnetism, gate-controlled superconductivity, and correlated states in Moiré superlattices. This Roadmap provides a comprehensive overview of recent progress and future directions in exploiting 2D materials for quantum sensing, computation, communication, and simulation. We survey advances spanning spin defects and quantum sensing, quantum emitters and nonlinear photonics, computational theory and data-driven discovery of quantum defects, spintronic and magnonic devices, cavity-engineered quantum materials, superconducting and hybrid quantum circuits, quantum dots, Moiré quantum simulators, and quantum communication platforms. Across these themes, we identify common challenges in defect control, coherence preservation, interfacial engineering, and scalable integration, alongside emerging opportunities driven by machine-learning-assisted design and integrated experiment-theory feedback loops. By connecting microscopic quantum states to mesoscopic excitations and macroscopic device architectures, this Roadmap outlines a materials-centric framework for integrating coherent quantum functionalities and positions 2D materials as foundational building blocks for next-generation quantum technologies.

  • 32 authors
·
Dec 16, 2025