40 papers across AI, ML, NLP, and CV from the last 24 hours.
Three threads run through today's batch: evaluation is becoming its own research object, retrieval is getting a rethink, and self-improvement is shedding its dependence on external supervision. Half a dozen papers attack the measurement problem from different angles — auditing benchmarks for conversational agents, tracing error lifecycles in long agent trajectories, certifying when agent evaluations can stop early (74x cheaper), and causally auditing whether visual tool-use operations actually influence model outputs. The pattern is clear: as agents move into production, the community is no longer satisfied with leaderboard scores and is building tools to verify that evaluations themselves are trustworthy.
Retrieval-augmented generation faces structural criticism. Beyond Top-K demonstrates that embedding-based retrieval is fundamentally mismatched for tabular-heavy documents like financial reports, while NeSy-RAG and TYTAN push neuro-symbolic approaches that make reasoning steps attributable to evidence rather than buried in black-box embeddings.
On-policy self-distillation without any supervision stands out. It shows that a model can improve using only its own generations filtered by internal consistency — no ground truth, no teacher model, no environment feedback. If this holds up, it undercuts the assumption that post-training necessarily requires external signals.
Notable absences: no new foundation model releases, no scaling-law analyses, no fresh benchmark leaderboards. The day's energy went into auditing, governing, and understanding systems rather than building bigger ones. That suggests a field entering a consolidation phase — less about what models can do, more about whether we can measure and trust what they do.
Soorya Ram Shimgekar, Michelle Hu, Dorisa Shehi · 2026-08-06
Electronic health record (EHR) feature engineering is a major bottleneck in clinical research and AI, accounting for 39-45% of data scientists' workload. This is especially pronounced in heart failure, which affects an estimated 6.7 million U.S. adults and requires integrating fragmented EHR data with disease-specific, guideline-based clinical reasoning. Existing rule-based and large language mode
Sarvesh Baskar, Zikui Cai, Shayan Shabihi · 2026-08-06
Real-world video benchmarks provide broad coverage, but their fixed clips entangle event count, rate, duration, and visual complexity, making failure modes hard to isolate. While existing programmatic benchmarks offer better control, they score only the final answer rather than auditing reported events against executable ground truth. To bridge this gap, we introduce trace-grounded parametric prof
Jerzy Stefanowski · 2026-08-06
This paper addresses the limitations of Explainable Artificial Intelligence (XAI) with respect to insufficient evaluation. They are illustrated through the DetoxAI image recognition system for bias detection and concept unlearning. Then, an example of a human-grounded evaluation of methods for explaining image classification is presented. The paper further explores methods for adapting explanation
Yunjia Qi, Zehua Yin, Xintong Shi · 2026-08-06
LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging. Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure. However, progress faces two main challenges. First, long trajectories make it difficult to identify individual errors, sin
Sagar Tamang, Ayush Vyas, Tabarakul Hazarika · 2026-08-06
Retrieval-augmented generation over long documents is dominated by one design: chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query. We argue that for an important class of documents -- financial statements, audit reports, regulatory returns -- this design is structurally unsound, and we make the argument measurable. On a 780-page government financial report, 86.
Varun Ursekar, Apaar Shanker, Yash Maurya · 2026-08-06
As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a
Arya Labroo, Mengjie Qian, Kate Knill · 2026-08-06
Automatic speaking assessment systems are increasingly deployed in high-stakes settings to mark second language (L2) learners' speaking tests, making it critical to show that their scores depend on speaking proficiency rather than irrelevant speaker attributes such as first language (L1) or age. Transformer-based foundation models have improved the accuracy of these L2 speaking graders, but their
Mutasim Fuad Sarker, Adiba Rahman Namira, Wafa Binte Alam · 2026-08-06
Cardiac arrest remains one of the most lethal conditions encountered in intensive care units. Despite the growing availability of electronic health record data, existing mortality prediction studies in this population largely depend on static summaries derived from early admission. Such approaches ignore the temporal progression of physiological deterioration and recovery that unfolds throughout a
Zhiheng Wang, Bo Peng, Lai Wei · 2026-08-06
The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visua
Omid Bazgir, Md Nasir, Jacob Hoffman · 2026-08-06
Synthetic clinical benchmarks for enterprise AI agents can pass existing utility checks and still remain structurally unrealistic, especially in privacy-sensitive healthcare settings where operational data are hard to access. We study how to improve such benchmarks without breaking the downstream utility checks already used in practice. We formulate benchmark revision as utility-constrained real
Markus Engelund Mathiasen, Jian Qian, Nikita Zhivotovskiy · 2026-08-06
Let $H\subseteq{-1,+1}^X$ be a class of finite VC dimension $d\ge1$. Writing $L$ for the binary risk and $L^=\min_{h\in H}L(h)$, we construct a learner achieving the statistically optimal risk bound: from an i.i.d.\ sample of size $n$, for every $0<δ\le 1/2$, with probability at least $1-δ$, [ L(\widehat h) \le L^+ 7\cdot10^8\left( \sqrt{\frac{L^*(d+\log(1/δ))}{n}} +\frac{d+\log(1/δ)}
Fanzhe Meng, Guoxin Chen, Jiale Zhao · 2026-08-06
Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. Executable validation establishes feasibility, yet does not reveal how a task behaves relative to a given solver setting. In this paper, we present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate
Chenglong Wang, Ziming Zhu, Yifu Huo · 2026-08-06
Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling
Yijiang Li, Bingyang Wang, Yijun Liang · 2026-08-06
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achi
Johann Birnick, Rayan Saab · 2026-08-06
We accelerate a family of algorithms for neural network quantization whose geometry is informed by any Kronecker-factored approximation of the Hessian. GPTQ-style adaptive rounding typically uses one-sided information derived from input activations. Two-sided Kronecker-factored Hessian approximations can additionally capture correlations across output coordinates, but applying GPTQ directly in the
Lev V. Utkin, Stanislav K. Kogan, Andrei V. Konstantinov · 2026-08-06
This work presents a novel attention-based framework for estimating the Individual Probability of Treatment Benefit (IPTB) in survival analysis contexts. The proposed model, called Surv-IPTB, directly quantifies the probability that a specific patient will experience extended survival time under treatment versus control. We reformulate IPTB estimation as a binary classification problem, leveraging
Iosif Lytras, Nikolaos Makras, Sotirios Sabanis · 2026-08-06
We study the problem of sampling from target distributions whose potentials are simultaneously non-smooth, subject to superlinear gradient growth, and non-convex. We introduce the Subgradient Tamed Unadjusted Langevin Algorithm (SG-TULA), a discretisation of the Langevin diffusion that operates directly on subgradients, without relying on computationally demanding smoothing procedures. To handle t
Zonghuan Xu · 2026-08-06
Model evaluations may fix all tests before observing any responses or select later tests using earlier responses. We study this choice in a conditional-query model on a finite outcome space $\mathcal{X}$ with $|\mathcal{X}|=N$. We first ask which pairs of distribution classes can be reliably distinguished. We then ask how many additional queries are required to match an adaptive tester when all qu
Yiting Zheng, Cheng Fang, Anthony Donofrio · 2026-08-06
Reaction yield prediction remains challenging because labeled data are scarce and reaction space is both combinatorially large and sparsely populated, limiting the generalization of existing reaction representations. String-, fingerprint-, and graph-based reaction encodings only partially capture chemical transformations, making accurate prediction difficult for reactions with complex substrates.
Dohyun Ku, Min Gu Kwak, Francisco J. Pasquel · 2026-08-06
Metabolomics knowledge is distributed across heterogeneous resources and remains difficult to translate into predictive representations. We developed MetaboLLM, a metabolomics-specialized large language model adapted through continual pretraining, supervised fine-tuning, and structured retrieval, together with MetaboLLM-GIN, which converts generated biochemical descriptions into metabolite graphs
Xian Sun, Wei Chow, Yingshuo Wang · 2026-08-06
Language models increasingly condition their answers on external signals, and a single misleading one can turn a correct answer wrong. The obvious remedy, training models to resist such signals, hides a failure mode: a model that ignores all context looks robust yet is useless when the context is worth trusting. We recast the problem as selective trust and introduce MIST, a human-annotated benchma
Ishan Patel, Sahil Sen, Elias Lumer · 2026-08-06
Tool use transforms LLMs into agents that act beyond their training data, and for code-capable models, programmatic tool calling extends this further by replacing rigid JSON calls with scripts that chain and parallelize naturally. However, a systematic evaluation of tools as code on an established benchmark across current and prior model generations under real-world task conditions has not been co
Xinye Wang, Junxiao Liu, Shujian Huang · 2026-08-06
Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-ling
Noam Koren, Roy Bar-Haim, Abigail Goldsteen · 2026-08-06
Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. Poor benchmarks may contain inconsistent tasks, simplistic scenarios, or limited policy coverage, leading to unreliable evaluations. We introduce a reference-free framework that uses LLM judges to assess benchmark consistency, complexity, and policy cover
Tao Wang, Qihao Yang, Rongjiao Liang · 2026-08-06
Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency.
This digest is generated automatically from arXiv submissions. Not affiliated with arXiv or Cornell University.