公开论文雷达

公开 arXiv 研究简报 · 2026-08-01T01:03:34.512957+00:00

先看结论和关键数字,再决定要不要读原文。候选只在首次出现时展示;旧候选若后来通过深读门槛,仍会进入重点。

评测与验证怎么做扎实:七篇工程卡对读

这组卡的共同关切是评测与验证的可信度。多篇从真实合并PR或仓库变更机械造任务与规则,并反复强调一件事:表面通过不等于真做对——功能测试过、内核接受目标、短历史排名,都可能掩盖缺陷。分歧在于目的:一半在压成本、复用环境与知识库,一半在造更严的尺子暴露隐藏失败。先读管道最完整的Change2Task,再顺着质量与验证盲点往下看。

推荐阅读顺序

  1. 2607.28591:分数最高,给出从合并PR三级重建可验证智能体任务的完整管道,是其余评测卡的基础参照
  2. 2607.27409:紧接着看:同样从真实PR造任务,却揭示功能通过≠改得好,补上非功能质量维度的盲点
  3. 2607.21962:方法论最锋利:先建事实再生对话,暴露记忆架构排名随历史时长整体反转
  4. 2607.28110:验证陷阱样板:内核接受目标≠蓝图修对,还给出三种修复接口的覆盖与成本对比
  5. 2607.27259:转到硬件形式验证,看可复用证明库如何削减同族参数化任务的重复推理
  6. 2607.17211:成本视角收尾:把LLM集中到一次性合成检测器,扫描阶段零调用、成本可控
  7. 2607.27374:最后读:召回优先分类器加RAG的两级解耦,样本较小、作者自定位为初步验证
共性方法
多篇都从真实合并PR或仓库变更机械造任务、规则与金答案,力求评测可复现、可归因;并共享一个警觉:表面通过不代表真做对——功能测试过、内核接受目标定理、短历史下的排名,都可能把没修好的当成修好了。它们普遍把能力评估拆成相互独立的两级,分别衡量。
关键分歧
目的不同。Change2Task、CircuitProver、Prefactory偏向省成本与复用:复用现代基础环境、积累证明库、一次性合成检测器,收益随同族任务累积。SWE-NFI、Ground Truth First、BlueprintRepair偏向造更严的尺子:揭示智能体系统性落后、架构排名反转、验证盲点,重点是暴露隐藏失败而非提速。
选择准则
想给智能体扩训练/评测数据或降环境成本,先看Change2Task和Prefactory;想验证选型是否被表面指标骗,先看SWE-NFI、Ground Truth First、BlueprintRepair;形式验证选CircuitProver或BlueprintRepair。

重点深读(7 / 7 篇)

形式化与程序验证(2 篇)

形式化与程序验证 10/30

CircuitProver: Agentic Lean 4 Theorem Proving with Reusable Circuit Proof Library for Hardware Verification

参数化硬件定理证明与可复用证明知识积累:模型检查逐实例验证硬件、推理不可复用;CircuitProver将参数化硬件与自然语言规格自动形式化为Lean 4,智能体迭代生成机器验证定理,证明轨迹提炼为可复用库。63任务全部通过,知识积累令验证时间减少23.2%。

两句看懂

模型检查逐实例求解、推理痕迹锁在求解器内部无法复用,跨关联设计须反复重建验证逻辑;CircuitProver改用Lean 4构建参数化定理,将证明轨迹提炼为可复用引理库。在首个63任务硬件定理证明基准上全部通过(100%),香草智能体仅92.1%且证明轮次多一倍,知识积累令验证时间减少23.2%。

核心判断

参数化硬件形式验证可由Lean 4智能体全自动完成,且证明轨迹提炼为可复用库后可显著降低同族任务的验证代价;63任务100%通过、时间减少23.2%为直接证据。

关键要点

1. 模型检查将硬件编码为AIGER/BTOR后由求解器判定,仅输出通过/失败或反例,推理过程不以显式制品形式保存,相关设计间的公共引理须反复生成,无跨任务复用机制。 2. CircuitProver构建三阶段流水线:自动形式化将参数化RTL与自然语言规格转为含符号参数n的Lean 4模型;智能体迭代调用Lean反馈修复证明;已验证引理以机器可验证形式存档,成功轨迹转为自然语言策略,均写入可复用库;消融对照为无库香草智能体。 3. 63任务基准上CircuitProver全部通过,香草智能体仅92.1%且平均证明轮次多一倍;消融显示知识积累令证明长度减少16.3%、验证时间减少23.2%,确认库的复用价值集中在跨相关任务的冗余推理削减而非单任务加速。

证据与结果

首个专门面向智能体硬件定理证明的基准,共63任务,覆盖多种参数化硬件模块(含计数器等),含功能正确性证明与参数化属性证明两类;评估指标包括证明成功率、证明轮次、证明长度(行数)和验证时间;基线为无知识库的香草智能体。结果:CircuitProver 100%通过,香草智能体92.1%、证明轮次多一倍;消融:知识积累令证明长度减少16.3%、验证时间减少23.2%。

打开论文原文
它要解决什么
能否用智能体自动将参数化硬件形式化为Lean 4并构建可复用证明库,避免跨相关设计的重复推理?
研究路径
①自动形式化:将参数化RTL与自然语言规格共同编译为统一Lean 4表示,保留符号参数;②智能体证明:调用Lean 4编译器获取错误反馈,迭代修复直至定理通过;③知识蒸馏:已验证引理以形式制品存档,成功轨迹提炼为自然语言策略,同步写入可复用库;④新任务先检索库,命中则直接引用已验证引理,未命中则新建并入库。
这对工程意味着什么
在参数化硬件验证流水线中部署证明知识库:每次成功验证后将引理和策略存档,同族新设计到来时智能体优先检索复用,可减少约四分之一验证时间;避免的误区是仅在单任务内重试而不向库写入,跨任务收益为零。
证据定位
63任务中CircuitProver全部通过(100%),香草智能体仅完成92.1%且平均需两倍证明轮次;知识积累令证明长度减少16.3%、验证时间减少23.2%。(筛选维度:形式化验证、可复核评测)
适用边界
基准仅63任务且集中于参数化组合/时序电路(如计数器),大型工业级设计(处理器、内存子系统)未覆盖;论文未报告自动形式化对规格描述质量的敏感性及形式化失败率。
方法与英文摘要

基准含63任务,覆盖多种参数化硬件模块与规格,含功能性证明和属性证明两类。输入为参数化RTL描述加自然语言规格;自动形式化流水线将其转为Lean 4模型(保留符号参数n);智能体在Lean反馈下迭代修复证明;已验证引理和证明策略存入可复用库,后续任务优先检索复用。

Modern integrated circuits (ICs) are becoming increasingly complex, making functional verification a major bottleneck. The dominant hardware formal verification methodology, model checking, verifies each design instance separately and exposes only pass/fail results, so the reasoning behind a proof stays locked inside solver heuristics and is repeatedly reconstructed across related designs. Interactive theorem proving instead yields explicit, reusable proof artifacts, but applying it to hardware remains largely manual, demanding expert effort for formalization, invariant discovery, and proof development. In this paper, we present CircuitProver, an agentic Lean 4-based verification framework supporting proof-accumulation and parameterized verification. CircuitProver automatically translates parameterized hardware designs and their natural language specifications into executable Lean 4 models. It then iteratively constructs machine-checked proofs through Lean feedback to establish that the hardware code complies with the specification. The proving traces and verified theorems are distilled into reusable libraries, where proving strategies guide future agent reasoning and verified lemmas support formal proof reuse across related hardware verification tasks. We further introduce the first benchmark suite for evaluating agentic hardware theorem proving, covering diverse parameterized hardware designs, specifications, proof tasks, and evaluation metrics. Across 63 tasks, CircuitProver successfully proves all benchmarks, while a vanilla agent solves 92.1% of them and requires twice as many proof rounds on average. Ablation studies show that accumulated proof knowledge reduces redundant proof construction across related verification tasks, reducing proof length by 16.3% and verification time by 23.2%.

形式化与程序验证 6/30

BlueprintRepair: Typed Local Edits for Failed Lean Proof Blueprints

修Lean蓝图首选类型化局部编辑:覆盖率与自由接口相当,成本最低:LLM 修复 Lean 证明蓝图(形式语句依赖图)失败时,常见做法是打源码补丁或整段重写。这项工作多给了一条路:10 种模式受限的局部编辑操作,并配了一个 142 个受控失败案例的基准(91 个局部缺陷、39 个证明撰写、12 个复合)。

两句看懂

LLM 修 Lean 蓝图时,只验证目标定理被内核接受不够:证明项内部可能引用了依赖图里未声明的蓝图引理,Lean 照样接受目标证明,依赖图却已经错了。142 个受控失败状态的对比里,类型编辑覆盖率与自由接口相近(局部缺陷约 79 对 81 次),但 10k token 内总解 103 个、每次修复成本最低——补丁贵 1.30 倍,重写贵 2.06 倍。

核心判断

类型化局部编辑修蓝图局部缺陷,覆盖率不输自由接口(79 对 81),token 消耗和单价都最低;同时,修复是否成功不能只看目标定理被内核接受,必须检查编译证明项里跨节点依赖的声明完整性。142 个状态、三接口的对比支撑这两点。

关键要点

1. 旧评估有盲点:内核接受目标定理不等于蓝图修复正确。证明项可能引用了另一个蓝图引理却没在依赖图中声明,Lean 仍接受目标证明,依赖图已经错了;正确评估要逐个检查编译证明项中每个跨节点依赖是否已声明。 2. 基准 142 个受控失败状态(例题含 miniF2F 等),分三层:91 个局部图/语句缺陷(12 个复合单独评估)、39 个证明撰写。三接口共享相同源码、反馈、模型、预算,各跑一 episode,接受条件统一要求跨节点依赖声明完整,唯一差异是修复表达方式。 3. DeepSeek-V4-Flash 下类型编辑 79 次、补丁与重写各 81 次,覆盖率接近;10k token 内类型 103 次、补丁 90、重写 87,成本优势明显(补丁贵 1.30 倍,重写贵 2.06 倍)。Qwen3.6-Flash 重复同一模式,且在证明撰写状态类型编辑领先;12 个复合状态和 39 个证明撰写状态成功率更低,链式缺陷与证明内容生成是局部修复的硬边界。

证据与结果

基准 142 个受控失败状态(例题含 miniF2F 等),每个状态带完整接受/拒绝修复轨迹,分三层:91 个局部图/语句缺陷(12 个复合状态单独分层)、39 个证明撰写。模型用 DeepSeek-V4-Flash 和 Qwen3.6-Flash,三接口各跑一 episode、预算统一;指标是解决状态数、提供商成本、10k token 内覆盖。DeepSeek-V4-Flash 上,91 局部状态类型 79 vs 补丁 81 vs 重写 81;10k token 内类型 103、补丁 90、重写 87。Qwen3.6-Flash 总量较少,但类型编辑在证明撰写状态领先;各接口各有数个独有解,三者组合覆盖超过任何单一接口。

打开论文原文
它要解决什么
Lean 证明蓝图出现局部缺陷时,类型化局部编辑、源码补丁、全模块重写三种接口,覆盖率和效率差多少?
研究路径
接口只暴露 10 种模式受限操作:修改语句、增删依赖边、分裂或删除节点、为单个节点写证明;每次操作必须指名目标节点,越界操作被模式检查拦下。每步由 Lean 机械执行并实时返回验证反馈。修复终点有两条硬条件:目标定理被内核接受,且编译证明项引用的每个蓝图引理都已在依赖图中声明;只要有一个跨节点依赖未声明,整次修复判失败。
这对工程意味着什么
具体做法:当修复任务以局部图结构缺陷为主时,直接用类型化局部编辑接口,以最低成本覆盖大多数可达状态。要避开的捷径:别把「目标定理被内核接受」当成修复成功——它只说明目标可证,证明项里未声明的跨节点依赖查不出来;必须逐项核对编译证明项中每个依赖是否已在图中声明。
证据定位
DeepSeek-V4-Flash 下,91 个局部缺陷状态中类型编辑解 79 个,补丁和重写各 81 个,覆盖率几乎持平。10k token 预算内,类型编辑总解 103 个,补丁 90 个,重写 87 个。成本上,补丁贵 1.30 倍,重写贵 2.06 倍。(筛选维度:形式化验证、可复核评测)
适用边界
基准只有 142 个受控失败状态,来源偏向形式化数学题库(例题含 miniF2F);每 episode 只跑单次执行、预算固定,没评估多轮迭代修复;复合状态只有 12 个,对链式缺陷的覆盖有限;也没有覆盖大规模真实项目蓝图的失败分布。
方法与英文摘要

先搭基准:142 个受控失败状态,每题都带完整接受/拒绝修复轨迹,这样三接口的对比可复现、可归因。再固定源码、反馈、模型、预算,让三接口各跑一 episode,唯一差异就是修复表达方式。类型编辑的 10 种操作每步由 Lean 实时验证,非法操作当场拒绝;接受条件要求编译证明项中所有跨节点依赖都已在图中声明,这保证了「目标可证」和「蓝图正确」不会被混为一谈。

LLM-based Lean proving systems increasingly organize a proof as a blueprint: a dependency graph of formal statements. We introduce BlueprintRepair, a repair interface that lets a model change this graph through ten schema-checked local operations. An operation names the node it edits, so the target theorem cannot be changed. Lean checks every applied change, and an accepted repair must declare every blueprint lemma its proof uses. We also construct BlueprintTrace, a benchmark of 142 controlled failures with complete accepted and rejected repair trajectories. We compare typed edits, exact source patches, and complete module rewrites under matched source, feedback, model, and budget, one episode per state and interface. With DeepSeek-V4-Flash, the three interfaces solve almost the same number of the benchmark's localized failures. Typed repair is the cheapest per solved state (patching is 1.30x as expensive, rewriting 2.06x), and within 10,000 completion tokens per task it reaches almost all of its final coverage, while both free-form interfaces are well behind. A second model, Qwen3.6-Flash, solves fewer states but keeps typed repair cheapest, puts it ahead on the proof-authoring states, and repeats the localized pattern.

软件工程与仓库智能(3 篇)

软件工程与仓库智能 11/30

Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

把仓库合并PR自动改造成可执行、可验证的编程智能体任务:核心判断:仓库合并历史足以批量产出可执行、可验证的编程智能体任务。Change2Task 把合并PR放到现代健康版本上,按演化深度分三级重建——补丁回滚、代码映射、智能体重建。1130 个源变更中 79.6% 通过验证,比 PR 直接构建基线多恢复 29.2% 已验证任务,历史与重建样例的吻合率达 98.0%。

两句看懂

绑定历史快照的任务集随依赖与接口演化逐渐失效,合成数据又缺真实维护意图,逐任务构建容器环境成本极高。Change2Task 以合并 PR 为源,在现代健康版本上三级重建并验证任务状态:1130 个源变更实现 79.6% 验证成功率,比 PR 直接构建基线多恢复 29.2% 已验证任务,历史与重建样例智能体评测吻合率达 98.0%。

核心判断

仓库合并历史能以 79.6% 成功率转化为可验证的编程智能体任务;三级重建策略比 PR 直接构建基线多恢复 29.2% 已验证任务,历史与重建样例吻合率达 98.0%。这些量化结果共同支持这一结论。

关键要点

1. 旧假设与评测缺口:历史基准任务绑定原始快照,随依赖与接口演化逐渐失去可执行性;合成方法可批量生成但脱离真实开发意图;逐任务构建容器环境成本极高,已有系统报告跨 12.8K 仓库耗资约 89.1 万美元。 2. 构建协议与控制变量:三级策略随演化深度递增——L1 直接逆向应用历史补丁、L2 代码映射处理接口漂移、L3 智能体重建任务状态;任务适配器统一定义目标、预言机和编辑范围;验证覆盖健康基础、任务、还原三状态节点;复用同一现代基础派生多个轻量变体。 3. 决定性结果与边界:1130 个候选变更中 79.6% 验证通过;匹配集上比 PR 基线多恢复 29.2% 已验证任务;历史与重建样例智能体评测吻合率达 98.0%;复用现代基础使全流程开销降低 10.8%;L3 依赖智能体自动重建,对高度演化仓库存在吻合率上限。

证据与结果

数据:1130 个源变更,覆盖缺陷修复、功能添加、测试生成、API 迁移、安全修复五类任务族。对照:与基于 PR 直接构建的基线在匹配候选集上比较。主要指标:验证构建成功率 79.6%;匹配集上额外恢复 29.2% 已验证任务;历史与重建样例在智能体评测下吻合率 98.0%;复用现代基础环境使全流程开销降低 10.8%。任务验证覆盖三状态节点,每个任务附带还原补丁支持可逆验证。各任务族分类成功率及 L3 失败模式分布未在摘录中单独列出。

打开论文原文
它要解决什么
如何从仓库合并历史批量生成可执行且可验证的编程智能体任务,同时把逐任务的环境构建与存储开销降下来?
研究路径
输入是合并 PR(描述、补丁、行为证据)和同仓库固定可运行的后代版本。L1 把历史补丁逆向应用回去,成本最低;补丁因接口演化落不下去时,L2 用代码映射对齐漂移;再不行,L3 由智能体自动重建任务状态。每个被接受的任务补丁都附带还原补丁,在健康基础→任务→还原三个状态节点全流程验证,保证任务可逆、可复现。任务适配器统一定义目标、预期输出、可执行预言机与编辑范围,使验证不依赖人工检查。
这对工程意味着什么
具体做法:从维护中仓库的合并 PR 派生任务,并复用单一现代基础环境生成多变体。要避免的捷径:为每个历史快照单独构建容器镜像——存储与冷启动开销会随任务数线性膨胀。
证据定位
四个数字支撑结论:1130 个源变更中 79.6% 构建验证通过;匹配候选集上比 PR 直接构建基线多恢复 29.2% 已验证任务;历史与重建样例在智能体评测下吻合率 98.0%;复用现代基础环境使全流程开销降低 10.8%。(筛选维度:形式化验证、可复核评测、软件工程方法)
适用边界
只评测五类主流任务族,各类别成功率未单独量化;L3 重建质量受所用智能体能力制约,高度演化仓库的失败率未量化;摘录未说明跨仓库、跨语言或跨领域的泛化范围。
方法与英文摘要

输入是同一仓库的合并PR(描述、补丁、行为证据)加一个固定可运行的后代版本,共 1130 个源变更。构建分三级,成本随演化深度递增:L1 直接逆向应用历史补丁,适合界面基本没变的提交;L2 用代码映射对齐接口漂移,处理补丁落不下去的情况;L3 让智能体自动重建任务状态,兜底深度演化。任务适配器统一定义目标、预期输出、可执行预言机和编辑范围,让每个任务都能被自动检查。验证覆盖健康基础→任务→还原三状态全生命周期。任务族五类:缺陷修复、功能添加、测试生成、API 迁移、安全修复。

Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification. To expand this supply, we present Change2Task, a system grounded in repository history that converts merged pull requests into verified tasks on healthy modern revisions of the same repository. It aligns historical evidence with evolved code, reconstructs task states through Patch Reversal, Code Mapping, or Agent Reconstruction, and validates the lifecycle from a healthy base to a task state and a restored state. By deriving multiple tasks grounded in developer evidence from maintained environments, Change2Task provides executable data for coding agent training and evaluation while reducing repeated environment setup, storage, and task construction effort. We evaluate the system through five common and widely adopted coding agent task families: Bug Fix, Feature Addition, Test Generation, Application Programming Interface Migration, and Security Repair. Starting from 1,130 source changes eligible for construction, Change2Task achieves 79.6% verified task construction success across these task families. On a matched candidate set, it recovers 29.2% more verified tasks than a construction baseline based on pull requests. Historical and reconstructed cases achieve up to 98.0% matched outcome agreement under agent evaluation, while reuse of modern bases reduces measured expenditure across the complete pipeline by 10.8%.

软件工程与仓库智能 8/30

SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements

编码智能体非功能改进基准SWE-NFI:能通过功能测试,却做不好结构改进:现有编码基准只测功能正确性;SWE-NFI 从真实合并 PR 造出 188 个任务和 92 条可执行规则,同时评估行为保留与非功能改进。最强智能体功能正确率 70.0%,结构性改进得分却只有 0.0–1.3,人类参考 1.5——差距是系统性的。

两句看懂

现有编码基准靠单元测试,量不出文档完善和结构简化这类非功能改进;SWE-NFI 为此建了 188 个真实 PR 任务和 92 条可执行规则。评测商业与开源主流智能体后,最强智能体功能正确率达 70.0%,但结构改进 NFI 得分只有 0.0–1.3,人类参考 1.5,多文件任务明显比单文件难。

核心判断

编码智能体功能正确性能做到 70%,但结构简化这类非功能改进系统性落后于人类开发者。双层评估中,结构性改进 NFI 得分(智能体 0.0–1.3 vs 人类 1.5)在重复执行下稳定存在,说明这不是偶发失误。

关键要点

1. 现有基准(如 SWE-bench)只用单元测试验证功能正确性,测不出文档缺失、错误处理缺陷、库过时或结构冗余;功能测试通过不等于维护质量合格。 2. SWE-NFI 从开源 Python 项目真实合并 PR 构建 188 个任务(145 单文件、43 多文件),把五类 NFI 方向提炼成 92 条可执行规则,每个任务用功能测试加规则打分双层验证;多文件任务是受控难度变量。 3. 最强智能体功能正确率 70.0%,但结构性代码改进 NFI 得分仅 0.0–1.3,人类参考 1.5;多文件任务显著更难,重复执行 NFI 得分稳定,说明差距源于能力边界而非随机波动。

证据与结果

数据集:188 个任务(145 单文件、43 多文件),取自开源 Python 项目真实合并 PR。评估双层:手工功能测试验证行为保留 + 92 条规则驱动 NFI 打分。对象:商业与开源主流编码智能体,基线是对应人类参考实现。结果:最优智能体功能正确率 70.0%;结构性代码改进 NFI 得分智能体 0.0–1.3、人类 1.5;多文件任务明显更难;重复执行 NFI 得分稳定。

打开论文原文
它要解决什么
编码智能体在功能正确之外,能不能完成保持行为不变的非功能性改进?和人类开发者比,差在哪些维度?
研究路径
从 GitHub 已合并 PR 提取补丁,手工写功能测试确认行为不变;把 SWE-IF 五类方向(文档、错误处理与安全、库现代化、编码风格、结构简化)提炼成 92 条可执行规则。智能体输出先通过功能测试套件,再按规则逐条打分,得到功能正确率和 NFI 得分两个独立指标,并与人类参考实现比较。这样行为不变性由测试保证,非功能质量由规则量化,不会互相掩盖。
这对工程意味着什么
一个具体动作:选型或微调评估时,在功能测试之外加入规则检查(文档、错误处理、结构简化),把 NFI 得分当作第二指标。一个要避开的捷径:别只拿功能测试通过率当唯一标准,它分不清可维护和不可维护的功能正确实现。
证据定位
最强智能体功能正确率 70.0%;结构性代码改进 NFI 得分各智能体 0.0–1.3,人类参考 1.5;多文件任务远比单文件难;重复执行时 NFI 得分稳定,说明差距来自能力边界,不是随机波动。(筛选维度:可复核评测、软件工程方法)
适用边界
任务只覆盖开源 Python 项目的 PR,不含闭源、多语言或大型单体仓库;92 条 NFI 规则来自现有研究和编码规范,列不全所有非功能质量维度;原文未披露参评智能体的名称和数量。
方法与英文摘要

从开源 Python 项目已合并 PR 构建 188 个任务(145 单文件、43 多文件),把 SWE-IF 五类非功能改进方向提炼成 92 条可执行规则。每个任务配手工功能测试用例和规则驱动 NFI 评分,形成双层评估套件;用商业与开源主流智能体执行任务,同时记下功能正确率和 NFI 得分,并与对应的人类参考实现对照。规则把“行为没变”和“改得好不好”分开度量。

Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developer-oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0\% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents' NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.

软件工程与仓库智能 8/30

From Backlog Items to Security Guidance: Towards Continuous Security Compliance

用召回优先分类器加四阶段RAG管道,给backlog自动标注安全项并关联合规条款:受监管的敏捷开发里,安全需求常藏在自由格式的待办项文本中,工程师不容易识别。论文给出一个召回优先分类器加四阶段RAG管道:分类器域内F2=0.774,五个公开基准零样本G-measure约0.65;RAG从工业待办项检索24条款,其中12条获从业者≥4/5评分;同时发布288条专家标注数据集,标注一致性κ=0.787。

两句看懂

受监管的敏捷开发中,安全需求隐在自由格式待办项里,工程师缺少即时合规指引。研究者构建召回优先分类器加四阶段RAG管道,直接在backlog中标注安全项并关联合规条款;分类器在288条专家标注数据上域内F2=0.774,五个基准零样本G-measure约0.65,RAG检索24条款中12条获从业者≥4/5评分,属于初步验证可用性。

核心判断

待办项的普通文本就足够:既能判断安全相关性,也能关联合规条款——分类器域内F2=0.774,跨域G-measure约0.65,RAG 24条款中12条获从业者≥4/5评分。

关键要点

1. 旧评测缺口:安全需求以隐式自由文本嵌在待办项里,缺少专用工具支持;Wu et al.五个公开基准覆盖多类安全文本,但与工业backlog存在分布偏移,零样本G-measure只有约0.65,低于域内F2=0.774,说明公开基准替代不了工业场景评估。 2. 数据构建与受控变量:288条待办项来自两个公开Atlassian项目,9名从业者独立标注达κ=0.787;分类器用F2优化召回而非精确率;四阶段RAG以内部安全策略和CIS Benchmarks为文档库,与分类器分开评估,两级能力分别衡量。 3. 决定性结果与边界:分类器持平或优于经典ML和开源GPT基线;RAG 24条款中12条获≥4/5评分,但评估只有2名从业者和24条检索结果,样本量限制定论强度,作者明确定位为初步验证。

证据与结果

分类器域内:288条Atlassian公开项目待办项,9名从业者标注,κ=0.787,F2=0.774。跨域:Wu et al.五个公开安全文本基准,零样本迁移,G-measure均值约0.65,持平或优于经典ML和开源GPT基线。RAG:用工业待办项,文档库为内部安全策略加CIS Benchmarks,检索24条款,2名从业者评分,12条达≥4/5;评估规模小(2人、24条结果),属初步指示性验证,不具统计显著性。

打开论文原文
它要解决什么
产品待办项中,安全需求隐在自由文本里。工程师能不能靠NLP系统自动标出安全相关项,并立刻拿到适用的合规条款?
研究路径
待办项自然语言先送入召回优先二分类器(F2目标)做二值过滤,尽量不漏掉安全项;被标记的项进入四阶段RAG管道,以内部安全策略和CIS Benchmarks为文档库检索适用条款;检索结果直接附加在原待办项上,工程师在现有backlog工具内即可查看,无需切换工作流。
这对工程意味着什么
具体动作:做安全backlog过滤时,用F2或G-measure盯住召回侧,不要只看F1。要避开的捷径:别拿学术安全文本分类基准直接充当工业backlog评估,两者分布偏移会让跨域G-measure掉到约0.65。
证据定位
分类器域内F2=0.774,五个基准零样本G-measure均值约0.65,持平或优于经典机器学习和开源GPT基线。RAG从工业待办项检索出24条款,其中12条获两名从业者≥4/5评分。(筛选维度:可复核评测、软件工程方法)
适用边界
数据仅288条、来自两个公开Atlassian项目,领域代表性有限;RAG评估只有2名从业者和24条检索结果,不支持统计显著性结论;零样本跨域实验未覆盖工业私有backlog分布。
方法与英文摘要

数据:288条待办项取自两个公开Atlassian项目,9名从业者独立标注,Fleiss' κ=0.787。分类:以F2为目标的召回优先二分类器,域内用这套数据评估,跨域在Wu et al.的五个公开基准上做零样本迁移。RAG:四阶段管道,文档库为内部安全策略和CIS Benchmarks,对工业待办项检索条款,再由两名从业者评分。

Continuous software engineering in regulated domains requires engineering teams to address security throughout the development lifecycle. Yet making security requirements explicit in backlog items is still problematic. Engineers must instead infer security relevance of backlog items from brief, free-form descriptions and often lack timely guidance on applicable requirements. We present an NLP-based backlog enrichment system that detects security-relevant backlog items and links them to relevant security requirements. The approach combines a security-relevance classifier with a retrieval-augmented generation (RAG) pipeline over security requirements documents. The approach was developed and evaluated in the context of a large enterprise in highly regulated domains. We present three contributions. First, we release a dataset of 288 backlog items labeled for security relevance by nine security practitioners, with substantial agreement (Fleiss' $κ=0.787$). Second, a recall-oriented classifier achieving $F2=0.774$ in-distribution and mean zero-shot G-measure $\approx 0.65$ across five established benchmarks, matching or outperforming most published classical-ML and open-source GPT baselines. Third, we preliminarily evaluated a four-stage security requirements document-grounded RAG pipeline with two practitioners on industrial backlogs using company-internal security policies and CIS Benchmarks. Of the retrieved 24 clauses, 12 were rated at least 4/5 for relevance. Our findings provide first indicators that NLP-based product backlog enrichment can support engineers in identifying security requirements early in the development process. With this work we aim to facilitate continuous security compliance through proactive introduction of security requirements in continuous software engineering.

代码质量与优化(1 篇)

代码质量与优化 5/30

Prefactory: Automated Discovery and Application of Library-Adoption Refactorings

让LLM合成检测器:自动发现并应用Python库替换重构:Prefactory把LLM的能力花在一次性合成词法和结构检测器上,扫描阶段完全不调用LLM。在PrefactoryBench(100个真实重构,来自61个项目)上,函数级检出56个,最强基线为32个;其中40个重构通过项目测试和差分测试的双重验证。端到端成本低于每个实例$0.05。

两句看懂

静态linter只能套固定规则,直接用LLM逐文件扫描则成本高、结果难复现;Prefactory改成让LLM合成可执行检测器,扫描阶段零LLM调用。在PrefactoryBench的100个真实样本上,函数级检出56个,最强基线32个,其中40个通过双重测试验证。

核心判断

核心判断是:把LLM用于一次性合成检测器,而不是逐函数提示,就能在代码不引用目标库的情况下批量发现库替换机会。PrefactoryBench上函数级检出56个(基线32个),40个重构通过双重测试,成本低于$0.05/实例。

关键要点

1. 核心瓶颈在检测:旧代码往往不引用目标库,与库API只有行为相似、没有语法重叠,固定规则和直接LLM提示都难以泛化到这类机会;现有静态工具只覆盖手工预定义规则,逐函数提示LLM则成本高且难复现。 2. PrefactoryBench由61个开源Python项目的100个真实库采用提交构成,覆盖18个库;每个实例配项目已有测试套件,外加LLM生成的持出差分测试——以原始实现为oracle,且对Prefactory不可见,两层测试独立验证行为等价性,并与最强已有基线对比。 3. 结果:文件级检出75(基线35),函数级56(基线32),56个检出中40个通过双测试验证;扫描平均1.3秒、零LLM调用、成本低于$0.05/实例;目前只评测Python,检测器质量依赖LLM对库元数据和项目词表的理解,生成重构阶段仍需LLM参与。

证据与结果

评测基准为PrefactoryBench:100个真实库采用重构,来自61个开源Python项目,覆盖18个库,含numpy、pandas等三方库及标准库模块。每个实例配参考执行环境、项目测试套件,以及LLM生成的持出差分测试(对Prefactory不可见,仅用于最终评估)。评测看文件级和函数级检出数,并与最强已有基线对比:文件级75对35,函数级56对32;56个检出的重构中40个通过双重测试。扫描平均1.3秒,端到端LLM成本低于$0.05/实例。

打开论文原文
它要解决什么
当代码没有显式引用目标库、和库API只有行为相似而没有语法重叠时,怎样自动发现并验证Python项目的库替换重构机会?
研究路径
给定项目和目标库名,先收集库API元数据和项目词表,作为检测器合成的依据;LLM据此生成词法检测器(捕获函数名、变量名等词汇信号)和结构检测器(捕获控制流或计算结构),把语义判断固化进代码;检测器以普通代码执行扫描,因此没有LLM调用、平均只要1.3秒;候选按启发式排名,降低后续重构生成的开销;LLM为排名靠前的候选生成重构;项目已有测试和LLM生成的差分测试(以原始实现为oracle)共同验证行为等价性,两层测试独立,能确认重构结果可靠。
这对工程意味着什么
具体做法:推广库替换重构时,先用LLM生成扫描脚本,再批量执行扫描,而不是让LLM逐个文件审查;前者成本可控、结果可复现。要避免的捷径:逐文件提示LLM看代码——成本高、输出不稳定,难以在CI中重复运行。
证据定位
文件级检出75个(最强基线35个),函数级56个(最强基线32个);56个检出的重构中,40个通过项目测试加差分测试的双重验证;扫描平均1.3秒,零LLM调用,端到端成本低于$0.05/实例。(筛选维度:可复核评测)
适用边界
目前只评测了Python项目;PrefactoryBench仅含100个实例、18个库,库和项目类型的覆盖有限;论文摘录没有详细分析检测器排名策略的鲁棒性,以及它在更多库类型上的泛化能力。
方法与英文摘要

数据集是PrefactoryBench:100个真实库采用重构,来自61个开源Python项目,覆盖18个库,包括numpy、pandas、scipy等三方库,以及collections、itertools等标准库。每个实例带参考执行环境、项目测试套件和LLM生成的持出差分测试。流程:给定项目和目标库名,先收集库元数据和项目词表,作为LLM生成检测器的输入;LLM据此生成词法检测器和结构检测器,把语义判断提前固化进代码;检测器以普通代码执行扫描,因此没有LLM调用;候选按启发式排名,减少后续生成重构的工作量;LLM再为排名靠前的候选生成重构;

Replacing hand-written code with library API calls is a common refactoring that can reduce code size, make code more idiomatic, and reuse well-tested implementations. Yet many library-adoption opportunities are hard to find automatically: the original code often does not mention the target library and may resemble the library API only in behavior, with little syntactic overlap. Existing tools, such as linters and static modernizers, cover only a small set of manually specified patterns. LLMs and LLM-based agents, on the other hand, can generalize to more patterns, but they are costly, difficult to reproduce and to apply systematically at scale. This paper introduces Prefactory, an automated approach for library-adoption refactoring in Python. The key idea is to use an LLM to synthesize executable search heuristics rather than relying on repeated LLM prompting over a codebase. Given a target project and a target library name, Prefactory collects library metadata and project vocabulary, then generates lexical and structural detectors. Prefactory executes the detectors during a scan phase to find candidate functions. It then heuristically ranks the candidate functions, generates refactorings for the highest-ranked ones using an LLM, and validates the results with project tests and newly generated differential tests. We evaluate Prefactory on PrefactoryBench, a benchmark of 100 real-world library-adoption refactorings from 61 open-source Python projects and 18 libraries. Prefactory detects 75 instances at the file level and 56 at the function level, compared with 35 and 32 for the strongest baseline (Codex CLI). From the 56 detected functions, Prefactory produces 40 test-validated refactorings.

UI 与 GUI Agent(0 篇)

本轮没有通过深读证据门的重点论文。

个人知识与本体(1 篇)

个人知识与本体 5/30

Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings

先建事实再生成对话:长期记忆评测显示架构排名随历史时长反转:现有记忆基准先生成对话再从中提取答案键,LoCoMo 上有 6.4% 的标注错误,且评测以短历史为主。本文把顺序反过来:先用确定性采样器输出带有效期和来源渠道的事实,再由渲染器生成对话和邮件,并用约 380 道、15 种题型的问题做受控评测。

两句看懂

现有记忆基准先生成对话再提取答案键,既带 6.4% 标注错误又几乎只测短历史;本文改为先输出带有效期和来源渠道的事实再渲染文本,并用固定应答模型、版本化裁判和三次随机复现在三周/九周两个时段对比五种架构。短期领先的预算策展图九周早期召回率从 96.3% 跌到 72.2%,来源类型图升到 90.4%,六位用户的架构排名全部反转,跨家族裁判确认 p=0.031。

核心判断

记忆架构的排名会随历史时长反转:短期领先的预算策展图在九周时召回率从 96% 降到 72%,来源类型图升到 90%,六位用户上的排名全部反转(p=0.031),说明短历史基准会系统性误排长期助手真正需要的架构。

关键要点

1. 现有基准先生成对话再提取答案键,LoCoMo 记录 6.4% 标注错误,前沿模型在 MemGym 上靠预训练先验就能答对 70–85%;评测几乎只覆盖短历史,完整历史读取与记忆系统在此区间无法区分,短历史排名会系统性误排长期助手所需架构。 2. 合成语料库由确定性种子采样器先输出带有效期区间、波动性类别和来源渠道的事实清单,LLM 渲染成对话和邮件,保真度验证器逐条确认,问题机械生成约 380 道、15 种题型;受控变量包括固定应答模型、版本化裁判、三次随机复现、三周/九周双时段、六位虚构用户和跨家族裁判完整复裁。 3. 预算策展图短时段领先(94.2%),九周早期召回从 96.3% 降至 72.2%,来源类型图升至 90.4%,分层架构 93.2%,六用户排名全部反转(跨家族裁判 p=0.031);弱写入事实造成 24% 对 2% 的下游失败率差距;来源类型图和原始记录在全部 14 道注入问题 42 次试验中零无来源断言,平坦断言存储失败于部分探测。

证据与结果

合成语料库约 380 道验证问题、15 种题型,包含逐事实有效期区间、发送/接收信任区分、注入探测(14 道唯一问题、42 次答案试验)和 as-of-date 问题集。五种记忆架构加无记忆对照组,在三周和九周两个历史时段、六位虚构用户上测试,用固定应答模型和版本化 LLM 裁判,并做三次随机复现。短时段预算策展图 94.2%,分层架构 96.8%,完整历史基准并驾或超越最优记忆系统;九周时段预算策展图早期召回 96.3%72.2%,来源类型图 90.4%,分层架构 93.2%。弱写入下游失败率 24%,干净写入 2%;来源类型图和原始记录在全部注入探测中零无来源断言,平坦断言存储失败于部分探测。

打开论文原文
它要解决什么
当智能体积累数月的交互历史时,现有以短历史为主的记忆基准还能准确区分不同架构的长期排名吗?
研究路径
确定性种子采样器先输出带有效期、波动性类别和来源渠道的事实,LLM 渲染器再据此生成对话和邮件,保真度验证器逐条确认,因此金答案在脚本层就有效,并在渲染语料中另行验证可回答。写入质量需要单独检验,弱写入会把下游失败率从 2% 拉到 24%;评测层面固定应答模型、用版本化裁判、做三次随机复现,并以跨家族裁判完整复裁,六用户排名全部反转才被确认为 p=0.031。
这对工程意味着什么
具体行动:给记忆系统评测加上长历史时段(如九周),并把写入质量作为独立门禁;需要注入防护时参考来源类型图的“零无来源断言”结果。要避免的捷径:只用短历史排名选型,因为短历史下完整历史读取与记忆系统无法区分,长时段排名会反转;分层架构(无界来源存储+策展预算视图+聚焦检索)在两个时段都最优。
证据定位
短时段:预算策展图 94.2%,分层架构 96.8%,完整历史基准并驾或超越最优记忆系统。九周时段:预算策展图早期召回从 96.3% 降到 72.2%,来源类型图升至 90.4%,分层架构 93.2%。六位用户上的排名全部反转,跨家族裁判 p=0.031;弱写入事实造成 24% 下游失败率,干净写入只有 2%。(筛选维度:可复核评测)
适用边界
语料库完全合成且虚构化,只有约 380 道问题和六位用户;注入探测是非适应性的良性探针,没有测试自适应攻击;九周时段主裁判与跨家族裁判的结论存在分歧,没有与裁判无关的准确率胜者。
方法与英文摘要

按“事实→文本”的顺序构造评测:确定性种子生活脚本采样器先输出事实清单,每项事实带有效期区间、波动性类别和来源渠道;LLM 渲染器把事实清单转成对话和邮件;保真度验证器逐条确认植入事实;问题从脚本机械实例化,约 380 道、15 种题型。评测对象是五种记忆架构(向量存储、时序知识图、自编辑记忆、预算策展图、分层架构)加一个无记忆对照组,统一使用固定应答模型、版本化 LLM 裁判、三次随机复现、三周/九周双历史时段和六位虚构用户,并用跨家族裁判完整复裁。

Benchmarks for LLM-agent memory typically generate conversations first and extract answer keys afterwards -- with documented label-error and contamination problems -- and they overwhelmingly measure short interaction histories. We invert the pipeline: a seeded life-script sampler emits facts with validity intervals, volatility classes, and source channels before any text exists; an LLM renderer writes chat and email from per-event fact manifests; a fidelity verifier confirms every planted fact; and questions are instantiated mechanically from the script, so gold answers are script-valid by construction and separately validated for answerability. The synthetic, fictionalized corpus (~380 questions, 15 types) embeds features absent from the benchmarks we survey: per-fact validity intervals, sent/received trust distinctions, injection probes in a benign harness, and as-of-date question sets. Benchmarking five memory architectures against a no-memory control (fixed answerer, versioned LLM judge, three replicates, two horizons), we find backend rankings invert with history length: the budgeted curated-map memory that leads at three weeks loses recall of evicted content by nine weeks (96% to 72%) while a provenance-typed graph rises to 90%; the inversion is positive for all six users under complete cross-family re-judging (exact p=0.031). A full-rendered-history baseline ties or exceeds the best memory system at the short horizon but shows no judge-independent advantage at nine weeks, at about twice the read cost. Write-stage quality strongly correlates with downstream quality (weakly-written facts fail 24% vs 2%), and injection resistance tracked whether provenance boundaries survive representation. A layered architecture performs best among the memory systems in both regimes (96.8% short-horizon) and is released as Veracium, an open-source library, with the corpus generator and harness.

人机协同与对齐(0 篇)

本轮没有通过深读证据门的重点论文。

本轮分类概览

同一论文只归入一个最先命中的赛道,避免重复计数;“新增候选”只统计首次展示的论文。

赛道新增候选重点
形式化与程序验证52
软件工程与仓库智能73
代码质量与优化01
UI 与 GUI Agent00
个人知识与本体31
人机协同与对齐00

近一个季度监测日历

北京时间。绿色表示有可阅读的新候选,灰蓝表示已监测但无新增,橙色表示部分降级;“无记录”不等于失败。

2026 年 6 月

1无记录2无记录3无记录4无记录5无记录6无记录7无记录8无记录9无记录10无记录11无记录12无记录13无记录14无记录15无记录16无记录17无记录18无记录19无记录20无记录21无记录22无记录23无记录24无记录25无记录26无记录27无记录28无记录29无记录30无记录

2026 年 8 月

115候选 / 7重点2345678910111213141516171819202122232425262728293031

近 14 次监测窗口

仅展示公开源的聚合运行状态,不含提示词、全文或个人数据。

本轮新增候选(15 篇)

按赛道、评分和日期展开;中文标签用于导航,英文摘要用于核验。已展示过的旧候选不会每日重复。

形式化与程序验证(5 篇)

形式化与程序验证 · 6/30 · 2026-07-30Lean证明蓝图的类型化局部修复通过十种模式检查的局部编辑修复失效蓝图,构建含完全修复轨迹的142个受控失败基准。BlueprintRepair: Typed Local Edits for Failed Lean Proof Blueprints

LLM-based Lean proving systems increasingly organize a proof as a blueprint: a dependency graph of formal statements. We introduce BlueprintRepair, a repair interface that lets a model change this graph through ten schema-checked local operations. An operation names the node it edits, so the target theorem cannot be changed. Lean checks every applied change, and an accepted repair must declare every blueprint lemma its proof uses. We also construct BlueprintTrace, a benchmark of 142 controlled failures with complete accepted and rejected repair trajectories. We compare typed edits, exact source patches, and complete module rewrites under matched source, feedback, model, and budget, one episode per state and interface. With DeepSeek-V4-Flash, the three interfaces solve almost the same number of the benchmark's localized failures. Typed repair is the cheapest per solved state (patching is 1.30x as expensive, rewriting 2.06x), and within 10,000 completion tokens per task it reaches almost all of its final coverage, while both free-form interfaces are well behind. A second model, Qwen3.6-Flash, solves fewer states but keeps typed repair cheapest, puts it ahead on the proof-authoring states, and repeats the localized pattern.

阅读 arXiv 原文
形式化与程序验证 · 3/30 · 2026-07-30程序验证技能自进化框架将验证失败归因于技能缺陷,提炼可复用教训并迭代改进,仅采纳提升验证性能且保持语义的修订。VeriSkill: A Self-Evolution Framework for Program Verification Skills

Automating program verification with LLM agents requires generating specifications, annotations, auxiliary lemmas, and tool invocations, all of which depend on reusable skills. A natural remedy is skill self-evolution: distilling skills from trajectories and refining them through feedback. However, existing evolution methods struggle with program verification tasks because they cannot reliably identify skill-specific failures or extract actionable signals from opaque verifier feedback. In this paper, we propose VeriSkill, a self-evolution framework built for program verification. It attributes verification failures to skill deficiencies, distills diagnostic signatures into reusable lessons, and iteratively refines candidate skills, admitting only revisions that improve verification performance while preserving program semantics. Experiments show that VeriSkill consistently outperforms all baselines across multiple verification tools, agent frameworks, and LLM backends.

阅读 arXiv 原文
形式化与程序验证 · 6/30 · 2026-07-30增量ICE框架合成循环不变式将IC3增量思想融入ICE学习框架,以引理序列逐步学习并合成复杂程序的完整循环不变式。LimICE: Integrating LLM into ICE Framework for Efficient Loop Invariant Inference

Loop invariant synthesis is a fundamental problem in program verification, yet the inherent undecidability makes it highly challenging. Recent studies have increasingly employed various machine learning techniques to generate loop invariants. However, most of these methods adopt a monolithic approach. Due to the inability to strictly constrain the learning process, learning-based methods struggle to simultaneously consider all necessary conditions and generate complete invariants when tackling complex problems. In fact, a loop invariant is often an ordered sequence of lemmas, rather than a single invariant formula. This motivates us to propose Incremental ICE, a novel learning framework for incremental synthesis. Our framework integrates the incremental philosophy of IC3 into the general invariant learning framework ICE. By defining a lemma-specific learning objective and introducing a counterexample filtering mechanism, we can achieve sound incremental learning. Under this framework, we instantiate a loop invariant synthesis tool, LimICE, which leverages LLMs to generate the ordered sequence of lemmas and incorporates ICE-DT as a fallback mechanism to complement the lemma sequence. Experiments on 367 linear benchmarks and 50 nonlinear benchmarks demonstrate the effectiveness of the proposed approach. LimICE solves 349 (out of 367) linear problems on an average of 15.2 seconds and 47 (out of 50) nonlinear problems on an average of 8.8 seconds. Compared to the state-of-the-art LLM-based baseline, our approach solves 12-24% more instances while running 36-63% faster across linear and nonlinear benchmarks. LimICE also consistently outperforms strong non-LLM baselines and solves at least 86 and 27 additional instances on the linear and nonlinear benchmarks, respectively.

阅读 arXiv 原文
形式化与程序验证 · 3/30 · 2026-07-29概率论教材的Lean形式化将十四章概率论教材做成机器检查的Lean配套形式化,明确假设并复用概率相关基础设施。From Lecture Notes to Lean: Formalizing a Textbook on Probability Theory

As large language models become increasingly capable of generating mathematical arguments, mathematics is likely to face not a scarcity of proofs but an abundance of plausible ones. In such an environment, verification, exposition, and incorporation into reusable mathematical infrastructure become central tasks. We report on an ongoing Lean formalization of "Measure-Theoretic Probability: With Applications to Statistics, Finance, and Engineering", a fourteen-chapter upper-level undergraduate textbook covering topics from Riemann--Stieltjes integration to martingales and limit theorems. The project produces a machine-checked companion to the textbook and contributes reusable infrastructure for future formalizations involving probability theory. A Lean formalization provides computer-checked statements and proofs, makes hypotheses explicit, and allows readers to inspect the precise logical content of textbook results. A central challenge is to bridge textbook-facing statements with Mathlib's more general measure-theoretic interfaces. We reuse Mathlib results when possible and introduce reviewable interface lemmas when the textbook formulation and library abstraction differ. The project illustrates how formalized textbooks can support teaching, clarify mathematical assumptions, and help build the formal foundations needed for reliable AI-assisted mathematics.

阅读 arXiv 原文
形式化与程序验证 · 10/30 · 2026-07-29面向硬件验证的Lean4证明框架将参数化硬件设计与自然语言规范自动翻译为可执行Lean4模型,支持证明积累与参数化验证。CircuitProver: Agentic Lean 4 Theorem Proving with Reusable Circuit Proof Library for Hardware Verification

Modern integrated circuits (ICs) are becoming increasingly complex, making functional verification a major bottleneck. The dominant hardware formal verification methodology, model checking, verifies each design instance separately and exposes only pass/fail results, so the reasoning behind a proof stays locked inside solver heuristics and is repeatedly reconstructed across related designs. Interactive theorem proving instead yields explicit, reusable proof artifacts, but applying it to hardware remains largely manual, demanding expert effort for formalization, invariant discovery, and proof development. In this paper, we present CircuitProver, an agentic Lean 4-based verification framework supporting proof-accumulation and parameterized verification. CircuitProver automatically translates parameterized hardware designs and their natural language specifications into executable Lean 4 models. It then iteratively constructs machine-checked proofs through Lean feedback to establish that the hardware code complies with the specification. The proving traces and verified theorems are distilled into reusable libraries, where proving strategies guide future agent reasoning and verified lemmas support formal proof reuse across related hardware verification tasks. We further introduce the first benchmark suite for evaluating agentic hardware theorem proving, covering diverse parameterized hardware designs, specifications, proof tasks, and evaluation metrics. Across 63 tasks, CircuitProver successfully proves all benchmarks, while a vanilla agent solves 92.1% of them and requires twice as many proof rounds on average. Ablation studies show that accumulated proof knowledge reduces redundant proof construction across related verification tasks, reducing proof length by 16.3% and verification time by 23.2%.

阅读 arXiv 原文

软件工程与仓库智能(7 篇)

软件工程与仓库智能 · 11/30 · 2026-07-30变更至可执行编码代理任务与环境基于仓库历史将合并PR转化为健康新版上的可验证任务,用于训练与评测,减少环境搭建成本。Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification. To expand this supply, we present Change2Task, a system grounded in repository history that converts merged pull requests into verified tasks on healthy modern revisions of the same repository. It aligns historical evidence with evolved code, reconstructs task states through Patch Reversal, Code Mapping, or Agent Reconstruction, and validates the lifecycle from a healthy base to a task state and a restored state. By deriving multiple tasks grounded in developer evidence from maintained environments, Change2Task provides executable data for coding agent training and evaluation while reducing repeated environment setup, storage, and task construction effort. We evaluate the system through five common and widely adopted coding agent task families: Bug Fix, Feature Addition, Test Generation, Application Programming Interface Migration, and Security Repair. Starting from 1,130 source changes eligible for construction, Change2Task achieves 79.6% verified task construction success across these task families. On a matched candidate set, it recovers 29.2% more verified tasks than a construction baseline based on pull requests. Historical and reconstructed cases achieve up to 98.0% matched outcome agreement under agent evaluation, while reuse of modern bases reduces measured expenditure across the complete pipeline by 10.8%.

阅读 arXiv 原文
软件工程与仓库智能 · 6/30 · 2026-07-30识别SWE类基准中PR与问题错配研究发现SWE-bench Verified实例中13.6%存在错配,覆盖五种模式,并提出多智能体检查系统PAPAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.

阅读 arXiv 原文
软件工程与仓库智能 · 6/30 · 2026-07-30文本需求到微服务架构的LLM评估研究LLM仅凭自然语言需求生成完整微服务架构,评估结构一致性与结果感知质量。From Textual Requirements to Microservice Architectures - A Comprehensive Evaluation of LLM-Based Design Synthesis

Microservice architectures have become dominant for modernizing monolithic systems, yet identifying appropriate services remains challenging and largely manual. Existing decomposition approaches are predominantly code-centric, limiting applicability in early design stages where only textual requirements are available. Despite advances in Large Language Models (LLMs), limited empirical evidence exists on their ability to synthesize complete microservice architectures from natural-language requirements, including service definitions and inter-service interactions. This study investigates whether an LLM can bridge requirements engineering and architectural design, generating architectures solely from textual requirements and evaluating structural agreement and perceived quality of results. We conduct a mixed-method study using OpenAI o3 under zero-shot (ZS) and few-shot (FS) prompting across two systems (Bookstore, PetClinic), one execution per system/condition. Architectures are evaluated through (i) comparison with reference architectures using precision, recall, and F1-score for service identification and communication recovery, and (ii) a blinded expert assessment of correctness, completeness, modularity, and plausibility, plus open feedback synthesis. OpenAI o3 identifies services with higher agreement under FS prompting (F1 = 0.79 for ZS versus = 0.97 for FS). Communication recovery is more challenging: ZS produces dense architectures with high recall but low precision (F1 = 0.61), while FS improves agreement, reaching F1 = 0.82 and reducing unsupported dependencies. Expert evaluation corroborates these results, with FS architectures perceived as more modular, coherent, and plausible than ZS outputs. OpenAI o3 shows potential for requirements-driven synthesis when guided by exemplar prompting. Results are model- and context-specific from two small systems, not model-independent proof.

阅读 arXiv 原文
软件工程与仓库智能 · 7/30 · 2026-07-30AI融入需求质量教学实证研究TPACK指导下将多智能体AI工具融入需求质量作业,学生主要将其用于分析评价而非自动化。Integrating AI into Requirements Quality Learning in Software Engineering Education: A TPACK-Guided Empirical Study

The rapid adoption of generative Artificial Intelligence (AI) in software engineering (SE) practice creates a need for pedagogically grounded approaches to AI integration in SE education, especially in conceptually intensive subjects such as requirements engineering (RE). This study examines a TPACK-guided integration of a multi-agent AI tool into a master-level RE assignment on requirements quality analysis. Using a mixed-methods design (N=100; 72 submissions analysed), we examine how structured assignment design shaped students' AI use, affected their understanding of user story quality criteria, and influenced their perceptions of AI's benefits and limitations. Results show that students used the AI tool selectively, mainly as support for analysis and evaluation rather than automation. Alignment improvements were most evident for structurally concrete requirements quality dimensions, such as value articulation and testability, while negotiability showed mixed effects. Students reported conditional trust, active refinement, and increased awareness of quality criteria, alongside moderate usability challenges. The findings show that TPACK-guided scaffolding can align AI affordances with pedagogical goals and RE content, offering design guidance for responsible AI integration in RE education.

阅读 arXiv 原文
软件工程与仓库智能 · 8/30 · 2026-07-29编码代理非功能改进评测基准含188个真实合并PR任务,将开发者导向的非功能改进转为92条可执行规则并评估代理表现。SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements

Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developer-oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0\% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents' NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.

阅读 arXiv 原文
软件工程与仓库智能 · 8/30 · 2026-07-29待办项安全需求链接增强系统利用分类器与检索增强生成,将安全相关待办项关联到安全需求,并发布288条标注数据集。From Backlog Items to Security Guidance: Towards Continuous Security Compliance

Continuous software engineering in regulated domains requires engineering teams to address security throughout the development lifecycle. Yet making security requirements explicit in backlog items is still problematic. Engineers must instead infer security relevance of backlog items from brief, free-form descriptions and often lack timely guidance on applicable requirements. We present an NLP-based backlog enrichment system that detects security-relevant backlog items and links them to relevant security requirements. The approach combines a security-relevance classifier with a retrieval-augmented generation (RAG) pipeline over security requirements documents. The approach was developed and evaluated in the context of a large enterprise in highly regulated domains. We present three contributions. First, we release a dataset of 288 backlog items labeled for security relevance by nine security practitioners, with substantial agreement (Fleiss' $κ=0.787$). Second, a recall-oriented classifier achieving $F2=0.774$ in-distribution and mean zero-shot G-measure $\approx 0.65$ across five established benchmarks, matching or outperforming most published classical-ML and open-source GPT baselines. Third, we preliminarily evaluated a four-stage security requirements document-grounded RAG pipeline with two practitioners on industrial backlogs using company-internal security policies and CIS Benchmarks. Of the retrieved 24 clauses, 12 were rated at least 4/5 for relevance. Our findings provide first indicators that NLP-based product backlog enrichment can support engineers in identifying security requirements early in the development process. With this work we aim to facilitate continuous security compliance through proactive introduction of security requirements in continuous software engineering.

阅读 arXiv 原文
软件工程与仓库智能 · 3/30 · 2026-07-29人机协作团队需求分类体系从14个跨领域来源提取361条需求,形成6大类21小类的人机协作需求层级分类法。A Taxonomy of Human-Robot Teamwork Requirements

Autonomous systems are increasingly deployed in safety- and mission-critical domains where humans and robots must operate as a team to complete complex tasks. Existing requirements for Human-Robot teamwork remain fragmented across disparate sources, with no unified framework that addresses complexities of collaborative Human-Robot tasks. We address this gap by presenting a taxonomy of Human-Robot Teamwork (HRT) requirements derived from analysis of (academic and industrial) literature, standards and regulatory guidance. We extracted a construction corpus of 361 requirements from 14 cross-domain sources. Through iterative classification and refinement, we develop a two-level hierarchical taxonomy comprising 6 high-level categories and 21 low-level subcategories that distinguish information provision, relational control, decision support, safety mechanisms, performance monitoring, and foundational system capabilities. We validate the taxonomy through expert evaluation with 5 domain specialists and a utility demonstration on an independently assembled corpus of 448 requirements drawn from 19 sources spanning six HRT domains.

阅读 arXiv 原文

代码质量与优化(0 篇)

本轮该赛道没有候选论文。

UI 与 GUI Agent(0 篇)

本轮该赛道没有候选论文。

个人知识与本体(3 篇)

个人知识与本体 · -3/30 · 2026-07-30反思式检索记忆用于多模态长程推理在实体中心多模态记忆图中增加反思经验记忆,从历史轨迹提炼跨任务可复用检索策略。RRM: Experience-Driven Reflective Retrieval Memory for Long-Horizon Multimodal Reasoning

Existing multimodal long-term memory agents use external memory to overcome the limited context available for long videos. However, most methods emphasize what to store rather than how stored memory should be retrieved. When retrieval becomes inaccurate or repeatedly fails to obtain useful evidence, existing agents lack mechanisms to diagnose failures from previous task trajectories and adapt future search strategies.We introduce Reflective Retrieval Memory (RRM), a reflective memory framework for long-horizon multimodal reasoning. RRM augments an entity-centric multimodal memory graph with reflective experience memory, which distills transferable procedural retrieval knowledge from historical task trajectories. Unlike episodic and semantic memories that preserve factual evidence from the current video, reflective experience memory captures reusable search strategies across tasks. RRM converts retrieved experiences into query-level guidance, while answer generation remains conditioned only on factual evidence newly retrieved from the current video. A lifecycle management mechanism further regulates experience memory through usage frequency, reuse feedback, and temporal decay, thereby reducing redundancy and noise. RRM consistently outperforms previous state-of-the-art approaches on M3-Bench-Robot, M3-Bench-Web, and Video-MME-Long, demonstrating the effectiveness of reflective retrieval memory for long-horizon multimodal reasoning.

阅读 arXiv 原文
个人知识与本体 · 0/30 · 2026-07-30代理记忆事务边界与状态恢复在答案模型外验证更新来源、解决事实冲突并恢复故障后状态,审计显示可区分支持更新与硬负例。MemTxn: A Transaction Boundary for Source-Supported Updates and Complete-State Recovery in Agent Memory

Persistent memory lets long-running large language model agents reuse information across sessions and tasks. Yet errors in writable memory can persist and corrupt future behavior. Existing systems improve storage and retrieval, but they do not provide a transaction boundary for reliable updates and recovery. We therefore propose MemTxn, a governance layer outside the answer model. MemTxn verifies whether an update is supported by its source. It also selects the visible version when facts conflict and restores the application-visible state after a fault. The system uses Ordered PatchTest to validate writes, a Temporal Resolver to select versions, and a durable snapshot journal to recover state. On an item-disjoint audit, MemTxn accepts all 60 supported originals and rejects all 179 hard negatives. Under persistent multi-key faults on LongMemEval-S and LoCoMo states, it restores the complete declared active map without knowing the actual physical write set. On MemoryAgentBench FactConsolidation, MemTxn achieves the highest average F1 across all twelve answer-model configurations. It outperforms Dense by 17.06--24.07 points in five representative settings.

阅读 arXiv 原文
个人知识与本体 · 5/30 · 2026-07-30代理记忆版本控制与语义回滚每次记忆写入提交整体快照,维护版本历史,并用混合检索将自然语言回滚意图映射到历史版本。ChronoMem: Version Control and Semantic Rollback for Large Language Model Agent Memory

LLM agents increasingly rely on long-term memory to support multi-session interaction and personalization. However, existing agent memory systems are designed around forward-only evolution, continuously accumulating, consolidating, and overwriting knowledge, with no principled mechanism to inspect, version, or revert prior states. This makes agents brittle under corrections, concept drift, and memory corruption, particularly after they have already been exposed to subsequent information. We present ChronoMem, a semantic version-control layer for agentic memory integrated into the production-ready, open-source Agent Development Kit by Google. ChronoMem commits whole-memory snapshots at each memory write, maintains structured version histories, and supports natural-language rollback requests by mapping undo intents to concrete historical versions through hybrid lexical and semantic retrieval, rank fusion, and reranking. We further introduce a post-exposure evaluation protocol that tests whether an agent can behave counterfactually after rollback by answering queries and summarizing history as if future updates had never occurred. On long-horizon conversational benchmarks augmented with evolving memory states and rollback tasks, ChronoMem substantially improves rollback-consistent question answering and history summarization relative to prompt-only and retrieval-only baselines, while achieving strong performance in semantic version selection. To our knowledge, ChronoMem is the first open-source system and benchmark for systematic semantic global memory rollback in LLM agents.

阅读 arXiv 原文

人机协同与对齐(0 篇)

本轮该赛道没有候选论文。