MSR 2026 Mining Challenge paper

Our paper “When is Generated Code Difficult to Comprehend? Assessing AI Agent Python Code Proficiency in the Wild” was accepted in the Mining Challenge track of MSR 2026, held in Rio de Janeiro, Brazil, in April 2026. AI coding agents now write much of the code that developers review, so the team asked how advanced that code is. They mined 591 pull requests with 5,027 Python files written by three AI agents and rated each construct with pycefr, on six levels from A1 (Basic) to C2 (Mastery). Over 90% of the constructs fell into the two basic levels and under 1% reached Mastery, a profile close to that of human pull requests. Nanthit Temkulkiat, the first author, was a 2025 summer intern at The University of Osaka.

• Most AI-generated code is basic, so developers with basic Python skills can review it.
• The most advanced code tends to appear in feature additions and bug fixes.
• Reviewing and maintaining that code still calls for advanced proficiency.