- As artificial intelligence increasingly integrates into software development workflows, the demand for objective evaluation of its capabilities becomes paramount.
- One study, focusing on AI’s role as code reviewers, utilized a meticulously prepared dataset of 60 known defects.
- Another experiment concentrated on AI’s ability not just to find but also to fix bugs.
AI in Software Development: Assessing Reviewer and Fixer Efficacy
As artificial intelligence increasingly integrates into software development workflows, the demand for objective evaluation of its capabilities becomes paramount. PROSTO24 conducted a series of experiments to measure the effectiveness of AI models in detecting and fixing software defects, moving beyond subjective impressions to concrete numerical data.
Evaluating AI Reviewers on 60 Known Bugs
One study, focusing on AI’s role as code reviewers, utilized a meticulously prepared dataset of 60 known defects. These defects, manually labeled long before the experiment, included 51 test holes, 8 equivalency issues, and 1 debatable case. Four different AI models were tested in two modes: with and without prompts. The analysis considered not only the defects found but also the amount of noise generated on untouched files, providing a comprehensive view of their performance.
AI Fixers: Repairing a Single Broken Script
Another experiment concentrated on AI’s ability not just to find but also to fix bugs. For this, a single broken bash script was used, containing seven real defects—ranging from cosmetic issues like outdated filenames in comments to critical flaws such as a summary table consistently displaying zeros regardless of actual events. Four AI models independently attempted to repair this script. Subsequently, seven additional models received all four repair variants and were tasked with assembling the best possible fix from them.
Discrepancies and AI Blind Spots
Preliminary results from both experiments revealed significant discrepancies in the models’ performance. No single model managed to find or fix all defects, and each exhibited its own distinct ‘blind spots.’ This underscores that even the most advanced AI models are far from perfect and demonstrate unique error patterns. Such empirical data is crucial for understanding the strengths and weaknesses of AI in software development and for guiding future technological advancements.
The PROSTO24 findings on AI efficacy in defect detection and repair are critical, particularly the observed ‘blind spots’ across various models. This highlights the inherent limitations in current LLM architectures for static code analysis, especially regarding semantic understanding versus syntactic pattern matching. The variance in performance across 51 test holes versus 8 equivalency issues suggests a need for more nuanced training data focusing on logical inconsistencies rather than just surface-level anomalies. This data is invaluable for refining AI-driven SDLC tools, pushing towards more robust, context-aware solutions rather than relying solely on generalized models.