What is it about?

This paper presents an empirical study on 4,066 ChatGPT-generated code snippets in Java and Python, scrutinizing their correctness, quality, and potential challenges. # Key Insights: - ChatGPT's performance significantly decreases when faced with more challenging and recently introduced tasks. - Code quality issues are prevalent, even in functionally-correct code. - Wrong Outputs and Code Style & Maintainability issues are the most common challenges faced by the ChatGPT-generated code - Like humans, ChatGPT can even counter simple mistakes such as unused variables or division by zero. # Mitigation Strategies: - Feedback from Static Analysis Tools resolves 20% of issues in the first round. - Iterative repairing with feedback proves effective, saturating after 4 rounds with a fixed rate of around 50%. - While promising, it is far from perfect, paving the way for future research in this direction.

Featured Image

Why is it important?

ChatGPT holds great promise in revolutionizing Software Engineering, especially in Code Generation. However, concerns persist around the reliability and quality of ChatGPT-generated code. Our empirical study carefully examines 4,066 ChatGPT-generated code snippets in Java and Python, scrutinizing their correctness, quality, and potential challenges.

Perspectives

Despite the promise of ChatGPT for code generation. We found that ChatGPT-generated code contains many code quality issues, which might compromise the reliability of software systems. Our efforts extend beyond academic boundaries, seeking to raise awareness within the community regarding quality issues in ChatGPT-generated code, promoting its responsible usage.

Thanh Le Cong
University of Melbourne

Read the Original

This page is a summary of: Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues, ACM Transactions on Software Engineering and Methodology, June 2024, ACM (Association for Computing Machinery),
DOI: 10.1145/3643674.
You can read the full text:

Read

Contributors

The following have contributed to this page