What is it about?
We have developed a novel approach to automatically test and find bugs in Virtualization Systems by integrating ChatGPT with two proven methodologies: search-based software engineering and fuzzing. In essence, we leverage ChatGPT to generate parameterise C programs. We compile the resultant code snippets and feed compilable ones to SearchGEM5, our extension to AFL++ fuzzer with customised new mutation operators. For our evaluation, we selected gem5, a system simulator commonly used in the software and chipset industries, as the system-under-test (SUT). We tested the reliability of gem5's simulation in X86 of compilable code snippets prepared via ChatGPT (aka software or binaries). We ran thus created 4005 binaries through our system under test, gem5, increasing its existing test coverage by more than 1000 lines. We discover 244 instances where the gem5 simulation of the binary differs from the binary’s expected behaviour. The dataset containing these bugs has been shared on Zenodo for further analysis and collaboration (DOI 10.5281/zenodo.8316685).
Featured Image
Photo by Markus Spiske on Unsplash
Why is it important?
In this work, we contribute to enhancing the reliability and quality of system simulators, offering a more efficient and effective testing approach and bug detection. This ultimately leads to higher quality and more dependable virtualization systems, which themselves are used to expedite the system development process by aiding in the detection, reproduction, and repair of flaws in the system.
Perspectives
Read the Original
This page is a summary of: SearchGEM5: Towards Reliable Gem5 with Search Based Software Testing and Large Language Models, December 2023, Springer Science + Business Media,
DOI: 10.1007/978-3-031-48796-5_14.
You can read the full text:
Resources
SearchGem5: Finding Bugs in Virtualization Systems with Fuzzers and ChatGPT
Endless Science's talk by Hector Menendez about SearchGem5.
SearchGem5: Finding Bugs in Virtualization Systems with Fuzzers and ChatGPT
The SearchGem5 GitHub project, where you can download and test gem5 yourself!
Artifact of SearchGem5: Finding Bugs in Virtualization Systems with Fuzzers and ChatGPT
The Zenodo record of the SSBSE 2023 paper including code and data.
Contributors
The following have contributed to this page