What is it about?

We have developed a novel approach to automatically test and find bugs in Virtualization Systems by integrating ChatGPT with two proven methodologies: search-based software engineering and fuzzing. In essence, we leverage ChatGPT to generate parameterise C programs. We compile the resultant code snippets and feed compilable ones to SearchGEM5, our extension to AFL++ fuzzer with customised new mutation operators. For our evaluation, we selected gem5, a system simulator commonly used in the software and chipset industries, as the system-under-test (SUT). We tested the reliability of gem5's simulation in X86 of compilable code snippets prepared via ChatGPT (aka software or binaries). We ran thus created 4005 binaries through our system under test, gem5, increasing its existing test coverage by more than 1000 lines. We discover 244 instances where the gem5 simulation of the binary differs from the binary’s expected behaviour. The dataset containing these bugs has been shared on Zenodo for further analysis and collaboration (DOI 10.5281/zenodo.8316685).

Featured Image

Why is it important?

In this work, we contribute to enhancing the reliability and quality of system simulators, offering a more efficient and effective testing approach and bug detection. This ultimately leads to higher quality and more dependable virtualization systems, which themselves are used to expedite the system development process by aiding in the detection, reproduction, and repair of flaws in the system.

Perspectives

SearchGem5 is an innovative testing tool for virtualization systems, that emerged from a collaboration between computer scientists from King's College London (KCL) and University College London (UCL) during The 64th CREST Open Workshop - SSBSE Challenge Track: Collaborative Jam Session (COW 64). The work won the 15th International Symposium on Search-Based Software Engineering (SSBSE): Challenge Track Award. The paper's accompanying 13 GB artifact on Zenodo gained significant attention, with 130 downloads recorded as of March 27, 2024. I personally enjoyed the research work, swiftly designing solutions for challenges, struggling with bad memory manipulations and investigating bugs in binary inputs, along with Engaging in discussions with gem5's engineers contributed significantly to my growth, and I felt a sense of fulfilment in identifying and reporting issues in an important system as gem5.

Dr Karine Even-Mendoza
King's College London

Read the Original

This page is a summary of: SearchGEM5: Towards Reliable Gem5 with Search Based Software Testing and Large Language Models, December 2023, Springer Science + Business Media,
DOI: 10.1007/978-3-031-48796-5_14.
You can read the full text:

Read

Resources

Contributors

The following have contributed to this page