LLM-based Test Generation - Does It Work?

Lassen sich mithilfe von LLM-generierten Tests effektiv Fehler finden? Oder validiert man damit unbeabsichtigt fehlerhaften Code? Angesichts der explodierenden Zahl von LLM-gestützten Coding- und Testing-Tools werden wir ein Experiment aus einem Paper von Mathews & Nagappan (2024) nachstellen.

• Objective
• Background
• Research Question(s)
• Sources

Objective

Given the discruptive role of AI in software development, this topic explores the question if LLM-based test generation really makes sense.

Background

Base of this topic is the paper by Mathews & Nagappan (2024), see below. Essentially, the team should replay the experiments (as far as possible), and back them up by research to come to its own conclusion.

Research Question(s)

• Does LLM-based test generation really make sense?
• If yes, what needs to be taken care of?

Sources

• Mathews & Nagappan (2024): Design choices made by LLM-based test generators prevent them from finding bugs