Kontakt
stefan.bente[at]th-koeln.de
+49 2261 8196 6367
Discord Server
Prof. Bente Personal Zoom
Adresse
Steinmüllerallee 4
51643 Gummersbach
Gebäude LC4
Raum 1708 (Wegbeschreibung)
Sprechstunde nach Vereinbarung
Terminanfrage: calendly.com Wenn Sie dieses Tool nicht nutzen wollen, schicken Sie eine Mail und ich weise Ihnen einen Termin zu.

GP »ChatGPT in Software Engineering Academic Education. Fraud Tool, or useful Helper? An Empirical Study« (SS23)

In this project, participants will investigate the potential and reliability of large language models like ChatGPT, their impact on education, and possible malicious (fraud) and constructive (fraud detection, explanation, writing helper, …) applications in the educational domain, focusing on software engineering education.

Keywords
artificial intelligence, chatgpt, software engineering, academic education
Study Program and Module Description
Master Digital Sciences (specialization Software Architecture) (see also module description in the study program web site)
Begin/End and Scheduling
20.03.2023 - 31.08.2023. Organized as an agile project with sprints every 2 weeks. Workload depends on the type of project, will be clarified at kickoff.
Location
The module takes place in a hybrid format. Please refer to the respective time slot or workshop day (see below on this page), to check if it is online or in presence. If there are no details given on this page, please check the other communication channels (e.g. Discord).
Room 1: 1708 (Building LC4 (library building). Room 1708 is our lab office space, with enough room for small workshops. However, the room is only accessible with by transponder. Thereore, please be at the door left of the library entrance (where you see the label "Kriminalkommissariat", since the Gummersbach police criminal investigation unit is our office neighbour :-). I will fetch you there. Call me (+49 176 8072 2689) or a fellow student by mobile phone in case you should run late), see also detailed directions
Room 2: Innovation Hub (Building LC7, the building to the right of the passage to Gummersbach station. The Innovation Hub is not not open to the public. However, the main entrance door is open during office hours. Come up to the second floor and ring the bell there. Please call me (+49 176 8072 2689) or a fellow student by cell phone if no one answers), see also detailed directions
Video conference link: https://th-koeln.zoom.us/j/4425088059?pwd=bxHOJB7D3pBHdTpruFHSdC1Y66e9Rr.1
Discord Server for fast Communication
Discord has been proven as a very effective platform for information sharing, discussions, and consultations. Therefore, please join the ArchiLab Discord Server at https://discord.gg/YYNYb5whU8. Navigate to channel #rollenzuweisung and click on gp. Then you automatically get access to the channel(s) relevant for this module.

Problem Description

ChatGPT has been attracting attention quickly in the recent months. With its introduction, OpenAI has developed a model that can understand and produce remarkable results, based on natural language. According to new research, large language models can generate coherent and meaningful text, even for scientific articles. Many studies are concerned about the destructive and malicious potential of these AI models, such as the generation of whole fake news articles, internet troll farms, fake scientific papers, or students cheating on exams.

ChatGPT as a fraud tool

Scientific documents, seminar assignments, and even whole software snippets can be generated by ChapGPT, seemingly very close to artifacts written by real humans. This makes ChatGPT a serious threat to the current way of education in school and university. Numerous questions arise, such as:

  • Practicals (Praktika), homeworks and manual assignments often serve as an ungraded precondition (unbenotete Vorleistung), before a student is allowed to write an exam. In a chronically under-resourced academic teaching context, such practicals and assigments cannot always be scrutinized thoroughly enough. Are they still a relevant means to make students study for an exam – if ChatGPT just generates the proper solutions?
  • Is it still advisable to allow open-book exams or home-based online tests, where the use of ChatGPT cannot be ruled out completely?
  • In Master study programs, where a seminaristic teaching style prevails, the grading is often based on written papers and project documentation, rather than written exams. Is this still a fair way of grading students, if ChatGPT can help writing the paper?

The potential of ChatGPT as a fraud tool needs to be analysed closely, in order to decide WHICH kind of assignment is problematic wrt. the use of ChatGPT. Are there types of assignments where supervisors can still recognize a fraud attempt by using ChatGPT? Are there, on the other hand, properties make an assignment or task “hard” or even “impossible” to cheat on using ChatGPT? This question will be analyzed in this Guided Project by conducting an empirical test. In order to narrow the field of analysis, the focus will be on IT and especially software engineering education, with inclusion of some other fields of study for comparison.

ChatGPT as a potential helper

On the other hand, there may be ways to use ChatGPT’s capabilities in a constructive and helpful manner in academic education, e.g. by

  • integrating such an AI model into the education system to evaluate essential assignments, or
  • as an explanation tool for students.

Studying such a way of constructively using ChatGPT in academic education is also part of the project.

Project Definition

In this project, participants will investigate the potential and reliability of large language models, their impact on education, and possible malicious (fraud) and constructive (fraud detection, explanation, writing helper, …) applications in the educational domain, focusing on software engineering education. As a final result, the students will incorporate their findings into a scientific paper that will be submitted to a journal or conference on software engineering education.

Research

The participants will research the potential and limitations of such large natural language models, based on an exhaustive evaluation of existing scientific studies.

Experiment Design

As one of the first steps, the participants will work on an research experiment design. The project will try to win a number of professors in IT education to cooperate, so that a blind study can be realized: The professors will grade some ChatGPT-generated solutions alongside solutions written by students, without knowing which is which. For comparison, some professors from other fields of study will also be asked to cooperate, so that a comparison between IT assignments and other fields can be done.

Data Gathering

The project will analyze if ChatGPT-generated solutions will pass the test, and to what percentage they can be spotted and distinguished from human-written solutions. The participants will develop and try various “fraud strategies” – from just copy-pasting the initial ChatGPT answer, up to a thoroughly optimized solution.

Evaluation and Paper Writing

As the final project step, the findings are evaluated and summarized in a scientific paper to be submitted to a fitting journal or conference.

Learning Outcomes

During this guided project, the students will . . .

  • acquire a deeper understanding of the way large language processing works, and how AI models can generate text
  • learn how to design and conduct an empirical experiment, in order to analyze the impact of AI on academic education in regards to
    • the opportunities it offers
    • the problems it presents
  • apply proper scientific work methods to write an academic paper

Requirements

  • Participants need a deep familiarity with IT and software engineering education, in order to prepare the “fraud” solutions in the study. This includes practical coding knowledge.
  • Participants must be interested in AI methods. Ideally, they have pre-existing knowledge in that area.
  • Participants must have proven ability to write scientific papers and reports. If you haven’t published a paper yet (which only very few students will), an excellent bachelor thesis with a strong academic research part, or other larger reports with similar properties written in Bachelor or Master will count here. Since the seats in this project are strictly limited (otherwise the way of working will not be effective), and a high interest in this project is to be expected, it is possible that interested applicants will have to provide a 3 min video on their motivation, in case of over-booking.

External Project Partner

-

Grading

The grade for this GP will consist of the following parts:

  • 80% quality of your contribution to the project
    • quality of your research
    • commitment and effort (willingness to go the extra mile, to make create a great result)
    • professionalism in sprint reviews and creativity in planning
  • 20% final presentation