PhD Position “End-user Programming and Large Language Model”

(note: it is also possible to apply for an internship!)

Is it possible to program non-trivial applications and customize code without knowing much about programming? Impressive showcases of Large Language Models (LLMs) like ChatGPT suggest a positive answer [1]. Some people have even claimed that programmers will be replaced or disappear [2]. So, can end-users create working and non-trivial applications, like games, just using instructions in natural language with ChatGPT?

In prior experiments [3], we considered the game Flappy bird, using Python. We have created 35 sessions, with prompts, code, observations, and results (videos) with ChatGPT-3.5 and ChatGPT-4. Our conclusion is as follows: Formulating a prompt and systematically getting a comprehensive and playable game is not yet a reality. ChatGPT can provide impressive results, but not all times. A magic prompt is missing to make ChatGPT reliable. Many interactions are rather needed to fix issues or expand the features, sometimes in non-technical terms and with instructions out of observations of the current game. But the control of the code is not that far and seems inevitable.

In this internship, we’re looking for solutions to improve the situation. The direction we’d like to explore is to improve the usability and the integration of ChatGPT ouputs into development environment. The back and forth between the IDE and ChatGPT is time-consuming. Moreover, some (informal) instructions of ChatGPT can be automatically applied onto the code base, limiting the user effort or the technical expertise required. The hope is to have a better feedback-loop.

Hence, an approach is to develop agents capable of interacting with tools and data (compilers, interpreter, error message, etc.) in an automated manner [4]. We also plan to consider a multi-modal approach where the end-user feedbacks and visual of the resulting code is leveraged. For instance, in case of Flappy bird, one can envision to execute the program, and analyze the outcome (difficulty of playing, diversity and consistency of visual artefacts, etc.) to then improve it.

An interesting application area concerns Web applications, where technical barriers limit contributions of end-users (or even developers)… We envision to have an end-user “IDE” boosted by LLM and agents.

[1] Bubeck et al. Sparks of AGI: early experiments with GPT-4 https://arxiv.org/abs/2303.12712 [2] The End of Programming By Matt Welsh Communications of the ACM, January 2023, Vol. 66 No. 1, Pages 34-35 10.1145/3570220 [3] https://github.com/diverse-project/enduserprogrammingLLM/ [4] Communicative Agents for Software Development by Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, Maosong Sun, 2023 https://arxiv.org/abs/2307.07924v3

Objectives

The outcomes can well be, but are not limited to:

• a review of existing works about end-user programming, LLMs and agents for software development
• innovative approach and architecture to orchestrate LLMs and agents together with humans’ feedbacks (prompts) and traditional tools (compiler, interpreter, etc.)
• reproducible experiments on a set of scenarios (eg Flappy bird)
• assessment of the proposal

Environment

The work will be realized in the DiverSE research team, joint to the CNRS (IRISA) and Inria. The DiverSE team is located in Rennes, Brittany, France. DiverSE’s research is in the area of software engineering. The team is actively involved in European, French and industrial projects and is composed of 9 faculty members, 20 PhD students, 4 post-docs and 3 engineers. The candidate will work more specifically with Prof. Mathieu Acher (University of Rennes, DiverSE team).

Skills and background

• Experience in software development and engineering
• Autonomy, and the ability to work in a distributed and international group.
• Fluent in English