These days (Jan 2025) one can hear or read a lot of news stories and announcement about Artificial Intelligence and Large Language Models.
Something we see much less on this topic is concrete, solid, experiment reports.
In particular, it is frequently claimed that AI will soon replace or is already replacing programmers and testers in the workplace.
Such a claim can provoke all kinds of reactions among people whose job is to create, test, maintain and evolve programs.
It seems to beg the question, since we know that AI tools can't think : is it necessary to think in order to perform a programming or a testing task ? If thinking is not required, then indeed, computers will probably eventually replace all programmers.
Note that while there is no doubt that AI is already helping a lot of programmers (as everyone can experience everyday with the current technology), the question asked here is rather : to what degree can we replace programmers in a given project with AI ?
The best way to answer this question is not to make more speculations or claims about a near future state of capacity for AI tools, but simply to experiment.
Here's a simple project on which one can experiment and maybe assess the capacity for AI tools to replace a programmer.
In a terminal, type
make bbr
You will need a C compiler. The library mos6502 (created by Gianluca Ghettini) which has been adapted for simple I/Os resides in the lib folder.
Your task is to study this program code and/or its execution, and answer the following questions:
- what is this program doing?
- what rules or specs does it follow?
- can you find bugs in this program regarding the rules or specs you have identified?
- what rules could be changed and new results obtained from the program by modifying only the input data ?
- how much time did you spend on this exercise?
- what approach did you put in place? What tools did you use?
or if you are one, but think that AI tools can replace a programmer or a tester, use your best AI tools available for the task to answer the following questions :
- what is this program doing?
- what rules or specs does it follow?
- can you find bugs in this program regarding the rules or specs you have identified?
- what rules could be changed and new results obtained from the program by modifying only the input data ?
- what information or knowledge did you manage to acquire about the program?
- was the IA tool able to provide reliable and verifiable answers to the questions?
- how much time did you spend on this exercise?
What do you think of this assessment experiment?
- I welcome and will appreciate any feedback on the exercise
- I will not partipate in discussions about AI capacity with people having not tried the exercise
Enjoy !