Initial prototype

[yassine-cc]

No description provided.

[ianjennings]

Awesome. Here is a loom with some feedback:
https://www.loom.com/share/2e91aa5091654c7aa56b00e4fd315ea9

Some changes needed

• Both of these may be solved by running in "terminal window" instead of spawn, but I know there's problems with detecting exit code and such
  • TD_VM=true should show the electron process when running.
  • When running tests, it would be cool to see the output
• I should be able to stop a running process
• AI should respond to general commands (I made edits for this already)
• Running a codeblock shouldn't produce a file in the current dir
• Each CoPilot chat window should have a single TestDriver instance. So if I run a block twice, it should happen in the same instance. If I make a new block, it should run in the same instance. If I make a new chat window, it should have a new instance.
• Should be able to click "play" button on steps and commands in YAML files

[ianjennings]

https://www.loom.com/share/a19f38daa5bb47b7b559aeb969f2d090

Great great job. Almost there.

• @testdriver is not working in GitHub Copilot chat, but that might be a vscode bug. It does work in "quick chat"
• I know you said this is hard, but the "one instance per chat" thing is needed. Even if it's a single global "development" instance, that works too. The user needs to be able to do a command, try again, do another command, in serial sequence on the same VM

Otherwise I think we're close to releasing this!

[ianjennings]

I've figured out that I had to toggle GitHub copilot chat to "ask" instead of "edit." Now @testdriverai appears there.

[ianjennings]

I can not get the parser to work properly, but everything else does!

https://www.loom.com/share/e5411a708fd840199ecb19d6f4fbedb9?sid=ed244b96-cb45-4c06-89ac-e998a74b3bac

[yassine-cc]

Not sure what's happening in that video, the chat does stream the response when i try it, the only issue i have like one in five times was the malformed codeblock (Not the same as the one from your video, the code block just ends then another empty is inserted after it) which i assumed came form the llm

[yassine-cc]

Here you can see that it does stream before pulling the latest changes, although the markdown block after the codeblock in the second prompt does output in one go for some reason, i'll have to investigate this further
You can also see how malformatted markdown/yaml can cause issues, and how tweaking my prompt to force the llm to output a proper yaml code block does solve it

recording.mp4

PS: For some reason the video shows up black in the browser, but downloading it will allow you to play it properly

[yassine-cc]

Here are the challenges that i found with the one instance thing:

When it comes to an instance per chat

• It's hard to keep track of the chat window as vscode provides no id, just context history
  This could be used to identify chat windows, but isn't very reliable as the history is easily altered, all requests even older ones from the history could be easily deleted with a press of a button, responses could be regenerated, etc

When it comes to a single instance for all chats (but this always applies anyway)

• Some commands do exit after completing, like the /dry command in chat for example
  So to make the single instance work reliably, i'd need to add some sort of repl or playground mode to testdriverai to never exit, and never change anything locally.

This is a bit challenging to do properly with the current structure of the CLI, i could hack around this technically by adding some flag that causes the exit and save functions to return immediately without executing, but i'm not a fan of the idea.

[ianjennings]

@yassine-cc I think this change fixed the "single global instance" problem:
c6cc767#diff-9f9536afc689550d0ec9557aab88152fcb5839bf4a2c91e73134f5e5cc03a2fdR65

One global instance is fine. so that should be resolved.

I think the other parser problem may be caused by different return characters on mac/windows. Will look a little more into it today.