Create a set of questions you might ask for use in benchmarking / testing LLMs.

[sabszh]

A nice idea, would be to collect a set of questions, to use those to test the performances of chunking / doc size / model.
If you come up with nice ways to test how well the chatbot is doing, feel free to leave them as a comment here!

[lightnin]

Good idea! I changed the title to make it a bit more legible to folks new to this work. Should people include the question and example answers they like / don't like?

[sabszh]

Good idea! I changed the title to make it a bit more legible to folks new to this work. Should people include the question and example answers they like / don't like?

Nice! Thank you.
I think that would be great actually, or a sense of expected direction. Else I was thinking a way could be to then test them out with the adjusted parameters, and put the results in a sheet and have you all take a look at it--and then do some kind of qualitative test on it (microphenomenology?)

[lightnin]

Hmm, well that wouldn't be microphenomenology but we could see if we get a corpus of decent sample questions to throw at different models. I don't think it'll ever shape up into much more than a "sanity check" function - and maybe it'll help us get more articulate about what we want. But the realm we are working in here is always going to be pretty subjective.