support ShareGPT dataset as data file

[tukwila]

Summary

Details

I hope data file can support ShareGPT as benchmark test data such as: ShareGPT_V3_unfiltered_cleaned_split.json; In this PR, user can abstract testing prompts from origin file and filter human prompts (10 < words < 1000) to save into local file, refer to:

• [ ]

Test Plan

Related Issues

• Resolves #

---

• "I certify that all code in this PR is my own, except as noted below."

Use of AI

• Includes AI-assisted code completion
• Includes code generated by an AI application
• Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes ## WRITTEN BY AI ##)

[tukwila]

related to:#138

https://github.com/LMCache/LMCache/tree/dev/benchmarks/multi-round-qa

[sjmonson]

This seems external to the GuideLLM. Can you please move all code and documentation to /contrib/sharegpt_preprocess.

[tukwila]

This seems external to the GuideLLM. Can you please move all code and documentation to /contrib/sharegpt_preprocess.

Done

[sjmonson]

Sorry I forgot about this PR due to the sudden flurry of new PRs. Can you also move the changes in docs/datasets.md to contrib/sharegpt_preprocess/README.md.

[tukwila]

Sorry I forgot about this PR due to the sudden flurry of new PRs. Can you also move the changes in docs/datasets.md to contrib/sharegpt_preprocess/README.md.

Done

[jaredoconnell]

Is the requirements.txt supposed to include all dependencies? I had to install datasets and transformers for it to work.

It may be beneficial to also note that you need to run it with the HF_TOKEN value set.

Once I addressed these it appears to have worked.

[tukwila]

Is the requirements.txt supposed to include all dependencies? I had to install datasets and transformers for it to work.

It may be beneficial to also note that you need to run it with the HF_TOKEN value set.

Once I addressed these it appears to have worked.

yes, i updated and retest it.