This repository was archived by the owner on Sep 9, 2025. It is now read-only.
sdg: Add details on how to configure serving for custom pipelines #115
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The design proposal in instructlab#109, and the corresponding implementation in instructlab/sdg#86, raised the importance of clearly defining how a custom pipeline that requires a model with custom adapters would be configured. This document explores that topic. It's possible this should just become a subsection of instructlab#109. Signed-off-by: Russell Bryant <rbryant@redhat.com>
russellb
force-pushed
the
sdg-serving-config
branch
from
a72e0f2 to
b4d3fec
Compare
russellb
commented
Jul 10, 2024
Comment on lines
+65
to
+70
| backend_args: | ||
| vllm_args: | ||
| - ... | ||
| - "--lora-modules" | ||
| - "my_custom_adapter=path/to/my_custom_adapter" | ||
| - ... |
Member
Author
There was a problem hiding this comment.
possible impact to this syntax here: instructlab/instructlab#1635
markmc
reviewed
Jul 10, 2024
| version: "1.0" | ||
| models: | ||
| - name: my_custom_adapter | ||
| description: a funky adaptor for generating questions |
Contributor
There was a problem hiding this comment.
The models section really isn't necessary in retrospect. This will work:
version: "1.0" models: - name: myfunkyadaptor description: a funky adaptor for generating questions block_configs: - block_type: LLMBlock block_config: block_name: gen_questions config_path: configs/skills/freeform_questions.yaml add_num_samples: True gen_kwargs: model_id: myfunkyadatpor output_cols: - question drop_duplicates: - question
And then the serving config just needs to have --lora-modules=myfunkyadapter=path/to/my_custom_adapter
Member
Author
|
The important details from here are now integrated into #109 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The design proposal in #109, and the corresponding implementation in
instructlab/sdg#86, raised the importance of
clearly defining how a custom pipeline that requires a model with
custom adapters would be configured. This document explores that
topic.
It's possible this should just become a subsection of #109.
Signed-off-by: Russell Bryant rbryant@redhat.com