Add support for onnxruntime-extensions via session option #101
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for dynamically loading onnxruntime-extensions via session options to enable custom operators like tokenizers for embedding models within onnxruntime-server.
- Implements a new
register_ort_extensionsfunction to load custom ops libraries - Adds
ortextensions_pathsession option to specify the library path - Includes documentation for the new ORT Extensions support feature
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| src/onnx/session.cpp | Adds ORT extensions registration logic and integrates it into session constructors |
| README.md | Documents the new ORT Extensions support feature with usage example |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
Hi @oga5, Thank you so much for your interest and contribution to the project! I'd like to discuss a couple of things. I want to create a test case for this feature. Would it be possible for you to provide a sample model that uses custom ops and the corresponding custom ops library? A very small one would be perfect for testing. If that's difficult, no worries, I can try to create one myself later. I noticed that Thanks again for your contribution! |
| ## ONNXRuntime Extensions Support | ||
|
|
||
| To use the [onnxruntime-extensions](https://github.com/microsoft/onnxruntime-extensions)(Custom Ops Library), set the | ||
| options as follows when creating a session. | ||
|
|
||
| ```json | ||
| { | ||
| "model": "string", | ||
| "version": "string", | ||
| "option": { | ||
| "cuda": ..., | ||
| "ortextensions_path": "/absolute/path/to/libonnxruntime_extensions.so" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| For more details on the session creation request, please refer to | ||
| the [API documentation](https://kibae.github.io/onnxruntime-server/swagger/#/ONNX%20Runtime%20Session/createSession). | ||
|
|
||
|
|
| ortextensions_path: | ||
| type: string | ||
| description: To use the ONNXRuntime Extension (Custom Ops Library), you must provide the library path. | ||
| nullable: false | ||
| required: false | ||
| example: /absolute/path/to/libonnxruntime_extensions |
| if (option.contains("ortextensions_path") && option["ortextensions_path"].is_string()) { | ||
| auto ext_path_str = option["ortextensions_path"].get<std::string>(); | ||
| const char *ext_path = ext_path_str.c_str(); | ||
| OrtStatus *status = Ort::GetApi().RegisterCustomOpsLibrary_V2(session_options, ext_path); | ||
| if (status != nullptr) { | ||
| const char *err = Ort::GetApi().GetErrorMessage(status); | ||
| std::string msg = err ? err : "unknown error"; | ||
| Ort::GetApi().ReleaseStatus(status); | ||
| throw runtime_error(std::string("Failed to register ORT extensions: ") + msg); | ||
| } | ||
|
|
||
| _option["ortextensions_path"] = option["ortextensions_path"]; | ||
| } | ||
|
|
There was a problem hiding this comment.
@oga5 I replaced the deprecated RegisterCustomOpsLibrary with the newer RegisterCustomOpsLibrary_V2 and moved the logic into the session constructor.
https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a4dc58d1aa01d67494a3fb4ce00e9a929
|
Hi @kibae, Thank you very much for your kind feedback and for taking the initiative to update the code to use RegisterCustomOpsLibrary_V2. Regarding the test case: I've created a small ONNX model that uses custom ops from onnxruntime-extensions. The model includes tokenizer.onnx, which demonstrates the use of custom ops. Thanks again for reviewing the PR! |
|
@oga5 The model size is quite large! It would be difficult to include it in the codebase for creating test code. I will merge this PR and release a new version. After that, I'll learn more about the custom library and then create test code. |
|
Hi @kibae, Thank you for merging the PR. I've created a minimal tokenizer.onnx sample: oga5/tokenizer_onnx_sample I hope it proves helpful for future reference. |
2 participants
Summary
This PR enables dynamic loading of onnxruntime-extensions via session options.
Usage
To enable custom ops from onnxruntime-extensions, set the following in your session options:
{ "ortextensions_path": "/absolute/path/to/libonnxruntime_extensions.so" } ## Motivation By enabling onnxruntime-extensions, we can run additional ops such as tokenizers for embedding models within onnxruntime-server, improving end-to-end inference workflows. ## Documentation Added a new subsection "ORT Extensions Support" in README.md under the How to use section, before ## Simple usage examples.