-
Notifications
You must be signed in to change notification settings - Fork 46
Overview/ait 189 intro token #3035
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Overview/ait 189 intro token #3035
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
GregHolmes
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this looks good! (I can't approve or anything as I raised it)
@rainbowFi I've left some comments on my thoughts.
I also think we need to be careful and remember that if some of this (such as the full list of agents/frameworks) isn't available on release, we need to remove the TODO comments.
400eb09 to
f8056cb
Compare
a5c00a1 to
f904645
Compare
mschristensen
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this - I left a few comments. Taking a step back, given this is such a key piece of the offering, I feel that we can do more to describe the value proposition for token streaming over Ably. Are there ways we can explicitly enumerate the key parts of the user experience that constitute a great token streaming experience? We could then contrast those with the complexities of achievng this in a connection-oriented HTTP streaming model, and how Ably solves this out of the box.
I think there is some overlap conceptually with the Sessions & identity overview, but I think it would be okay to repeat some of that here, with a token-streaming rather than session emphasis.
Let's discuss in our catch up tomorrow :)
| meta_description: "Learn about token streaming with Ably AI Transport, including common patterns and the features provided by the Ably solution." | ||
| --- | ||
|
|
||
| Token streaming is a technique used with Large Language Models (LLMs) where the model's response is emitted progressively as each token is generated, rather than waiting for the complete response before transmission begins. This allows users to see the response appear incrementally, similar to watching someone type in real time, giving an improved user experience. This is normally accomplished by streaming the tokens as the response to an HTTP request from the client. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, we prefer to use single word "realtime" at Ably.
(This is not what most of the internet seems to do, but alas this is our convention)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"This is normally accomplished by streaming the tokens as the response to an HTTP request from the client."
I think this can be moved out into a new paragraph. I think the intro paragraph should focus on the description of what token streaming is before getting into how it is implemented.
Then, I would suggest colocating this statement with the content that follows after the image, since that paragraph starts by describing the weakness of this approach.
| If an HTTP stream is interrupted, for example because the client loses network connection, then any tokens that were transmitted during the interruption will be lost. Ably AI Transport solves this problem by streaming tokens to a [Pub/Sub channel](docs/channels), which is not tied to the connection state of either the client or the agent. A client that [reconnects](/docs/connect/states#connection-state-recovery) can receive any tokens transmitted while it was disconnected. If a new client connects, for example because the user has moved to a different device, then it is possible to hydrate the new client with all the tokens transmitted for the current request as well as the output from any previous requests. The detailed mechanism for doing this will depend on which [token streaming pattern](#patterns) you choose to use. | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a bit of a wall of text, but there are some nice bits of value prop in there. Can we pull those out, perhaps into bullets?
aebe2c1 to
ea0ac8d
Compare
d2cb41d to
b560500
Compare
General overview intro page for AIT, giving a summary of major feature groups
Overview page for token streaming - set direction, link to later pages
Co-authored-by: Paddy Byers <paddy.byers@gmail.com>
This reverts commit 78b0411.
72732e1 to
ee4ba8a
Compare
mschristensen
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, couple of small suggestions but otherwise LGTM
38cd7a1
into
AIT-129-AIT-Docs-release-branch
Description
Adds overview pages to the documentation covering
the overall AIT product, listing the major features and linking them to other documentation
token streaming, including an overview of the proposed architecture and patterns