Skip to content

Conversation

@GregHolmes
Copy link
Contributor

Description

Adds overview pages to the documentation covering

the overall AIT product, listing the major features and linking them to other documentation
token streaming, including an overview of the proposed architecture and patterns

@GregHolmes GregHolmes added the review-app Create a Heroku review app label Dec 16, 2025
@coderabbitai
Copy link

coderabbitai bot commented Dec 16, 2025

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ably-ci ably-ci temporarily deployed to ably-docs-overview-ait--4ak0uj December 16, 2025 18:38 Inactive
@GregHolmes GregHolmes mentioned this pull request Dec 16, 2025
3 tasks
Copy link
Contributor Author

@GregHolmes GregHolmes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good! (I can't approve or anything as I raised it)

@rainbowFi I've left some comments on my thoughts.

I also think we need to be careful and remember that if some of this (such as the full list of agents/frameworks) isn't available on release, we need to remove the TODO comments.

@ably-ci ably-ci temporarily deployed to ably-docs-overview-ait--4ak0uj December 19, 2025 16:45 Inactive
@ably-ci ably-ci temporarily deployed to ably-docs-overview-ait--4ak0uj December 19, 2025 16:49 Inactive
@matt423 matt423 force-pushed the AIT-129-AIT-Docs-release-branch branch from 400eb09 to f8056cb Compare December 23, 2025 10:41
@rainbowFi rainbowFi force-pushed the overview/ait-189-intro-token-fixed branch from a5c00a1 to f904645 Compare January 6, 2026 09:56
@mschristensen mschristensen added review-app Create a Heroku review app and removed review-app Create a Heroku review app labels Jan 6, 2026
@ably-ci ably-ci temporarily deployed to ably-docs-overview-ait--n4mudo January 6, 2026 18:49 Inactive
Copy link
Contributor

@mschristensen mschristensen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this - I left a few comments. Taking a step back, given this is such a key piece of the offering, I feel that we can do more to describe the value proposition for token streaming over Ably. Are there ways we can explicitly enumerate the key parts of the user experience that constitute a great token streaming experience? We could then contrast those with the complexities of achievng this in a connection-oriented HTTP streaming model, and how Ably solves this out of the box.

I think there is some overlap conceptually with the Sessions & identity overview, but I think it would be okay to repeat some of that here, with a token-streaming rather than session emphasis.

Let's discuss in our catch up tomorrow :)

meta_description: "Learn about token streaming with Ably AI Transport, including common patterns and the features provided by the Ably solution."
---

Token streaming is a technique used with Large Language Models (LLMs) where the model's response is emitted progressively as each token is generated, rather than waiting for the complete response before transmission begins. This allows users to see the response appear incrementally, similar to watching someone type in real time, giving an improved user experience. This is normally accomplished by streaming the tokens as the response to an HTTP request from the client.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, we prefer to use single word "realtime" at Ably.
(This is not what most of the internet seems to do, but alas this is our convention)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"This is normally accomplished by streaming the tokens as the response to an HTTP request from the client."

I think this can be moved out into a new paragraph. I think the intro paragraph should focus on the description of what token streaming is before getting into how it is implemented.

Then, I would suggest colocating this statement with the content that follows after the image, since that paragraph starts by describing the weakness of this approach.

Comment on lines 10 to 11
If an HTTP stream is interrupted, for example because the client loses network connection, then any tokens that were transmitted during the interruption will be lost. Ably AI Transport solves this problem by streaming tokens to a [Pub/Sub channel](docs/channels), which is not tied to the connection state of either the client or the agent. A client that [reconnects](/docs/connect/states#connection-state-recovery) can receive any tokens transmitted while it was disconnected. If a new client connects, for example because the user has moved to a different device, then it is possible to hydrate the new client with all the tokens transmitted for the current request as well as the output from any previous requests. The detailed mechanism for doing this will depend on which [token streaming pattern](#patterns) you choose to use.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit of a wall of text, but there are some nice bits of value prop in there. Can we pull those out, perhaps into bullets?

@mschristensen mschristensen force-pushed the AIT-129-AIT-Docs-release-branch branch from aebe2c1 to ea0ac8d Compare January 7, 2026 11:48
@rainbowFi rainbowFi force-pushed the overview/ait-189-intro-token-fixed branch from d2cb41d to b560500 Compare January 7, 2026 17:07
@ably-ci ably-ci temporarily deployed to ably-docs-overview-ait--n4mudo January 7, 2026 17:07 Inactive
@ably-ci ably-ci had a problem deploying to ably-docs-overview-ait--n4mudo January 9, 2026 10:00 Failure
@rainbowFi rainbowFi force-pushed the overview/ait-189-intro-token-fixed branch from 72732e1 to ee4ba8a Compare January 9, 2026 10:02
@ably-ci ably-ci temporarily deployed to ably-docs-overview-ait--n4mudo January 9, 2026 10:02 Inactive
@ably-ci ably-ci temporarily deployed to ably-docs-overview-ait--n4mudo January 9, 2026 10:27 Inactive
@ably-ci ably-ci temporarily deployed to ably-docs-overview-ait--n4mudo January 9, 2026 10:30 Inactive
Copy link
Contributor

@mschristensen mschristensen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, couple of small suggestions but otherwise LGTM

@ably-ci ably-ci temporarily deployed to ably-docs-overview-ait--n4mudo January 12, 2026 15:51 Inactive
@rainbowFi rainbowFi merged commit 38cd7a1 into AIT-129-AIT-Docs-release-branch Jan 12, 2026
3 of 5 checks passed
@rainbowFi rainbowFi deleted the overview/ait-189-intro-token-fixed branch January 12, 2026 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-app Create a Heroku review app

Development

Successfully merging this pull request may close these issues.

7 participants