Hard question I'm thinking about: how do we enable the community to add successful chains while mitigating the risk of bad actors adding chains that will hurt performance?

[mshumer]

See title. Need to think on this.

[mshumer]

Potentially, eventually, we can have people (and/or models!) review these and if enough agree that it's a good example, commit it to the system.

[mshumer]

@Alec2435 suggested some form of ELO-based reasoning arena, where users can vote on chains and we can have a rank of the best ones, for training even eventually.

[Stevenic]

I think you need to clearly identify your extensibility points and then encourage the community to build on top of these extensibility when integrating the engine into their own projects. They can then suggest certain common implementation be in the core library but there should be a really high bar for what goes in the core versus wheat's layered on through external libraries.

My philosophy (educated by a decade of designing SDK's) is that less is more... The tighter you can keep your core the better off you are. That's the issue with project like LangChain. It's too opinionated and tries to be a kitchen sink that does everything for everyone. You can't do that...

Things should generally start off as just a code sample that builds on the core... Once a sample proves that its a common pattern that seems like a solid design pattern it gets folded into the core.

[mshumer]

Yep — the way I'm thinking about it, the general rule will be, if something doesn't increase capabilities, it isn't added. There are a lot of fancy techniques floating around that don't actually make a difference and add bloat — want to avoid that.

[Stevenic]

@mshumer once you get the basics of the engine the way you want I'm happy to review everything with an eye towards the overall design of the SDK. The engine is currently still finding its legs and figuring out what it wants to be...

[Stevenic]

Here's a bit of a quick initial assessment. Here's your high level concept list:

• Step-by-step reasoning with integrated tools (calculator, Python interpreter, web access, Wolfram Alpha)
• Semantic chain search powered by Cohere embeddings so the models within the system can continually learn to better use this engine's rails
• Parallel ensemble processing with multiple models (MoA)
• Stateful Python interpreter sessions, enabling multiple threads at once
• Model-agnostic API supporting multiple providers (OpenAI, Anthropic, etc.)

The obvious extensibility points that jump out:

• Models: Define a clear abstraction for integrating new models. Building in an OpenAI model because that's the gold standard and maybe an Anthropic model implementation but don't go crazy chasing every model under the sun. People can just layer on new models.
• Tools: Same for tools. Define a clear abstraction for tools but don't go crazy trying to build every tool under the sun into the core. A great bar for what tools get built in is how many packages do they depend on? If they have a bunch of external dependencies they're probably too opinionated and shouldn't be built into the core.

Not so obvious extensibility points:

• Orchestrator: I haven't looked too closely at your ensemble stuff yet but I'm assuming this is probably another interface that's a specialization of a model. Some models can orchestrate other models but not all models are orchestrators.
• ChainStore: I haven't looked too closely at how storing of chains works but the phrase "Cohere embeddings" says to me that's an opinion some people may not want in their application. This is probably another extensibility point but I haven't looked at the code yet.

[mshumer]

Thank you! Almost there, I think. I want to make it OpenAI-compatible, add a search-over-steps implementation (tricky part is doing that while maintaining interpreter state) and then it should be mostly there.

Models — right now it works with any OAI-compatible endpoint, so should be super easy to use with almost any model. OpenRouter makes this easy.

Tools — for v1, at least until I think of a nice way to make it extensible, here's what I'm doing:

• Internal tools are defined by me, directly. Python, web search, Wolfram. This is what the system will have access to internally for its reasoning steps on day 1.
• The user can define output tools, just like with any model API. In the final step, where it pulls together its answer, it can choose to use/return one of these tools to the user.

Orchestrator —  it's a bit messy and requires updates almost every time I add a new thing so I may put this aside for now, and add it when everything else is settled.

Chains — totally optional. I wanted to think about how the system might learn over time, this is what enables it. Successful chains are saved, pulled in as-needed, and we build plans based on learnings from them. Changed the language in the readme to make it clearer.

[Stevenic]

Ok so back on topic... Thinking through how these reasoning chains might interface with the engine via a ChainStore abstraction I would probably treat them like a mountable database or catalog of chains (you'll find everything is just a catalog to me.) So a ChainStore would load a set(s) of chains that the engine can pull from. There are built-in chains but again there should be a high-bar for what chains get added to the built-in sets.

One thing developers generally want is stability so any chains should be added to versioned sets that a developer can opt-into or out-of taking. Once I get the engine working the way I want it with a given set of chains I just don't want it to change. It's bad enough that the models are constantly changing because of fine-tuning. Let's not add to the chaos.

[mshumer]

Yes, exactly! This is the .json file I have in the repo. It's definitely not the best way to do this (maybe a public database or something) but for now it's fine. I have some ideas for approaches to allow people to propose chains to add to the database but it's tricky, needs some time to marinate.

Versioning is a great idea!