Suggestion: Evaluate GPT-OSS with its native browsing tools for a more accurate assessment

[Hannibal046]

Hi team,

First, thank you for the excellent work on the BrowseComp-Plus benchmark. It's a valuable contribution to the fair and transparent evaluation of Deep Research Agents.

I am writing to open a discussion about the evaluation methodology for GPT-OSS.

In the paper and current benchmark setup, GPT-OSS is evaluated using external retrieval tools. However, as GPT-OSS was natively trained with a browsing toolset, I believe an evaluation using its native tools would provide a more authentic and insightful measure of its intended capabilities.

The Challenge

I understand that a key reason for not using the native tools might be the flawed implementations in popular serving frameworks like vLLM and SGLang, which make a correct evaluation difficult.

Proposed Solution & My Work

To address this, I've developed a repository that enables a full and proper evaluation of GPT-OSS on BrowseComp-Plus using its native browsing tools. This implementation works around the existing framework limitations.

You can find the project, code, and some interesting preliminary findings here: https://github.com/Hannibal046/GPT-OSS-BrowseCompPlus-Eval

Thanks again for your great work! I'm happy to discuss this further.

[MXueguang]

Hi @Hannibal046,

Thank you very much for implementing the browsecomp-plus evaluation on the natural behaviour of OSS.

During the development of the evaluation, we also noticed that tool usage behavior and context management methods play a critical role in effectiveness. However, to ensure a fair evaluation under controlled conditions, we chose to use the same strategies for all methods in the main evaluation. (i.e. top5 search with 500 token contexts for each doc.)

In this setting, some methods like OSS may suffer from out-of-distribution (OOD) tool usage behavior and suboptimal performance due to not fully accessing the complete content of the documents. In Table 5, we also show that using search + get_doc can provide additional gains.

I think your implementation of the natural behavior of OSS is very helpful in enriching the discussion, and showing the importance of comparing the Agent's in the in-domain and ood tool usage behavior.

Let’s discuss this further; in the meantime, we will add a pointer in our README to your implementation.

[Hannibal046]

Hi,

Thank you for the quick and thoughtful reply, and for your willingness to link to my repository in the README. I appreciate your engagement on this important topic.

You're right, it's definitely tough to find a one-size-fits-all setup for the tools. That's really where the flexibility of open-source models comes in handy, letting us find the best way to give each one a fair shot.

I think how you handled the Search-R1 model is a perfect example of this—adapting its input to let it perform at its best. It shows the goal is really to measure the model's true capabilities.

My suggestion for using native tools with GPT-OSS comes from that same idea: to evaluate it in the way it was designed to work, giving us a more authentic picture of its performance.

Thanks again for being open to this, and for all the excellent work on the benchmark.

[MXueguang]

Hi,
I’m merging the PR here: #10.

Thanks for the additional comments. I think an important aspect of the evaluation is to isolate the factors that affect effectiveness. Based on your suggestions, I’m proposing to run an additional setting with the browsing tool, but with the maximum context length per document limited to 512 tokens.

In addition to determining the optimal accuracy for gpt-oss, its also worth to address the following questions:
1. How much of the effectiveness difference comes from using function calls vs. the browsing tool? (A small note here: function calling is not an unnatural behavior for the model, as the search tool is intended to be part of general tool usage. It’s worth understanding the gap between these approaches.)
2. How much of the effectiveness difference comes from the additional context visibility provided by full-page access during browsing? It’s valuable to understand how important extended context is for supporting deep research agents.

Fortunately, BrowseComp-Plus is designed to support these different inference strategies. We’ll explore your implementation and share additional results in this thread.