[ENHANCEMENT] Provide setting which controls whether the indexer respects .gitignore

[emeraldcheshire]

Problem (one or two sentences)

Roo Code indexer always ignores files listed in .gitignore regardless of .rooignore.
Consequently, there is no way to have files in the project that are indexed but not tracked by git.

Context (who is affected and when)

Users with projects that require external material such as APIs (headers, etc) for full analysis. These external APIs must be excluded via .gitignore to prevent them from being accidentally committed. These APIs are important for Roo Code to index because they provide context for what the actual code is interacting with.

Desired behavior (conceptual, not technical)

Roo Code should provide a per-project setting which controls whether the indexer respects .gitignore. The default should continue to be True: respect .gitignore and .rooignore. When False: respect only .rooignore.

Note: all discussion of .gitignore applies to .gitignore and .git/info/exclude

Constraints / preferences (optional)

No response

Request checklist

• I've searched existing Issues and Discussions for duplicates
• This describes a specific problem with clear context and impact

Roo Code Task Links (optional)

No response

Acceptance criteria (optional)

No response

Proposed approach (optional)

No response

Trade-offs / risks (optional)

No response

[roomote]

I've analyzed this issue and here's my proposed implementation plan:

Summary

The indexer currently always respects .gitignore patterns in two places: (1) ripgrep file discovery, and (2) an ignoreInstance filter passed to both the DirectoryScanner and FileWatcher. The goal is to add a per-project setting that allows users to disable .gitignore respect for indexing, so only .rooignore is used for filtering.

Proposed Changes

1. Add a new setting to codebaseIndexConfig

Add a boolean field codebaseIndexRespectGitIgnore (default: true) to the config stored in global state. This controls whether the indexer respects .gitignore when scanning files.

Files:

• src/services/code-index/config-manager.ts - Read the new setting from codebaseIndexConfig, expose it via a getter
• src/services/code-index/interfaces/config.ts - Add to CodeIndexConfig interface
• packages/types/src/global-state.ts (or wherever GlobalState types live) - Add the field to the config type

2. Pass the setting through to the indexer pipeline

Files:

• src/services/code-index/manager.ts (_recreateServices) - Conditionally skip loading .gitignore patterns into ignoreInstance based on the new config setting. When the setting is false, pass an empty ignore instance instead.
• src/services/code-index/processors/scanner.ts (scanDirectory) - The scanner calls listFiles() which uses ripgrep. Ripgrep respects .gitignore by default. We need to pass --no-ignore-vcs to ripgrep when the setting is false. This means listFiles needs an optional parameter to disable .gitignore respect.
• src/services/glob/list-files.ts - Add an optional respectGitignore parameter (default true) to listFiles(). When false, add --no-ignore-vcs to the ripgrep arguments and skip creating the gitignore-based ignore instance for directory filtering.

3. Propagate the setting to the FileWatcher

Files:

• src/services/code-index/processors/file-watcher.ts - The ignoreInstance is already optional. When the setting is false, the manager simply won't pass a .gitignore-based instance, so the watcher will skip the .gitignore check at line 527.

4. Wire up the UI

Files:

• src/core/webview/ClineProvider.ts - Include the new setting when sending/receiving codebaseIndexConfig state
• src/core/webview/webviewMessageHandler.ts - Handle saving the new setting
• webview-ui/ - Add a toggle in the code index settings UI (likely in the settings panel for code indexing)

5. Handle config change detection

Files:

• src/services/code-index/config-manager.ts (doesConfigChangeRequireRestart) - Changing this setting should trigger a re-index (restart required), since the set of indexed files will change.

6. Tests

• Unit tests for the config manager with the new setting
• Unit tests for listFiles with respectGitignore: false
• Unit tests for scanner/file-watcher behavior when gitignore is disabled
• Integration test verifying end-to-end that gitignored files get indexed when the setting is disabled

Design Notes

• Default is true (respect .gitignore), preserving backward compatibility
• The setting is per-project via the existing codebaseIndexConfig global state
• Changing the setting triggers a full re-index since the file set changes
• .rooignore always takes precedence regardless of this setting

Please mention @roomote in a comment saying that you approve this plan, and I'll proceed with the implementation.

[emeraldcheshire]

@roomote does this issue apply to more than the Indexer? For example, is listFiles() used in contexts such as an AI task requesting a directory list? The expectation is that when the option is enabled it would be found by the Indexer AND by any other Roo Code functionality which looks for files in the project. Should the option's scope be widened beyond the Indexer?

[roomote]

I've seen this comment and it's a great point. After investigating, listFiles() is used in 4 contexts beyond just the indexer:

1. Code Indexer scanner (scanner.ts) - the original issue scope
2. ListFilesTool (ListFilesTool.ts) - when the AI uses the list_files tool during tasks
3. getEnvironmentDetails (getEnvironmentDetails.ts) - workspace file listing shown in environment context
4. WorkspaceTracker (WorkspaceTracker.ts) - tracks file paths in the workspace

The .gitignore respect happens at two layers inside listFiles():

• ripgrep respects .gitignore by default when scanning files
• A separate createIgnoreInstance function loads .gitignore patterns for directory filtering

So yes, the setting's scope should absolutely be widened beyond the Indexer. The cleanest approach is to add an optional respectGitignore parameter to listFiles() itself. That way all callers automatically benefit when the setting is false.

I'll implement this as a project-level setting (stored in global state, not tied to codebaseIndexConfig) since it affects all file-listing functionality. The setting will default to true (preserving current behavior), and when set to false, it will:

• Pass --no-ignore-vcs to ripgrep
• Skip loading .gitignore patterns in the ignore instance for directory filtering
• Skip loading .gitignore in the indexer's _recreateServices()

I'll open a draft PR shortly with my attempt at this.

[roomote]

Opened a draft PR with my attempt at addressing this: #11250

The PR widens the scope of the setting beyond just the indexer, as suggested. The new codebaseIndexRespectGitIgnore setting (default: true) affects all listFiles() callers -- the indexer, the list_files tool, environment details file listing, and the workspace tracker. A toggle is available in the Code Index advanced settings UI.

Feedback welcome!

[emeraldcheshire]

Adding a bit more context for reviewers on why this comes up at enterprise scale:

In systems with 10–20M SLOC, repos are commonly split for size and access control. Teams often rely on external API/SDK snapshots for semantic analysis, but intentionally exclude them via .gitignore to prevent accidental commits. If those files aren’t indexed, Roo Code has incomplete visibility into the codebase.

One additional side effect I’ve observed is performance-related: if external APIs can’t be .gitignored, Roo Code’s shadow git repos used for checkpointing grow very large, and checkpoint initialization has exceeded the max 60-second timeout due to the extra files. Being able to ignore them for version control while still indexing them avoids that issue.