Codex/find ways to enhance tool effectiveness

README.md

@@ -10,7 +10,7 @@
 
 <p class="align center">
 
-ffufai is an AI-powered wrapper for the popular web fuzzer ffuf. It automatically suggests file extensions for fuzzing based on the target URL and its headers, using either OpenAI's GPT or Anthropic's Claude AI models.
+ffufai is an AI-powered wrapper for the popular web fuzzer ffuf. It automatically suggests file extensions or contextual wordlists for fuzzing based on the target URL, headers, and response signals, using Gemini (default priority), OpenAI, Anthropic, Groq, or OpenRouter models.
 
 </p>
 
@@ -20,15 +20,23 @@ ffufai is an AI-powered wrapper for the popular web fuzzer ffuf. It automaticall
 <img width="600  " alt="image" src="https://github.com/user-attachments/assets/0384d4f0-3a07-48d9-9805-ea1e76b6b693">
 
 - Seamlessly integrates with ffuf
-- Automatically suggests relevant file extensions for fuzzing
-- Supports both OpenAI and Anthropic AI models
+- Auto mode (extension or wordlist) with stack-aware suggestions
+- Multi-phase AI inference (plan → generate → verify) to reduce false positives
+- Multi-provider routing with Gemini-first priority and optional consensus mode
+- Profiles and goals to bias toward critical targets
+- Wappalyzer-style signature detection for fast tech hints
+- Tech-aware wordlist selection via catalog mappings
+- Optional feedback loop that refines wordlists based on ffuf results
+- Active-learning persistence of successful findings
+- DNS/TLS and error-page context enrichment
+- Caching for faster repeated scans
 - Passes through all ffuf parameters
 
 ## Prerequisites
 
 - Python 3.6+
 - ffuf (installed and accessible in your PATH)
-- An OpenAI API key or Anthropic API key
+- At least one API key: Gemini, OpenAI, Anthropic, Groq, or OpenRouter
 
 ## Installation
 
@@ -40,7 +48,7 @@ ffufai is an AI-powered wrapper for the popular web fuzzer ffuf. It automaticall
 
 2. Install the required Python packages:
    ```
-   pip install requests openai anthropic
+   pip install requests openai anthropic beautifulsoup4
    ```
 
 3. Make the script executable:
@@ -63,6 +71,36 @@ ffufai is an AI-powered wrapper for the popular web fuzzer ffuf. It automaticall
    ```
    export ANTHROPIC_API_KEY='your-api-key-here'
    ```
+   Or for Gemini:
+   ```
+   export GEMINI_API_KEY='your-api-key-here'
+   ```
+   Or for Groq:
+   ```
+   export GROQ_API_KEY='your-api-key-here'
+   ```
+   Or for OpenRouter:
+   ```
+   export OPENROUTER_API_KEY='your-api-key-here'
+   ```
+
+   You can also provide multiple API keys for rotation:
+   ```
+   export GEMINI_API_KEYS='key1,key2'
+   export OPENAI_API_KEYS='key1,key2'
+   export ANTHROPIC_API_KEYS='key1,key2'
+   export GROQ_API_KEYS='key1,key2'
+   export OPENROUTER_API_KEYS='key1,key2'
+   ```
+
+   Optional model overrides:
+   ```
+   export GEMINI_MODEL='gemini-3.5-pro'
+   export OPENAI_MODEL='gpt-4o'
+   export ANTHROPIC_MODEL='claude-sonnet-4-20250514'
+   export GROQ_MODEL='llama-3.1-70b-versatile'
+   export OPENROUTER_MODEL='openrouter/auto'
+   ```
 
    You can add these lines to your `~/.bashrc` or `~/.zshrc` file to make them permanent.
 
@@ -80,7 +118,7 @@ Or if you've created the symbolic link:
 ffufai -u https://example.com/FUZZ -w /path/to/wordlist.txt
 ```
 
-ffufai will automatically suggest extensions based on the URL and add them to the ffuf command.
+ffufai will automatically suggest extensions or wordlists based on the URL and add them to the ffuf command.
 
 ## Parameters
 
@@ -92,6 +130,78 @@ ffufai accepts all the parameters that ffuf does, plus a few additional ones:
 - `--max-extensions`: Sets the maximum number of extensions to suggest. Default is 4.  
   Example: `ffufai --max-extensions 6 -u https://example.com/FUZZ -w wordlist.txt`
 
+- `--mode`: Choose `extensions`, `wordlist`, or `auto` (default).  
+  Example: `ffufai --mode wordlist -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--profile`: Tuning profile (`balanced`, `critical`, `stealth`, `depth`, `api-only`, `spa`, `admin-heavy`, `cloud-native`, `legacy`, `mobile-backend`, `partner-portal`, `healthcheck`, `cdn-edge`, `auth-heavy`, `storage`, `payments`).  
+  Example: `ffufai --profile critical -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--goal`: Primary hunting goal (`general`, `auth-bypass`, `data-exfil`, `rce`, `misconfig`, `idor`, `ssrf`, `lfi`, `sqli`, `xss`, `csrf`, `file-upload`, `secrets`, `infra`, `compliance`, `webhooks`, `graphql`, `mobile`, `bypass-waf`).  
+  Example: `ffufai --goal data-exfil -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--consensus`: Use all available providers to cross-check suggestions.  
+  Example: `ffufai --consensus -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--cache-path`: Path to the cache file (default `~/.cache/ffufai/cache.json`).  
+  Example: `ffufai --cache-path /tmp/ffufai-cache.json -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--no-cache`: Disable cache usage.  
+  Example: `ffufai --no-cache -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--state-path`: Path to the rotation state file (default `~/.cache/ffufai/state.json`).  
+  Example: `ffufai --state-path /tmp/ffufai-state.json -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--findings-path`: Path to the findings persistence file (default `~/.cache/ffufai/findings.json`).  
+  Example: `ffufai --findings-path /tmp/ffufai-findings.json -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--knowledge-path`: Path to the global knowledge base (default `~/.cache/ffufai/knowledge.json`).  
+  Example: `ffufai --knowledge-path /tmp/ffufai-knowledge.json -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--signature-path`: Path to the tech signature JSON file (default `config/tech_signatures.json`).  
+  Example: `ffufai --signature-path /tmp/tech_signatures.json -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--wordlist-catalog`: Path to the wordlist catalog JSON file (default `config/wordlist_catalog.json`).  
+  Example: `ffufai --wordlist-catalog /tmp/wordlist_catalog.json -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--providers`: Comma-separated provider order (gemini,openai,anthropic,groq,openrouter).  
+  Example: `ffufai --providers gemini,openai,groq -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--no-rotate`: Disable provider/key rotation.  
+  Example: `ffufai --no-rotate -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--probe-methods`: Use OPTIONS to check allowed HTTP methods and include in AI context.  
+  Example: `ffufai --probe-methods -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--dns-tls`: Enrich context with DNS and TLS metadata.  
+  Example: `ffufai --dns-tls -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--error-probe`: Probe a random error page for context.  
+  Example: `ffufai --error-probe -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--ai-strategy`: Use AI to tune mode and list sizes.  
+  Example: `ffufai --ai-strategy -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--recon`: Enable recon-driven wordlist generation (robots/sitemap/JS).  
+  Example: `ffufai --recon -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--recon-max-js`: Max JS files to mine for paths (default 5).  
+  Example: `ffufai --recon --recon-max-js 10 -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--no-persist`: Disable persistence of successful findings.  
+  Example: `ffufai --no-persist -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--report`: Print a concise, AI-generated attack plan report.  
+  Example: `ffufai --report -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--feedback-loop`: Run a refinement pass based on ffuf results (wordlist mode).  
+  Example: `ffufai --wordlists --feedback-loop -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--feedback-rounds`: Number of refinement rounds (default 1).  
+  Example: `ffufai --wordlists --feedback-loop --feedback-rounds 2 -u https://example.com/FUZZ -w wordlist.txt`
+
+- `--targets-file`: Run against multiple URLs from a file (one URL per line).  
+  Example: `ffufai --targets-file targets.txt -w wordlist.txt`
+
 - `-u`: Specifies the target URL. This parameter is required and should include the FUZZ keyword.  
   Example: `ffufai -u https://example.com/FUZZ -w wordlist.txt`
 
@@ -104,7 +214,18 @@ All other ffuf parameters can be used as normal. For a full list of ffuf paramet
 
 - ffufai requires the FUZZ keyword to be at the end of the URL path for accurate extension suggestion. It will warn you if this is not the case.
 - All ffuf parameters are passed through to ffuf, so you can use any ffuf option with ffufai.
-- If both OpenAI and Anthropic API keys are set, ffufai will prefer the OpenAI key.
+- Provider priority defaults to Gemini → OpenAI → Anthropic → Groq → OpenRouter (override with `--providers`).
+- Wappalyzer-style signature data lives in `config/tech_signatures.json` and can be extended.
+- Wordlist catalog mappings live in `config/wordlist_catalog.json` and can be extended.
+
+## Research Directions
+
+Ideas to push AI-assisted fuzzing further:
+
+- Hybrid wordlist generation that blends static knowledge bases with live target telemetry.
+- Multi-model voting with confidence scoring and rate-aware routing.
+- Passive asset graphing (JS maps, API schemas) feeding scoped fuzz queues.
+- Active learning that promotes repeated high-signal discoveries into persistent dictionaries.
 
 HUGE Shoutout to zlz, aka Sam Curry, for the amazing idea to make this project. He suggested it and 2 hours later, here it is :)    
 <img width="744" alt="image" src="https://github.com/user-attachments/assets/9f914cc4-fe5f-4dbc-b7d9-548473ea2134">

ROADMAP.md

@@ -0,0 +1,29 @@
+# ffufai Roadmap
+
+This roadmap maps the major capability areas and how they build on each other.
+
+## Phase 1: Accuracy & Signal
+- Multi-phase AI inference (plan → generate → verify) for extensions and wordlists.
+- Technology fingerprinting from headers + HTML + cookies to improve context.
+- Wappalyzer-style signatures and config-driven tech mappings.
+- Tech-aware wordlist catalog selection and recon-driven list generation.
+- Profiles/goals to bias toward critical targets and reduce noise.
+
+## Phase 2: Performance & Scale
+- Caching of AI results for repeated scans.
+- Auto-mode selection to choose wordlist vs extension fuzzing.
+- Targets file support for batch runs.
+
+## Phase 3: Intelligence & Refinement
+- Consensus mode across Gemini, OpenAI, Anthropic, Groq, and OpenRouter.
+- Feedback loop using ffuf JSON results to refine wordlists.
+- Adaptive category-driven wordlist switching.
+- Concise attack-plan reporting to guide follow-up actions.
+- DNS/TLS and error-page enrichment signals.
+
+## Phase 4: Expansion
+- Expand tech knowledge base and platform indicators.
+- Add more profiles (e.g., API-only, SPA) and goals.
+- Add persistence for successful findings and active learning.
+- Cross-target learning with global knowledge base.
+- Add provider rotation and multi-key management.