Optimize IOC extraction: remove redundant operations and data structure conversions

[Copilot]

Identified and eliminated performance bottlenecks throughout the IOC extraction pipeline.

Changes

• extractor method: Fixed incorrect list[str | None] return type and removed unnecessary filter(None, pages) and "".join() operations. pdfminer.extract_text() returns str directly.

• get_patterns method: Eliminated redundant set() → sorted() → set() conversions. Now converts once, processes, and sorts only when displaying results. Fixed counter logic to increment only after all filtering.

• detect_language method: Changed from iterating 6 fixed languages to only iterating detected languages via .items(). Reduces typical iterations from 6 to 0-2.

• patts method: Replaced explicit loop with dictionary comprehension. Maintained inline URL whitespace cleaning using conditional expression.

Example

Before:

pages = self.extractor(pdf=pdf_path)
text = "".join(filter(None, pages))  # Unnecessary operations

sorted_patterns = sorted(set(pvals))
# ... process ...
self.print_and_write_patterns(key, set(sorted_patterns), output, title)  # Convert back to set

After:

text = self.extractor(pdf=pdf_path)  # Direct assignment

unique_patterns = set(pvals)
# ... process ...
if unique_patterns:
    sorted_patterns = sorted(unique_patterns)
    self.print_and_write_patterns(key, sorted_patterns, output, title)

Net reduction: 8 lines of code (30 insertions, 38 deletions)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• astral.sh
  • Triggering command: /usr/bin/curl curl -LsSf REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

Identify and suggest improvements to slow or inefficient code

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.