feat: Add llms.txt for better LLM discoverability

[Montekkundan]

Description

I've implemented llms.txt support using @sablier/docusaurus-plugin-llms following the llmstxt.org specification. This helps LLMs better understand and navigate Tangle's documentation when assisting users.

• llms.txt

• llms-full.txt

Implementation:

• Auto-generates llms.txt and llms-full.txt at /docs/
• Content cleaning enabled (removes imports, duplicate headings)
• Documentation ordered by sidebar structure
• Syncs automatically on every build

Open Questions

I'd appreciate feedback on these optional enhancements:

1. Generate individual markdown files?

• generateMarkdownFiles: true creates .md files instead of linking to HTML pages
• Better llmstxt.org compliance but adds more files

2. Create targeted LLM files?

• llms-getting-started.txt for beginners
• llms-component-dev.txt for developers
• Helps LLMs provide focused responses based on user level

3. Add custom root content?

• Add Tangle-specific context (key features, navigation hints)
• Helps LLMs understand what makes Tangle unique vs Kubeflow/Airflow

4. Include external resources?

• Link to TangleML/tangle repo and pipeline_components
• Could help with comprehensive answers but may dilute focus

Happy to implement any of these based on maintainer preferences. Let me know if you'd like me to open separate PRs for each enhancement.