OpenViking add_resource耗时优化，欢迎一起讨论

[kkkwjx07]

目前来看在大规模文件导入时，OpenViking耗时依旧有较大的优化空间.

以下是我在给OpenViking导入linux源码的代码（运行前需要先安装rich库）

import time

# Set configuration file path before importing openviking to ensure it's loaded
if os.path.exists("ov.conf"):
    os.environ["OPENVIKING_CONFIG_FILE"] = "ov.conf"

from rich.console import Console
from rich.live import Live
from rich.panel import Panel
from rich.table import Table

import openviking as ov
from openviking.storage.observers import QueueObserver
from openviking.storage.queuefs import get_queue_manager
from openviking.utils import run_async


def main() -> None:
    repo_url = "https://github.com/torvalds/linux"
    resource_path = os.getenv("OPENVIKING_RESOURCE_PATH", repo_url)

    console = Console()
    client = ov.SyncOpenViking(path="./ov_data")

    client.initialize()

    console.print(Panel(f"Adding resource: {resource_path}", title="Add Resource"))
    start_ts = time.perf_counter()
    result = client.add_resource(path=resource_path)
    root_uri = result.get("root_uri", "")
    console.print(f"[bold blue]Root URI:[/bold blue] {root_uri}")

    console.print("[bold green]Waiting for processing...[/bold green]")
    queue_manager = get_queue_manager()
    observer = QueueObserver(queue_manager)
    with Live(
        Panel(observer.get_status_table(), title="Queue Status"),
        console=console,
        refresh_per_second=4,
    ) as live:
        while True:
            live.update(Panel(observer.get_status_table(), title="Queue Status"))
            if run_async(queue_manager.is_all_complete()):
                break
            time.sleep(0.5)
    elapsed = time.perf_counter() - start_ts
    console.print(f"[bold green]Done in {elapsed:.2f}s[/bold green]")

    queries = [
        "what is linux",
    ]

    for query in queries:
        console.print()
        console.rule(f"[bold cyan]Query: {query}[/bold cyan]", align="left")
        results = client.find(query, target_uri=root_uri, limit=5, score_threshold=0.0)
        table = Table(show_header=True, header_style="bold magenta")
        table.add_column("Score", justify="right", style="cyan", no_wrap=True)
        table.add_column("URI", style="green")
        table.add_column("Abstract", style="white")
        for item in results.resources:
            abstract = (item.abstract or "").strip()
            if len(abstract) > 160:
                abstract = abstract[:160] + "..."
            table.add_row(f"{item.score:.4f}", item.uri, abstract)
        console.print(table)

    client.close()


if __name__ == "__main__":
    main()

运行截图如下：

目前看到Semantic经过了DAG并发改造（https://github.com/volcengine/OpenViking/pull/121） 已经吞吐有较大提升，但是至少有以下优化空间：

• Embedding纯串行：可考虑改成并行
• 目录文件move用了20+min：目前是递归copy文件，可考虑改成move目录

欢迎一起探讨和贡献源码

[MaojiaSheng]

点赞，也许小参数模型做摘要也是个路子