LLM data crawler

[Msiusko]

We need to build a web crawler - add a website as a substitution of manual input of data.

Impacting

• up to date information
• speed of data fulfillment
• accuracy

Functioning

• check privacy project's webs
• update new info constantly (push PR? propose clean CSV-dataset?)
• sensitive data - moderated, basic data - not (need to define differences)

Early scouting options

• Venice AI (Diem) (free besides the cost of Purchasing DIEM) (can be used for lightweight queries for the explorer)
• Akash (test out chat.akash.network) - plug in the explorer-data, give it a prompt, explore the models, try out, test out, give it the spec, give an example
• Another option is a local model running on someones computer
• build a decentralized AI solution (Hackathon project)

[vpavlin]

Given that I came up with the items in Early scouting options, let me elaborate:)

• Akash and Venice serve the same purpose - get access to the cheap/free LLM-inference.
  • Main difference is that Akash seems to be tightening the bolts - used to be no-sign-up free for all, no it is 1 token per user, eventually payment might come
  • Venice feels like reasonably aligned and longer term solution - maybe there is a way to get "sponsored" if you reach out to Erik?
  • Might also reach out to other "private AI" projects (Nillion, Dria and others?) for access to compute...
• If someone in the community has unused/under-utilized GPU, then we can definitely build on top of that - would be free (+ the energy costs), but also depending on that person providing access
• The "decentralized AI" is less about actually doing something AI specific, more about "decentralized way to distribute tasks" - where task is "processing a website using AI to extract information" - actually Dria might be interesting partner to reach out to as they are kind of doing that already

How to prototype this

• Figure out the output spec (ideall use actual JSON spec or something)
• Give it to the AI as an expected output
• Download the source of information (a project website?) as markdown files and attach them to the prompt
• Create a prompt that will instruct the LLM to analyze the markdown files and output information based on the spec

This might not be enough because the website might not contain all the info + it might be too big for the context, so a RAG or some kind of MCP might come into place to make sure the LLM/scraper/agent can properly search the contents.

Decentralized AI crawler

• Allow users to run a "scraper node" which advertises 1) model(s) it has, 2) capacity (e.g. tokens-per-second for each model)
• Allow some "special" users (based on some whitelist, token, RLN..) to publish a task
• Scraper node runners acknodledge a task, task submitter confirms the task to one node runner
• Node runner has some amount of time (e.g. 1 minute?) to publish the response
• Task submitter acknowledges the response or re-publishes the task based on the result
• Task submitter updates the data

---

• Communication can happen over a p2p network (e.g. Waku) or via blockchain (careful with gas fees, could be interesting to look into Status L2?)
• Responses from sraper nodes are JSON or YAML objects which may require some post-processing
• PR is pushed to the explorer-data repo with new data
• PRs might need human approval

The idea of this is that people can plug-in any LLM-inference provider - Akash, unused Venice API quota, OpenAI, Grok, local LLMs - and we can just send them tasks - this is actually what Dria does:-) But it has some explicit goals and requirements, so we might need a bit more flexible setup:)

[lapets]

Nillion Labs CTO here. We would love to support anyone working on this with free access to TEE-powered private LLM inference! Feel free to reach out to us (@buildonnillion) and we can quickly arrange API access.

[vpavlin]

@lapets Very cool:) Bear in mind that private inference might not be needed here - assuming this will consume public resources and just crunch the data:) But it might be a good way to scale test Nillion API :D

[airpods69]

If someone in the community has unused/under-utilized GPU, then we can definitely build on top of that - would be free (+ the energy costs), but also depending on that person providing access

@vpavlin Staying at the mercy of a partner wouldn't be the best idea honestly. depends on that person and if needed then you'd be kicked pretty quickly off (Maybe a bit too radical to think like this but definitely something to think about)

As for how this could run, if payments are needed to stay closer to the lower sides then using deepseek's API could prove cheaper compared to the other partners. I believe groq would be even more cheaper but the problem with groq.com is that they don't let you access it over any VPN which is a big red flag for me?! Using nano-gpt.com could get you around this which is a big win in my opinion, groq models are pretty fast for a little extra charge (we are still talking about cents)

Second: @Msiusko

We will at all times have an input of the websites that are needed to be scrapped or do you want something closer to a crawler? that would be more interesting to have but then also way more difficult. Crawl and Scrape O_O

Implementing something for a webscrapper should not be an issue + there are tonnes of resources available for the same LLM + website scrapping.

^ This is just the web scrapper bit

Now, the decentralization part of it, that is something which is kinda cool ngl

It is just running the scrapper code + the plug in for which model the user wants to run it with (like what @vpavlin mentioned)
But making sure the goal was achieved is the problem
The user could post a "fake" output since now we are at the mercy of their compute to figure out if they did perform a computation or not. Unlike actual mathematical operations, this is easier to fake since API responses can be mocked too and almost all LLM requests are API requests (either ollama running locally or an online provider)

What if a user posts a generic response everytime for a scrapper task? :O

Just my rough thoughts

[undecidedpathways]

@airpods69 We're looking for a way to substitute inputting the data manually via this webpage. The goal would be for the crawler to be given a website for a new project, and from that website, the crawler would then be able to fill out as much info as can be found within the webpage and input into an index.yaml file for that project.

[madjin]

@airpods69 We're looking for a way to substitute inputting the data manually via this webpage. The goal would be for the crawler to be given a website for a new project, and from that website, the crawler would then be able to fill out as much info as can be found within the webpage and input into an index.yaml file for that project.

Do ya'll have a schema for the entire submission input data? Would be useful to post it in this issue.

[morkeltry]

Allow users to run a "scraper node" which advertises 1) model(s) it has, 2) capacity (e.g. tokens-per-second for each model)
Allow some "special" users (based on some whitelist, token, RLN..) to publish a task
Scraper node runners acknodledge a task, task submitter confirms the task to one node runner
Node runner has some amount of time (e.g. 1 minute?) to publish the response
Task submitter acknowledges the response or re-publishes the task based on the result
Task submitter updates the data

@vpavlin Staying at the mercy of a partner wouldn't be the best idea honestly. depends on that person and if needed then you'd be kicked pretty quickly off (Maybe a bit too radical to think like this but definitely something to think about)

Speaking as someone who would perhaps use my underutilised GPU for this, I don't think anyone wants to advertise themselves as a node and then have a node runner dial in with tasks.
I imagine, if community help is a way forward, and if the tasks are pretty small in the scheme of things as they appear to be (parse 700 websites?), it would look more like the central server could maintain a list of tasks to do and scraper nodes could dial in and grab one.
That would allow me to run a start-scraper-node script when I go out to the shops for an hour and not deal with the scraping competing with me for my own use of the machine.

As long as the amount of tasks are small, as they appear to be, being 'kicked off' wouldn't be a problem as if tasks back up, a human can just put a shout out out-of-band to the community to do some scraping.
If the tasks are big, the architecture would still work, but the node runner script would be set to use a remote, paid for (or donated ;), LLM provider instead.

[vpavlin]

I don't see a difference between your architecture and my proposal from the perspective of

That would allow me to run a start-scraper-node script when I go out to the shops for an hour and not deal with the scraping competing with me for my own use of the machine.

But sure, there are many ways of doing this - some more centralized, some more decentralized, some private, some not:-)

[wslyvh]

The decentralized AI/LLM discussion is interesting, and I love a good engineering challenge :)

But trying to add some more practical feedback as this is something I suggested initially. If the goal is to reduce the amount of data/inputs are required in the form: https://explorer-app-mu.vercel.app/project/create

I'd start with just a simple scenario. Add website, and then fetch standard info/metadata (e.g. title, description, etc.) from SEO/OpenGraph. That's step 1.

Step 2: Send website to a LLM (e.g. OpenRouter) to gather additional info. Start with default, structure data e.g. Links

Step 3: Ping all those links again to check activity. Not sure how up to date this is, but social presence, nr of Github repos, latest commits, packages, etc. https://paragraph.com/@web3privacy-now/explorer-scoring-mechanism-breakdown

Step 4: Token info from any tracker like Coingecko, DefiLlama, and funding from something more traditional like Crunchbase

That's probably a pretty good starting point for most of the manual input data. Happy to jam more about next steps, reproducible builds, trakcers, audits, etc.

[M0nkeyFl0wer]

I have built someting similar in at least a beta version. You can check it out here:https://github.com/M0nkeyFl0wer/ethereum-cypherpunk-research

Feedback always appreciated.