With Neos.js it is easier to scrap the contents of a set of folders, and it is easier to do the scrapping this way than with the Neos plugin method. We can then scrap the folders at regular intervals to keep the local database up to date. We need to decide on a frequency that will not cause much burden to Neos' servers.
There would need to be code keeping tracks of duplicates, and updating the text embeddings for the new records
Complementarily, we can also allow users to trigger an update of a particular folder. This could be limited to one of the higher "Datreon" tiers