Project Status & Next Steps 🚧

[Disane87]

Current Situation

The project is not abandoned. I’m still committed to it, but as life sometimes has its own plans, development has slowed down. The current codebase has grown historically and makes it quite hard to maintain or extend. Because of this, I decided to rework the foundation before continuing with new features.

Why the Rework?

Cleaner and more modular structure

Easier and faster implementation of new features

Better maintainability for the future

I personally rely on this tool as well, which makes it a priority for me

Timeline

There is no fixed timeline right now. I prefer not to make promises I can’t guarantee, since that usually only creates frustration for both contributors and users.
Think of it as: good things take time.

Commitment

This project is important to me, both personally and practically, and I will not abandon it. The upcoming rework will hopefully create a much stronger foundation for everyone who uses it.

➡️ Feel free to comment, open issues, or suggest ideas — just keep in mind that progress will take some time and that new code for the current version won’t be merged.

[ZachHandley]

Hey! Just wanted to chime in and say thank you for this repo -- it's neat and I couldn't find another way before I found this, if it's any boost to how you feel about it! :) I would definitely recommend a basic GUI which could run via a terminal command, same as before, that would make it a lot easier I think to configure and what not, and then just download from the GUI -- or a terminal like Commander or whatever they are, there's so many that work.

[Frank-Steiner]

Here are some ideas for the new version which I have just implemented for myself in the old version. Maybe you want to consider adding those...

1. If an order has no invoices, one should distinguish:
   • If the order has been canceled it can safely be ignored
   • If the order has been delivered than it likely is a marketplace seller where you have to request the invoice. I collect these orders and write them to a separate file at the end so that I can check these order manually.
2. If an order has no invoices and has not yet been delivered it will not be considered again if a more recent order has an invoice. In the next run with --onlyNew only order newer than the most recent one with an invoice will be considered, but those older orders that had no invoices before are missed.
   Therefore I've been using a simple variable remembering the most recent order number where the invoice has been saved, and whenever I find an order without invoice that has not yet been delivered I reset this variable to empty and then fill it with the next order-with-invoice found. This way I store the most recent order with invoice which is older than any not-yet-delivered order. And thus, the next run will consider those previously not delivered orders again.

For these cases I had to check the order state which (as far as I could see) can only be fetched from the textstring below div.yohtmlc-shipment-status-primaryText. So this depends on the language used and will be a problem with different languages. Maybe it is possible to enforce english language no matter what the user has set in the amazon account. Or one can allow the user to define the matching strings, for example like "Storniert", "Zugestellt|Abgeholt" or "Zustellung|wird zugestellt" for German.

Maybe this is something just interesting for me, but I thought I'd share it in case you find it interesting.

Many thanks for the docudigger :)

[dominik-ba]

Hi @Disane87 ,
I just stumbled over this repo as I recently introduced paperless ngx in my homelab and your solution really looks promising!
Anyhow I see issues for breaking Amazon changes and this issue were you state you first want to refactor basics before doing anything else. And at the same time you state you rely on this tool, too.

So I wonder will these problems be solved now or will you really first redo the basics? And how far did you progress on that? Can the community help on that?
I know very well how life can throw sticks into the plans so I think it is important to stay flexible on priorities based on changing demands.

Thanks for your work so far and sharing it with the world :)

[Disane87]

Thanks a lot for your kind words and for the thoughtful question – much appreciated 🙂

Short answer: yes, the issues you mentioned will be addressed, but not in the current repo.

The current repository is effectively in maintenance mode. Over the last months I started a more fundamental refactor, because the original architecture made it increasingly hard to react to breaking changes (like Amazon) in a clean and sustainable way.

This refactor is now ~90% done and will live in a new repository once final tests are finished.
The new architecture consists of:

• a backend with a REST API
• a small frontend for debugging and running scrape jobs
• fully atomic and composable actions defined via jsonc
• no hard-coded providers – everything is interchangeable

In later versions there will also be an editor to create new scrape jobs and compose them into workflows, which can then be shared via a kind of “store”.

Here is a current draft of the UI (still work in progress, but functionally there):

Regarding community help: once the new repo is public, help will be very welcome – especially for testing, writing providers, and improving documentation. I’ll make sure to clearly mark good first issues.

I completely agree with you that priorities sometimes have to shift – that’s exactly what happened here. The goal of the refactor is to avoid the same class of problems in the future.

Thanks again for your interest and patience 🙂

[Disane87]

Status Report — Scrape Dojo

In the last days/weeks I got really good traction with the new codebase — Scrape Dojo is shaping up into a workflow-first scraping toolkit that’s fun to use and easy to extend.

The new project will be named: Scrape Dojo

🎯 Core Features

• 📝 JSON/JSONC Workflows: Validated via JSON Schema — IntelliSense included
• 🎨 Template Engine: Handlebars templating for dynamic parameters ({{variables.*}}, {{secrets.*}})
• 🔑 Variables (Runtime/Env/Workflow): Runtime parameters, per-workflow variables (metadata.variables), global variables via SCRAPE_DOJO_VAR_*, and secrets via SCRAPE_DOJO_SECRET_* (clear resolution order)
• 🔄 Data Flow: previousData & currentData for clean action chaining
• 🔁 Control Flow: Looping & flow actions (e.g. loop, break) to build robust workflows
• 📊 Extraction: DOM reading via actions like extract, extractAll, get, getAll
• 🔧 Transformation: JSONata transforms for complex data processing inside the workflow
• 📎 Artifacts: Built-in download + screenshot actions; outputs land in ./downloads/
• 🧠 Action Metadata: Action metadata endpoint to drive UI/tooling (discoverable actions)
• 💾 Database Support: Configurable DB backend (sqlite default, plus postgres/mysql via config)

🎨 UI & Monitoring

• 📺 Live Monitoring: Real-time events/status so you can follow scrapes while they run
• 📋 Logs & Events: Detailed logs + event stream for transparent debugging
• 🚀 Quick Start: Start scrapes from the dashboard (one-click runs)
• 📊 Run History: Track executions, inspect runs, and clean up old runs
• 🛑 Stop Running Scrapes: Stop the currently running scrape from the UI/API
• 🗓️ Scheduling: Per-scrape schedules + scheduler status endpoints
• 🔎 OTP Detection: waitForOtp + OTP submit endpoint so the UI can request and forward user input
• 🔔 Notifications: UI notification support (toggle + UX hooks) — great for “run finished” / “OTP required”

🔐 Security & Auth

• 🔑 Secrets Management: Encrypted secrets at rest, CRUD + linking secrets to workflows
• 👤 Multi-User: JWT-based authentication (optional/configurable)
• 🧑‍💼 User Management: Admin bootstrap + user management UI/route for managing accounts
• 🌐 SSO/OIDC: OIDC/SSO support for enterprise setups
• 📱 MFA: TOTP-based two-factor authentication
• 📱 Device Management: Trusted devices for MFA (view + revoke devices, remove all other devices)
• 🔒 API Keys: API key mode for headless/automation scenarios

🛠️ Developer Experience

• 🏗️ Nx Monorepo: Clean separation of API/UI/Docs with fast, repeatable targets
• 📚 OpenAPI: API is documented and ships an openapi.json spec
• 🐳 Docker Ready: Docker Compose setup with persistent volumes (data, downloads, logs, browser-data, config)
• 🧪 Testing Foundation: Jest/Vitest + E2E structure in place (currently stabilizing failing UI tests)

✨ User Experience

• 🧩 Zero-Code per Site: Workflows live as versionable files under config/ — team-friendly and reproducible
• 🔎 Traceability: Runs, logs, events, and artifacts make it easy to debug and iterate quickly
• 🍪 Session Persistence: Optional persistent browser profile (browser-data) for smoother logins and fewer re-auth prompts
• 🌍 Multilingual UI (DE/EN): Current UI ships with German + English translations (Transloco)
• 🌓 Dark/Light Theme: Light, dark, and system theme modes (CSS-variable based)

🚧 Work in progress

• 💅 Polish & Bugfixing: Tighten UX, fix edge cases, and make the first MVP feel rock solid
• 🧪 Make tests green: UI test failures are currently blocking confidence — focus on stabilizing them
• 🛜 PWA Support: Make the UI installable (web app manifest + service worker), and add offline-friendly caching for the dashboard where it makes sense
• 📘 More recipes & docs: More copy/paste workflows + troubleshooting to maximize Day-1 success

🖼️ Screenshots

Have a great start into the new year 🚀🎆

[dominik-ba]

That reads and looks amazing! If you need some early human testers let me know :)