Max crawl depth not working #482
Unanswered
sasa-andjelic-nqode
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
When I start a crawl without setting the max crawl depth, the crawler scans for links on all pages.
But when I set the max crawl depth to some int value (for example 100), the crawler only scans the first layer of links and crawls those.
I took a peek into Crawler->addToDepthTree() implementation and I see that if the max depth is null any url is instanciated as node. The rest of the conditions inside the method implementation do not cover a case for child links.
I also didn't notice any increment for current depth of the link.
It looks like max depth was never implemented completely. Is it that or am I perhaps missing something obvious which I should pay attention to?
Beta Was this translation helpful? Give feedback.
All reactions