Skip to content

Added urllib-downloader koubae#7

Open
Koubae wants to merge 1 commit intoWuLC:masterfrom
Koubae:urllib-downloader
Open

Added urllib-downloader koubae#7
Koubae wants to merge 1 commit intoWuLC:masterfrom
Koubae:urllib-downloader

Conversation

@Koubae
Copy link

@Koubae Koubae commented Dec 11, 2020

Added

download_images function

* extra_keywords default to None, so that pic can be search if only a keyword
* Change the default dir.
* Choose total pic to choosen value, if 'all' (capital doesn't matter) is passed, then download all found ulrs
* Parameter download, if False, then won't download, if True it will
* General refactoring

download_manager function

Same parameters as download_images but added

* multiprocess if set to True then it run in multiprocess mode
* debug => if set to True if will run log_to_stderr(logging.DEBUG)

logger

* Prints and add to log the messages from the functions

Fixed

* Readability 
* More user control

Removed

* os.getpid() for getting the os id I switch to current_process().name instead
we can also add Process().pid

* The download_page to downloader is delegaed to download both whole page or single url. 

Let me know what you think, I'm planning to add more functionallyty with argparse.

Signed-off-by: Koubae 58447627+Koubae@users.noreply.github.com

Signed-off-by: Koubae <58447627+Koubae@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments