Adaptation to Python 3, new features, bug fixes by jtara1 · Pull Request #58 · HoverHell/RedditImageGrab

jtara1 · 2016-09-14T21:14:15Z

If possible, please merge this pull request into a new branch.

My fork has been adapted to Python 3. Here's a summary of my changes in my merge-ready branch:

Python 3 only
--num cli arg counts by reddit submission rather than individual image from imgur
file ._history.txt stores last reddit id downloaded tied to local directory, subreddit name, & subreddit sort_type which is used to resume download position on the subreddit when the 3 aforementioned are matched.
arguments changed
- <subreddit> will autodetect whether a text file or subreddit name was passed
- --restart begin downloading from the beginning of the subreddit
- --subreddit-list srl-filename download from each of the subreddits found in a text file

In addition to these changes there are fixes to the original code.

See my readme for more details
https://github.com/jtara1/RedditImageGrab/tree/merge-ready#jtara1-fork

…date to rename

…ch of them

…ine character encountered

…king now

…t sent in header in request

rachmadaniHaryono

need a lot of change until it can be merged. and also testing is required.

rachmadaniHaryono · 2016-09-15T00:09:32Z

README.md

 any JPEG or PNG formatted image that it found listed in the specified
 subreddit and download them to a folder.

+## jtara1 Fork


only for fork

rachmadaniHaryono · 2016-09-15T00:11:32Z

README.md

+
+* \-\-num cli argument now counts by reddit submission rather than individual image
+
+    * added my fork (jtara1/imgur-downloader) to handle all imgur downloads (making the above feature possible)


fork only info. maybe change it into

added new library to handle ...

rachmadaniHaryono · 2016-09-15T00:30:46Z

README.md

+    * added my fork (jtara1/imgur-downloader) to handle all imgur downloads (making the above feature possible)
+
+
+* file .\_history.txt contains reddit id of last downloaded and is identified by subreddit & ARGS.sort\_type, e.g.:


file ._history.txt contains reddit id of last downloaded and is identified by subreddit & ARGS.sort_type, e.g.

rachmadaniHaryono · 2016-09-15T00:31:30Z

README.md

+
+    > {'wallpapers': {'topmonth': {'last\-id': '4x4so2'}}}
+
+* positional arguments, \<subreddit\> and \<dest\_file\>, changed to optional cli arguments


positional arguments, <subreddit> and <dest_file>, changed to optional cli arguments

rachmadaniHaryono · 2016-09-15T00:37:14Z

README.md

+
+    *    positional arguments changed
+
+    *    --dir is required


if dir is required, don't use -- prefix.

Do you think its ok to revert back to using two positional arguments but have the first autodetect whether it's a subreddit name, multireddit name, or subreddit-list file?

it is possible. use os.path.isfile to check if input is filename and check if it have .txt extension. something similar like this

if os.path.isfile(filename) and os.path.splitext(filename)[1] == '.txt':

rachmadaniHaryono · 2016-09-15T01:50:56Z

redditdownload/redditdownload.py

+            f.write(str(write_data))
+            return True
+    else:
+        print ('Error in historyLog func: invalid mode')


use logging.debug so user don't see any print from this debug func.

rachmadaniHaryono · 2016-09-15T01:55:25Z

redditdownload/redditdownload.py

+        write_data: any    - only relevant if mode == 'write', output is converted to string and written to file_log
+    '''
+    path = os.path.join(os.getcwd(), wdir) if not os.path.isdir(wdir) else wdir
+    if mode == 'read':


mode_dict = { 'read': 'r', 'write': 'w', 'append': 'a' } if mode in mode_dict: with open(os.path.join(path, file_log), mode_dict[mode]) as f: if mode == 'read': # http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html return eval(f.read()) # this is dangerous else: f.write(str(write_data)) return True else: logging.debug(dbg_message) # return False # ?

rachmadaniHaryono · 2016-09-15T02:00:05Z

redditdownload/redditdownload.py

-        start_time = time.clock()
+    if ARGS.subreddit_list:
+        ARGS.subreddit_list = ARGS.subreddit_list[0]
+        SUBREDDIT_FILE = os.path.join(os.getcwd(), ARGS.subreddit_list)


variable not written in capital letters. old arguments are lot like this and yet not fixed. we don't want to continue that.

rachmadaniHaryono · 2016-09-15T02:02:26Z

subreddits.txt

@@ -0,0 +1,3 @@
+pc-wallpapers:


need more example. function doc have more.

rachmadaniHaryono · 2016-09-15T02:03:34Z

redditdownload/tests/test_redditdownload.py

+import os, sys
 from os import getcwd
-
+sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))


does it have to append it to system path? if you use nose or pytest i think it can find it.

rachmadaniHaryono · 2016-09-15T02:12:48Z

subreddit file list is a good feature, but it still need a lot improvement.

here is an example how text file can be used as input list.

https://github.com/Bakkingamu/tumblr_image_download_script

it support commenting and category(in our case we can use sort type etc).

try also documenting the function and method. use google doc style or sphinx style. if it goes right, we can build sphinx docs from the project.

jtara1 · 2016-09-15T02:47:16Z

Thanks for all the feedback; these mostly look like small changes. I haven't collaborated on programming projects like much until recently.

jtara1 · 2016-09-17T06:19:38Z

Here's most of the changes @rachmadaniHaryono recommended. In addition there's a critical fix for imgur-downloader, and a new feature, --restart command line interface argument.

Not sure if my readme.md is exactly how you'd like.

ohyou and others added 30 commits June 19, 2016 12:15

Perform 2to3

563c140

Make it work as a module

56b361d

Remove unnecessary output

2a4477e

Fix errors when downloading from gfycat

60f0856

Handle some errors

236a592

update .gitignore

8c1311f

Merge https://github.com/ohyou/RedditImageGrab

ab18926

downloads imgur imgs using jtara1/imgur-album-downloader

bf02892

fix func names & update .gitignore

0edcfb3

fix slugify func so --filename-format arg works

75f4c8b

added func to make cleaner filename, update readme

3d29e43

updates & fixes, func process_deviant_url works, & ImgurDownloader up…

1758e32

…date to rename

new feature, last-id tracking in file '._history.txt'

9258a07

new feature, load subreddits from subreddits text file and process ea…

d32ca53

…ch of them

update readme.md & add todo.md

b401526

fix, [--subreddit-list srl] and [--dir <dest_file>] work properly now

1c85413

update DOWNLOADED, ERRORS, & other vars to keep track of progress

0186590

update readme

2c5321d

critical fix, parse_subreddit_list will now break main loop when newl…

167e5df

…ine character encountered

critical fix, --subreddits-list arg and everything else should be wor…

6539870

…king now

parse_subreddit_list.py now skips iteration (continue) when line=='\n'

e90d53d

reorganized loop to handle subreddits more clearly, changed user-agen…

82a89d8

…t sent in header in request

update readme & todo

839fc57

fix ARGS.num checker, update readme

72a60b9

update readme

af0b953

add jtara1/imgur-downloader repository

3361953

added jtara1/imgur-downloader & rm test imgs

28bf0d1

update, rename some files

ec208d5

CRIT FIXES: url corrected in reddit.py, PROG_REPORT variables corrected

a3c430c

update

90e6860

jtara1 added 6 commits September 12, 2016 01:17

hot fix for gfycat & printing is more helpful

1cd0764

update readme

3517e8d

update readme

5a92db0

fix gfycat downloading, Does Not Exist errors are handled properly now

ba4e8e6

update txt

9177594

update readme

8c9be20

jtara1 changed the title ~~Python 3 - Merge ready~~ Adaptation to Python 3, new features, bug fixes Sep 14, 2016

rachmadaniHaryono suggested changes Sep 15, 2016

View reviewed changes

jtara1 added 9 commits September 15, 2016 01:47

update imgur-downloader

eeb204e

rm imgur-downloader/readme.md

3426c5d

add subreddit list examples

8e4787e

update history_log, cli args, readme, & vars renamed

e113d2c

new cli arg, --restart (begins downloading from beginning of subreddit

ab53b4a

update readme, todo, minor fixes

f9a27f1

update history_log func, imports relocated in gfycat.py

ef89c68

update parse_subreddit_list, CRIT FIX: imgur-downloader

2c9ffda

update imgur-downloader docstrings

5399043

hotfix for when last ITEM is comment thread (caused infinite looping)

ddf0745

rachmadaniHaryono mentioned this pull request Apr 22, 2018

Python v3 update #82

Open


		* \-\-num cli argument now counts by reddit submission rather than individual image

		* added my fork (jtara1/imgur-downloader) to handle all imgur downloads (making the above feature possible)

		* added my fork (jtara1/imgur-downloader) to handle all imgur downloads (making the above feature possible)


		* file .\_history.txt contains reddit id of last downloaded and is identified by subreddit & ARGS.sort\_type, e.g.:


		> {'wallpapers': {'topmonth': {'last\-id': '4x4so2'}}}

		* positional arguments, \<subreddit\> and \<dest\_file\>, changed to optional cli arguments

Conversation

jtara1 commented Sep 14, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rachmadaniHaryono left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rachmadaniHaryono Sep 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rachmadaniHaryono Sep 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rachmadaniHaryono commented Sep 15, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jtara1 commented Sep 15, 2016

Uh oh!

jtara1 commented Sep 17, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jtara1 commented Sep 14, 2016 •

edited

Loading

rachmadaniHaryono Sep 15, 2016 •

edited

Loading

rachmadaniHaryono Sep 15, 2016 •

edited

Loading

rachmadaniHaryono commented Sep 15, 2016 •

edited

Loading