Skip to content

IMDB Data Parser #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
breaclark wants to merge 59 commits into
base: master
Choose a base branch
from
Open

IMDB Data Parser #15

breaclark wants to merge 59 commits into from

Conversation

@breaclark
Copy link

@breaclark breaclark commented Jun 12, 2015

Zafer Cakmak and others added 30 commits January 2, 2013 08:22
README updated. Running help added
8d44544
@xaph
added license file for imdbparser. #1
422018c
@aykutakin
Changes regex and hardcoded csv to seperator
6fbf7c0
@aykutakin
Adds genre parser
c2d056e
@aykutakin
Adds rating parser
2bf5032
@xaph
#1 added full text of GPLv3
5a23522
@xaph
closes #1 project is GPLv3 now.
3114bed 
- GPL statement added to all source files.
@xaph
started to convert filehandler to object
fd43af0
@aykutakin
Adds directors parser and improves #TITLE regex
4ef0b08
@aykutakin
Fixes skipped rows numbers and flow
fce0432
@destan
refactor Parsing classes and CLI arguments
e7d3cad
@destan
fix help text: executing section
d3d5067
@xaph
updated example settings file to use all active lists
78a8d3f
@xaph
removed real paths from example file
ed0f417
@xaph
added database config to example settings file #11
9148a42
@xaph
parse_all method splitted as parse_one
07684ef
@xaph
file handler objectified
46921cb
@xaph
parsers uses objectified file handler
6cb2e7c
@aykutakin
Fixes encoding problem & improves director regex
7360ce3 
A few miss still there for director regex..
Also regex must be improved for this kind of records: Warman, Bernardo
(I) Dragonboy (2011)
@aykutakin
adds end of data separator and #fixes 13
abdc237
@aykutakin
fixes #5 and fixes #4
f8fdeb6
@aykutakin
fixes #16 and fixes #15
6a4c34d 
Log file path is same as OUTPUT_DIR.. If you want to change log path you
can create a new setting variable and change loggerprovider.py file..
@aykutakin
fixes #14
39e2074
@aykutakin
Adds time to folder and initial commit for trivia
ecb0352
@aykutakin
Adds trivia parser
9eca8f8
@destan
fix 'No such file or directory' error
d46dfb4 
...by converting line endings to unix endings
@destan
add comment to debugging snippet, format comments
2c5b96f
@destan
add comments
d791cb9
@xaph
partial fix #11: db class implemented
438c98b 
- a new db class waits us. it can create new tables and inserts data
- tests for db class added
- readme updated. psycopg2 added as requirement
@yasakbulut
add python-google-api-python-client dependency to README
ca56e05
yasakbulut and others added 29 commits March 5, 2013 15:26
@yasakbulut
fix wrong package name. doh!
0f34982
@yasakbulut
add initial version of freebaseagent.
48554a1
@yasakbulut
remove unneccessary dependency
b0f067b
@yasakbulut
add comments, missing parameter in getImdbId()
9457f95
@destan
close #12 use decorators to measure parsing durations
9cd9b4c 
* removed old comments and TODOs
* added a few docs
@xaph
db interactions removed
de75c86
@xaph
#26 #30 alpha stage of db output
ef4c6c0
@xaph
#26 movie names escaped for sql inserts
b191b32
@xaph
updated output directory path. removed space character from directory…
d9e1100 
… name
@xaph
closes #31 file creation moved to base parser. base parser checks whi…
75cee99 
…ch file to create now
@destan
fix args logs
91d9d36 
make use of processed versions of parameters where available instead of raw versions directly from argument list
@destan
edit 'finish log' to be more informative
1d65cb1
@destan
add log to print outputfolder at the end
bba3d31
@destan
fix #35 plot parser produces 0 byte in tsv mode
989c6c3 
* added endOfDumpDelimiter abstract property to BaseParser
- Some dump files has comments at the end of the dump as well as at the start. Comments at the start are avoided by numberOfLinesToBeSkipped property and
comments at the end are avoided by endOfDumpDelimiter property.
- endOfDumpDelimiter property is used for stoppping parsing of associated file when encountered. Hence indicates end of the dump.
For each file this may be different or non-exist at all.
@destan
add -u option and smth about dump files to README
d9bb083
@destan
fix typo in -h help text
d4149d1
@xaph
closes #26 sql dumps for movies added.
e1f1d96 
- readme updated
- sql dump for movies finished
- db removed from modes. we're just creating sql or tsv files
@xaph
#25 genres SQL added.
7afe94b
adds more parametric db operations
81d5c98
adds more parametric db operations
6807407
tries to implement naming convention
19a2859
@aykutakin
Adds sql scripts
1598830
@aykutakin
Adds commit to sql files
c882428
@aykutakin
Fix sql file bug
7aa48a7
@aykutakin
Some files cannot be opened with utf-8
67c094c
'FileHandler' has no attribute 'get_full_path' error fixed.
f9e1bcd
@mairbek
Fixed path string building
179b941
@xaph
Merge pull request #36 from mairbek/fixpath
a0640e5 
Fixed path string building
@aykutakin
closes #7 writing processed lines to console increases time too much
20671ab 
Necessary line is still inside the baseparser.py file. If you want to see process of parsing, you can uncomment out that line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@breaclark @xaph @destan @yasakbulut @aykutakin @mairbek