Skip to content

Feature/ingestion/deduplicate listings#43

Open
immangat wants to merge 42 commits intomainfrom
feature/ingestion/deduplicate_listings
Open

Feature/ingestion/deduplicate listings#43
immangat wants to merge 42 commits intomainfrom
feature/ingestion/deduplicate_listings

Conversation

@immangat
Copy link
Contributor

@immangat immangat commented Feb 22, 2024

Description

Issue Link: Please add the link to the issue here

This pull request is a part of the data cleaning process within the ingestion module. It implements a check to ensure that duplicate listings are not saved to the datastore.

Type of Change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Currently not tested, tests will be added in a future pull request

Checklist:

Before you submit your pull request, please make sure you have completed the following:

  • I have read the CONTRIBUTING document.
  • I have checked that my code adheres to the code style of this project.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • My changes generate no new warnings.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • Any dependent changes have been merged and published in downstream modules.

Screenshots (if applicable)

Include any relevant screenshots or screen recordings demonstrating your changes.

Additional Notes

  • The pipeline will be modified in a future PR to make it more general so it can work with call types of datastores, not only csv.

immangat added 30 commits February 5, 2024 09:38
@immangat
Setup empty scrapy project.
90795ca
@immangat
Added idea folder to gitignore.
254ab34
@immangat
Updated ingestion module readme.
ba5e436
@immangat
Added readme for the scrapy project.
9fe60ab
@immangat
Added requirements.txt for pip installation.
6f7253e
@immangat
Updated README.md.
6720eda
@immangat
Updated the ingestion module readme
f2bd300
@immangat
Deleted redundant readme
1fc5795
@immangat
Fixed some formatting.
4bad930
@immangat
Merge branch 'main' into ingestion_module/project_setup
d343fee
@immangat
Made Initial listing spider.
d10e303
@immangat
Added ipython shell for easier testing for requests etc using scrapy
66dc23e
@immangat
implement parse to get airbnb listing ids for one page.
4f478ef
@immangat
parse can now get listings on all pages.
90512a0
@immangat
Added item that represents an airBnB Listing
d519da7
@immangat
Added registration id to listing Item.
94c3c38
@immangat
Added ability to get registration_ID.
c9e2ab6
@immangat
Changed obey robot.txt to false.
fd792a8
@immangat
Added docstrings and made helper methods.
b997600
@immangat
Updated requirements.txt to include only needed dependencies.
488a70a
@immangat
Updated README.md to include usage instructions.
00f6085
@immangat
updated doc strings for handle_listing
1a1c114
@immangat
Merge remote-tracking branch 'origin/main' into ingestion_module/feat…
c93c34a 
…ures/listing_spider

# Conflicts:
#	ingestion/README.md
#	ingestion/airbnb_scrapping/airbnb_listings/items.py
#	ingestion/airbnb_scrapping/airbnb_listings/settings.py
#	ingestion/airbnb_scrapping/scrapy.cfg
#	ingestion/requirements.txt
@immangat
deleted commented out code.
656a8d7
@immangat
updated item to include latitude and longitude field.
42ca249
@immangat
Added code to get latitude and longitude of listing.
124689c
@immangat
Added docstrings to items.py
81af792
@immangat
Added pipeline to check for duplicate airbnb listing values in csv
ab7f2dc
@immangat
toggles AirBnbListingsDuplicatePipeLine is settings.py
e6a2148
@immangat
Added docstrings.
ee47373
immangat and others added 11 commits February 19, 2024 12:45
@immangat immangat added the enhancement New feature or request label Feb 22, 2024
@immangat immangat added this to the 1. MVP milestone Feb 22, 2024
@immangat immangat self-assigned this Feb 22, 2024
@immangat immangat requested a review from umsu2 as a code owner February 22, 2024 01:51
@immangat immangat linked an issue Feb 22, 2024 that may be closed by this pull request
[MVP] Dealing with duplicate Listings #42
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@umsu2 umsu2 Awaiting requested review from umsu2 umsu2 is a code owner

At least 0 approving reviews are required to merge this pull request.

Assignees

@immangat immangat

Labels

enhancement New feature or request

Projects

Airbnb regulation project
Status: No status

Milestone

1. MVP

Development

Successfully merging this pull request may close these issues.

[MVP] Dealing with duplicate Listings

1 participant

@immangat