Skip to content

Feature/ingestion/data cleaner#44

Open
immangat wants to merge 48 commits intomainfrom
feature/ingestion/data_cleaner
Open

Feature/ingestion/data cleaner#44
immangat wants to merge 48 commits intomainfrom
feature/ingestion/data_cleaner

Conversation

@immangat
Copy link
Contributor

@immangat immangat commented Feb 22, 2024 •
edited by Zara-Ra
Loading

Description

Issue Link: Please add the link to the issue here

This PR adds a pipeline to check for an empty registration number and to format the title and name of the listing consistently.

Type of Change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration.

This was tested manually.

Checklist:

Before you submit your pull request, please make sure you have completed the following:

  • I have read the CONTRIBUTING document.
  • I have checked that my code adheres to the code style of this project.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • My changes generate no new warnings.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • Any dependent changes have been merged and published in downstream modules.

Screenshots (if applicable)

Include any relevant screenshots or screen recordings demonstrating your changes.

Additional Notes

Add any other notes about the pull request here.

immangat added 30 commits February 5, 2024 09:38
@immangat
Setup empty scrapy project.
90795ca
@immangat
Added idea folder to gitignore.
254ab34
@immangat
Updated ingestion module readme.
ba5e436
@immangat
Added readme for the scrapy project.
9fe60ab
@immangat
Added requirements.txt for pip installation.
6f7253e
@immangat
Updated README.md.
6720eda
@immangat
Updated the ingestion module readme
f2bd300
@immangat
Deleted redundant readme
1fc5795
@immangat
Fixed some formatting.
4bad930
@immangat
Merge branch 'main' into ingestion_module/project_setup
d343fee
@immangat
Made Initial listing spider.
d10e303
@immangat
Added ipython shell for easier testing for requests etc using scrapy
66dc23e
@immangat
implement parse to get airbnb listing ids for one page.
4f478ef
@immangat
parse can now get listings on all pages.
90512a0
@immangat
Added item that represents an airBnB Listing
d519da7
@immangat
Added registration id to listing Item.
94c3c38
@immangat
Added ability to get registration_ID.
c9e2ab6
@immangat
Changed obey robot.txt to false.
fd792a8
@immangat
Added docstrings and made helper methods.
b997600
@immangat
Updated requirements.txt to include only needed dependencies.
488a70a
@immangat
Updated README.md to include usage instructions.
00f6085
@immangat
updated doc strings for handle_listing
1a1c114
@immangat
Merge remote-tracking branch 'origin/main' into ingestion_module/feat…
c93c34a 
…ures/listing_spider

# Conflicts:
#	ingestion/README.md
#	ingestion/airbnb_scrapping/airbnb_listings/items.py
#	ingestion/airbnb_scrapping/airbnb_listings/settings.py
#	ingestion/airbnb_scrapping/scrapy.cfg
#	ingestion/requirements.txt
@immangat
deleted commented out code.
656a8d7
@immangat
updated item to include latitude and longitude field.
42ca249
@immangat
Added code to get latitude and longitude of listing.
124689c
@immangat
Added docstrings to items.py
81af792
@immangat
Added pipeline to check for duplicate airbnb listing values in csv
ab7f2dc
@immangat
toggles AirBnbListingsDuplicatePipeLine is settings.py
e6a2148
@immangat
Added docstrings.
ee47373
immangat and others added 18 commits February 19, 2024 12:45
@immangat
Deleted unnecessary imports.
63014e1
@immangat
AirBnbListingsDuplicatePipeLine is now initialized with custom settin…
281f6c3 
…gs from settings.py
@immangat
Added settings for csv file name.
1126ad2
@immangat
Moved feeds setting from spider to settings.py
367df39
@immangat
Added more start_urls.
106c64e
FEEDS added to setting to save listings as csv file
6d4d677
@immangat
Added docstrings.
b69f83b
Merge remote-tracking branch 'origin/main' into feature/ingestion_mod…
4a420dc 
…ule/repository
README updated for configuring how to save listings in a file
d830b63
added Custom Settings to store csv file
c09ff2d
@immangat
Merge branch 'feature/ingestion_module/repository' into feature/inges…
512fc06 
…tion/deduplicate_listings

# Conflicts:
#	ingestion/airbnb_scrapping/airbnb_listings/settings.py
@immangat
Added data cleaner pipeline
1fcd753
@immangat
Added setting to enable pipeline.
8b31b71
@immangat
Merge branch 'feature/ingestion/deduplicate_listings' into feature/in…
db38c02 
…gestion/data_cleaner
@immangat
Added new pipeline for data cleaning.
a25dec7
@immangat
enabled data cleaner pipeline in the settings.
5b586fb
@immangat
Updated docstrings in AirbnbListingsPipelineDataCleaner
0ec8cb6
@immangat
changed variable name.
2b577b7
@immangat immangat added the enhancement New feature or request label Feb 22, 2024
@immangat immangat added this to the 1. MVP milestone Feb 22, 2024
@immangat immangat self-assigned this Feb 22, 2024
@immangat immangat requested a review from umsu2 as a code owner February 22, 2024 02:43
@immangat immangat linked an issue Feb 22, 2024 that may be closed by this pull request
[MVP] Create a data cleaner #21
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@umsu2 umsu2 Awaiting requested review from umsu2 umsu2 is a code owner

At least 0 approving reviews are required to merge this pull request.

Assignees

@immangat immangat

Labels

enhancement New feature or request

Projects

Airbnb regulation project
Status: No status

Milestone

1. MVP

Development

Successfully merging this pull request may close these issues.

[MVP] Create a data cleaner

1 participant

@immangat