Instructions

ImageClassification.py: takes in jpeg in a pre-specified folder (./images), runs a ??? model to return tags app.py: flask webserver running react app sql.py: has sql POST functions for tag and image

1. Download / git clone the code. Unzip if required.
2. install chrome extension (open chrome, extensions > dev mode > load unpacked > select the extension folder)
3. Open folder in VSCode, make a virtual env (<=python 3.10) [varies by operating system, google how to do it]
4. run pip install -r requirements.txt in terminal
5. open the root folder of the project in VSCode, run app.py
6. In another terminal, run: npm install npm start
7. add images of your wardrobe items to ./images
8. go to uniqlo, click on products and should work!

If it doesn't, work, troubleshoot by:

1. Try simply restarting the extension from extension settings.
2. Try reloading the Uniqlo website.
3. Inspect element Uniqlo site, open console, search 'contentScript', and the contentScript.js 's logs should be visible.

Inspiration

Overspending, and impulse buying clothes and accessories, when much can be done with what you already have.

Initial Concept Roadmap

What it does

When on a website such as Uniqlo (currently the only supported one), matches one's wardrobe under ./images, and compares it to the current image of a product on screen. If the item on the website is similar enough (determined by a machine learning model), an alert will show up informing you of such. Other features might make it if we have time! such as AI suggested outfits, and tagging showing up on react page.

How we built it

Using react and tailwind css for the front end, python flask for the backend and 2 machine learning models -> one for tagging, the other for determining similarity of two images. An SQL database of tags and images is also created to link tags to images.

Challenges we ran into

1. Tailwind css has a learning curve!
2. The first python ML similarity model using euclidian distancing, image flattening and resizing to achieve a similarity ranking. Though, this would function differently in varying situations, being generally unreliable, requiring a new model that used ORB (Oriented FAST and Rotated BRIEF) for feature detection and Brute Force Matcher to compare the descriptors of keypoints in both images to find the best matches.
3. The browser extension was very new to everyone involved, and required a team effort to pull off. We faced problems related to javascripting functions and responses; DOMContent listener would never actually respond in time, or finish, thus decided to implement a 3.2 second timer to call the functions needed for scraping needed data. Finding the data itself was a tough challenge, finding images on a dynamic website like Uniqlo to save, and run comparisons on.
4. As we were all inexperienced in many of the facets of making, coordinating a project, in addition to not knowing much of the technologies such as react developement, ML model training, flask backend, and the extension.

Accomplishments that we're proud of

Everything; the GUI looking nice, with dynamic tiles updating on the react home page if one drags in an image into ./images. The backend flask server interfacing with the browser extension and responding to calls, running functions and replying to REST requests.

What we learned

React, tailwind css, teamwork, teambuilding, coordination, ideation, opencv, image matching, working separately on parts of a project to bring it all together.

What's next for Closet Companion

Add more websites than Uniqlo, use openai API chat embeddings to send images, and recieve combinations of auto-suggested outfits. Currently the react page's "add" section doesn't save the image in to the local folder, which needs to be worked on. In addition, the tags that are created by sql.py can be displayed under the images on the home page, and a search function should help users search large wardrobes for certain pieces.

Final GUI / UX after 1 day hacking!