Skip to content

Akanksha #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
akanksha-sarkar wants to merge 21 commits into
base: main
Choose a base branch
from
Open

Akanksha #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
15a52ee
adding info upto now
atharvas Jul 28, 2025
9597c84
adding info upto now
atharvas Jul 28, 2025
0aeb99a
update again
atharvas Jul 28, 2025
2010847
adding scikit-learn context
atharvas Jul 29, 2025
c54ec72
small changes
atharvas Aug 16, 2025
ea44e65
Merge branch 'main' of github.com:formula-code/datasmith into better_…
atharvas Aug 16, 2025
e90fdfb
upgrade pkgs
atharvas Aug 16, 2025
5ad6e95
saving progress till now; implemented an agent for filtering commits …
atharvas Aug 19, 2025
b482f40
validation working
atharvas Aug 22, 2025
39cfdbc
partial auto-installer working
atharvas Aug 23, 2025
3c99152
auto-installer working pipe-flush single
atharvas Aug 23, 2025
002f766
partially working autobuilder
atharvas Aug 23, 2025
913524d
added portkey
atharvas Aug 24, 2025
428658e
Better backup sync
atharvas Aug 28, 2025
03d8266
update benchmarking
atharvas Aug 28, 2025
e0d629e
use dotenv
atharvas Aug 28, 2025
73b651b
remove unneccessary type hints
atharvas Aug 28, 2025
98a9c4e
remove support for 3.13
atharvas Aug 28, 2025
b5f5c43
updating build agent to be reactive
atharvas Aug 30, 2025
29daac8
small errors
atharvas Aug 31, 2025
72fd7b8
log of updates. Dockerfile revamp, pipeflush previous layers
atharvas Sep 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
python-version: ["3.10", "3.11", "3.12"]
fail-fast: false
defaults:
run:
Expand Down
6 changes: 5 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,4 +195,8 @@ cython_debug/

tokens.env
benchmark_results/
artifacts/
scratch/artifacts/
output

# Database files
*.db
126 changes: 0 additions & 126 deletions CONTRIBUTING.md
View file
Open in desktop

This file was deleted.

21 changes: 0 additions & 21 deletions Dockerfile
View file
Open in desktop

This file was deleted.

36 changes: 22 additions & 14 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,54 +1,62 @@
.PHONY: install
install: ## Install the virtual environment and install the pre-commit hooks
@echo "🚀 Creating virtual environment using uv"
@echo "Creating virtual environment using uv"
@uv sync
@uv run pre-commit install

.PHONY: backup
backup: ## Create a backup of the datasets, results, and analysis directories
@echo "🚀 Creating backup archive"
@echo "Syncing backup mirror with rsync"
@/usr/bin/env bash -euo pipefail -c '\
if [ ! -f tokens.env ]; then \
echo "❌ Error: tokens.env file not found"; exit 1; \
fi; \
BACKUP_DIR=$$(grep -E "^BACKUP_DIR=" tokens.env | head -n1 | cut -d "=" -f2-); \
BACKUP_DIR=$$(awk -F= '"'"'/^BACKUP_DIR=/{print $$2; exit}'"'"' tokens.env); \
if [ -z "$$BACKUP_DIR" ]; then \
echo "❌ Error: BACKUP_DIR not defined in tokens.env"; exit 1; \
fi; \
mkdir -p "$$BACKUP_DIR"; \
zip -qr "$$BACKUP_DIR/datasmith.bckp" artifacts/benchmark_results artifacts/raw; \
cp -f artifacts/cache.db "$$BACKUP_DIR/datasmith.cache.bckp"; \
DEST="$$BACKUP_DIR/datasmith.mirror"; \
mkdir -p "$$DEST"; \
rsync -a --delete --human-readable --info=stats1 \
scratch/ "$$DEST/scratch/"; \
'

.PHONY: check
check: ## Run code quality tools.
@echo "🚀 Checking lock file consistency with 'pyproject.toml'"
@echo "Checking lock file consistency with 'pyproject.toml'"
@uv lock --locked
@echo "🚀 Linting code: Running pre-commit"
@echo "Linting code: Running pre-commit"
@uv run pre-commit run -a
@echo "🚀 Static type checking: Running mypy"
@echo "Static type checking: Running mypy"
@uv run mypy
@echo "🚀 Checking for obsolete dependencies: Running deptry"
@echo "Checking for obsolete dependencies: Running deptry"
@uv run deptry src

.PHONY: test
test: ## Test the code with pytest
@echo "🚀 Testing code: Running pytest"
@echo "Testing code: Running pytest"
@uv run python -m pytest --cov --cov-config=pyproject.toml --cov-report=xml

.PHONY: build
build: clean-build ## Build wheel file
@echo "🚀 Creating wheel file"
@echo "Creating wheel file"
@uvx --from build pyproject-build --installer uv

.PHONY: clean-build
clean-build: ## Clean build artifacts
@echo "🚀 Removing build artifacts"
@echo "Removing build artifacts"
@uv run python -c "import shutil; import os; shutil.rmtree('dist') if os.path.exists('dist') else None"

.PHONY: docker-clean
docker-clean: ## Clean up dangling Docker images and containers
@echo "Cleaning up dangling Docker images and containers"
@docker system prune -f


.PHONY: help
help:
@uv run python -c "import re; \
[[print(f'\033[36m{m[0]:<20}\033[0m {m[1]}') for m in re.findall(r'^([a-zA-Z_-]+):.*?## (.*)$$', open(makefile).read(), re.M)] for makefile in ('$(MAKEFILE_LIST)').strip().split()]"



.DEFAULT_GOAL := help
Loading
Loading