From bf1b6b90f774ac11ca4dedfeebe8d27220f218a1 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 29 Jan 2026 13:19:44 +0000 Subject: [PATCH 1/8] Initial plan From cb62fe5316f3566c8831adc7fc29288c515de025 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 29 Jan 2026 13:21:56 +0000 Subject: [PATCH 2/8] Add comprehensive Copilot instructions for the repository Co-authored-by: fredli74 <6632456+fredli74@users.noreply.github.com> --- .github/copilot-instructions.md | 167 ++++++++++++++++++ .../test/account/VwuUsLiYUkv-7YTPY5MWyw.info | Bin 0 -> 36 bytes 2 files changed, 167 insertions(+) create mode 100644 .github/copilot-instructions.md create mode 100644 server/test/account/VwuUsLiYUkv-7YTPY5MWyw.info diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..4d5b67f --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,167 @@ +# Hashbox Project - GitHub Copilot Instructions + +## Project Overview + +Hashbox is a cross-platform, general-purpose data block storage system with an efficient backup client (Hashback) featuring full data de-duplication. The project is a derivate of a proprietary backup system (BUP) invented around 2001. + +**Key features:** +- Variable-length block storage with optional compression +- Unique 128-bit hash-based block identification +- Cross-platform backup client with rolling checksum splitting +- Transactional dataset management with garbage collection +- Efficient incremental backups with local caching + +**Target audience:** System administrators and users needing reliable, de-duplicated backup solutions. + +**Project status:** Beta stage - not recommended for critical production use. + +## Tech Stack + +- **Language:** Go 1.22.5 +- **Core libraries:** + - `github.com/fredli74/bytearray` - Byte array handling + - `github.com/fredli74/cmdparser` - Command-line parsing + - `github.com/fredli74/lockfile` - File locking +- **Compression:** zlib (built-in) +- **Build system:** Shell scripts (`build_all.sh`) +- **Testing:** Go's built-in testing framework + +## Project Structure + +``` +hashbox/ +├── core/ - Core library with block handling, protocol, and client code +├── server/ - Server implementation with storage and account management +├── hashback/ - Backup client with store, restore, and diff operations +├── docs/ - Documentation and setup guides +└── build_all.sh - Multi-platform build script +``` + +## Coding Guidelines + +### General Principles +- Follow standard Go conventions and idioms +- Use `go fmt` for consistent formatting (integrated into `build_all.sh`) +- Maintain cross-platform compatibility (Windows, macOS, Linux, FreeBSD) +- Write single binaries with minimal runtime dependencies +- Prioritize performance and efficiency for large-scale data operations + +### Code Organization +- Keep platform-specific code in separate files: `*_windows.go`, `*_unix.go`, `*_mac.go` +- Use build tags when necessary for platform-specific functionality +- Minimize external dependencies (per project design goals) + +### Error Handling +- Use Go's standard error handling patterns +- Log errors with appropriate context and timestamps +- Include connection/client identifiers in server logs for debugging + +### Testing +- Write tests using Go's built-in testing framework +- Place test files alongside implementation: `*_test.go` +- Run tests with `go test -v` in respective directories +- Tests should pass on multiple platforms + +### Naming Conventions +- Follow Go naming conventions (exported vs unexported identifiers) +- Use descriptive variable names for clarity +- Package names should be lowercase, single-word when possible + +### Comments and Documentation +- Use the standard ASCII art header for copyright notices: + ```go + // ,+---+ + // +---+´| HASHBOX SOURCE + // | # | | Copyright 2015-2024 + // +---+´ + ``` +- Write godoc-style comments for exported functions and types +- Document complex algorithms and data structures +- Include usage examples in command-line tools + +### Performance Considerations +- Optimize for large file operations and network efficiency +- Use buffering appropriately for I/O operations +- Implement connection pooling where applicable +- Consider garbage collection impact in long-running processes + +### Security +- Handle authentication securely (MD5 challenge-response currently used) +- Validate user input in command-line interfaces +- Sanitize file paths to prevent directory traversal +- Be mindful of data integrity and checksum validation + +## Building and Testing + +### Building the project +```bash +# Format and build all binaries for multiple platforms +./build_all.sh +``` + +### Running tests +```bash +# Test core library +cd core && go test -v + +# Test server +cd server && go test -v + +# Format code +go fmt +``` + +### Running the server +```bash +# Add a user +./hashbox- adduser + +# Start the server +./hashbox- [-port=] [-data=] [-index=] [-loglevel=] + +# Run garbage collection +./hashbox- gc [-compact] [-compact-only] [-force] [-threshold=] +``` + +### Using the backup client +```bash +# Setup connection +./hashback -user= -password= -server=: -progress -saveoptions + +# Create a backup +./hashback -retaindays=7 -retainweeks=10 store ... + +# List datasets or files +./hashback list [|.] [""] + +# Restore files +./hashback restore |. [""...] +``` + +## Additional Resources + +- **README:** [README.md](/README.md) - Main project documentation +- **Documentation:** [docs/](/docs/) - Setup guides and specifications +- **License:** MIT License (see [LICENSE](/LICENSE)) +- **Repository:** https://github.com/fredli74/hashbox +- **Build Status:** https://semaphoreci.com/fredli74/hashbox + +## Development Roadmap + +When working on new features, be aware of planned improvements: +- Platform-specific file information storage +- Server admin interface (API) +- Scheduled/triggered garbage collection +- Online GC mark and sweep +- Server mirroring +- Client GUI +- Data encryption +- Quota calculations and restrictions + +## Notes for AI Assistance + +- Always test changes across platforms if modifying platform-specific code +- Verify backward compatibility with existing data stores when changing storage formats +- Consider the impact on existing backups when modifying the protocol +- Maintain the single-binary philosophy - avoid adding heavy dependencies +- Keep the permissive MIT License in mind when suggesting new dependencies diff --git a/server/test/account/VwuUsLiYUkv-7YTPY5MWyw.info b/server/test/account/VwuUsLiYUkv-7YTPY5MWyw.info new file mode 100644 index 0000000000000000000000000000000000000000..9979840c293f0743816a2bb5a92425f87930cc13 GIT binary patch literal 36 qcmZQzVBjf9EiO?=Ois=(%_~Vs+RC8zq01ok>8W{7-J32l0098v+YMF# literal 0 HcmV?d00001 From 8a620984356f7db6d9beaa32ad525bb7230d53ed Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 29 Jan 2026 13:22:11 +0000 Subject: [PATCH 3/8] Remove test artifacts and update .gitignore Co-authored-by: fredli74 <6632456+fredli74@users.noreply.github.com> --- .gitignore | 1 + server/test/account/VwuUsLiYUkv-7YTPY5MWyw.info | Bin 36 -> 0 bytes 2 files changed, 1 insertion(+) delete mode 100644 server/test/account/VwuUsLiYUkv-7YTPY5MWyw.info diff --git a/.gitignore b/.gitignore index e3b3376..0b68b48 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ desktop.ini hashback/hashback server/server +server/test/ diff --git a/server/test/account/VwuUsLiYUkv-7YTPY5MWyw.info b/server/test/account/VwuUsLiYUkv-7YTPY5MWyw.info deleted file mode 100644 index 9979840c293f0743816a2bb5a92425f87930cc13..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 36 qcmZQzVBjf9EiO?=Ois=(%_~Vs+RC8zq01ok>8W{7-J32l0098v+YMF# From e5788965effec330e32499a969684df69a994f62 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 29 Jan 2026 13:23:44 +0000 Subject: [PATCH 4/8] Fix Go version in Copilot instructions to reflect minimum version Co-authored-by: fredli74 <6632456+fredli74@users.noreply.github.com> --- .github/copilot-instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 4d5b67f..e52d89d 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -17,7 +17,7 @@ Hashbox is a cross-platform, general-purpose data block storage system with an e ## Tech Stack -- **Language:** Go 1.22.5 +- **Language:** Go (minimum 1.22.1, most modules use 1.22.5) - **Core libraries:** - `github.com/fredli74/bytearray` - Byte array handling - `github.com/fredli74/cmdparser` - Command-line parsing From 6617d759eb5456e5471c8dcf37ef3db0db973283 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 29 Jan 2026 13:26:01 +0000 Subject: [PATCH 5/8] Fix typo and clarify platform-specific file naming conventions Co-authored-by: fredli74 <6632456+fredli74@users.noreply.github.com> --- .github/copilot-instructions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index e52d89d..c16578c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -2,7 +2,7 @@ ## Project Overview -Hashbox is a cross-platform, general-purpose data block storage system with an efficient backup client (Hashback) featuring full data de-duplication. The project is a derivate of a proprietary backup system (BUP) invented around 2001. +Hashbox is a cross-platform, general-purpose data block storage system with an efficient backup client (Hashback) featuring full data de-duplication. The project is a derivative of a proprietary backup system (BUP) invented around 2001. **Key features:** - Variable-length block storage with optional compression @@ -47,7 +47,7 @@ hashbox/ - Prioritize performance and efficiency for large-scale data operations ### Code Organization -- Keep platform-specific code in separate files: `*_windows.go`, `*_unix.go`, `*_mac.go` +- Keep platform-specific code in separate files: `*_windows.go`, `*_unix.go`, and specific files like `hashback_mac.go` when additional platform differentiation is needed - Use build tags when necessary for platform-specific functionality - Minimize external dependencies (per project design goals) From 14df6f40b55be4149a46ec2486c19dbaaffc28b2 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 29 Jan 2026 13:35:19 +0000 Subject: [PATCH 6/8] Add project philosophy and coding principles to Copilot instructions Co-authored-by: fredli74 <6632456+fredli74@users.noreply.github.com> --- .github/copilot-instructions.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index c16578c..09fc4c8 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -46,6 +46,16 @@ hashbox/ - Write single binaries with minimal runtime dependencies - Prioritize performance and efficiency for large-scale data operations +### Project Philosophy +1. **Follow first principles and design by contract** - Design interfaces with clear contracts and expectations +2. **Use asserts to catch incorrect usage** - Do not guard-rail invalid input outside the contract +3. **Panic on unexpected errors** - Only recover when errors are expected (rare) +4. **Optimize and reduce redundancies** - Split into functions if the same work repeats +5. **Test, iterate, and test again** - Continuously validate changes through testing +6. **Second-pass verification** - Ensure all code is needed; delete if possible +7. **When refactoring, compare logic** - Ensure behavior is unchanged; if changed, explain why +8. **Avoid aliases/duplicate helpers** - Prefer a single canonical API. Do not keep backward-compat shims unless strictly required, and remove them promptly after migration + ### Code Organization - Keep platform-specific code in separate files: `*_windows.go`, `*_unix.go`, and specific files like `hashback_mac.go` when additional platform differentiation is needed - Use build tags when necessary for platform-specific functionality From 19e840409b190f2edd4531cef324a61a126667b8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 29 Jan 2026 13:44:05 +0000 Subject: [PATCH 7/8] Update testing documentation with module tests and E2E scripts Co-authored-by: fredli74 <6632456+fredli74@users.noreply.github.com> --- .github/copilot-instructions.md | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 09fc4c8..92b411e 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -34,6 +34,7 @@ hashbox/ ├── server/ - Server implementation with storage and account management ├── hashback/ - Backup client with store, restore, and diff operations ├── docs/ - Documentation and setup guides +├── scripts/ - Test scripts including E2E smoke tests └── build_all.sh - Multi-platform build script ``` @@ -111,14 +112,28 @@ hashbox/ ### Running tests ```bash -# Test core library -cd core && go test -v +# Test per module (default approach) +cd core && go test ./... +cd server && go test ./... +cd hashback && go test ./... -# Test server +# Or test a specific package cd server && go test -v +``` + +### End-to-End Smoke Tests +```bash +# Run E2E smoke test from repository root (local mode) +./scripts/e2e_hashbox.sh + +# Run E2E smoke test in Docker mode +E2E_DOCKER=1 ./scripts/e2e_hashbox.sh -# Format code -go fmt +# Environment variable overrides available: +# - E2E_SOURCE_DIR: Source directory for backup tests +# - E2E_DATASET: Dataset name for tests +# - E2E_TMP_ROOT: Temporary directory root +# See script header for full documentation ``` ### Running the server From 55b88dfd659f2299f2a07ca0f6f450a88fe86f4c Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 31 Jan 2026 02:45:38 +0000 Subject: [PATCH 8/8] Fix project structure to reflect current repository organization Co-authored-by: fredli74 <6632456+fredli74@users.noreply.github.com> --- .github/copilot-instructions.md | 35 +++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 92b411e..f6542a4 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -17,25 +17,33 @@ Hashbox is a cross-platform, general-purpose data block storage system with an e ## Tech Stack -- **Language:** Go (minimum 1.22.1, most modules use 1.22.5) +- **Language:** Go 1.25.6 - **Core libraries:** - `github.com/fredli74/bytearray` - Byte array handling - `github.com/fredli74/cmdparser` - Command-line parsing - `github.com/fredli74/lockfile` - File locking - **Compression:** zlib (built-in) -- **Build system:** Shell scripts (`build_all.sh`) +- **Build system:** Shell scripts in `scripts/` directory - **Testing:** Go's built-in testing framework +- **Containerization:** Docker with docker-compose support ## Project Structure ``` hashbox/ -├── core/ - Core library with block handling, protocol, and client code -├── server/ - Server implementation with storage and account management -├── hashback/ - Backup client with store, restore, and diff operations -├── docs/ - Documentation and setup guides -├── scripts/ - Test scripts including E2E smoke tests -└── build_all.sh - Multi-platform build script +├── pkg/ - Shared packages +│ ├── core/ - Core library with block handling, protocol, and client code +│ ├── accountdb/ - Account database management +│ ├── storagedb/ - Storage database management +│ └── lockablefile/ - Lockable file utilities +├── server/ - Server implementation +├── hashback/ - Backup client with store, restore, and diff operations +├── util/ - Utility commands (hashbox-util) +├── scripts/ - Build and test scripts (build_all.sh, e2e_hashbox.sh, docker_*.sh) +├── docs/ - Documentation and setup guides +├── Dockerfile - Container image definition +├── docker-compose.yml - Docker Compose configuration +└── .golangci.yml - Linter configuration ``` ## Coding Guidelines @@ -107,15 +115,22 @@ hashbox/ ### Building the project ```bash # Format and build all binaries for multiple platforms -./build_all.sh +./scripts/build_all.sh + +# Build Docker image +./scripts/docker_build.sh ``` ### Running tests ```bash # Test per module (default approach) -cd core && go test ./... +cd pkg/core && go test ./... +cd pkg/accountdb && go test ./... +cd pkg/storagedb && go test ./... +cd pkg/lockablefile && go test ./... cd server && go test ./... cd hashback && go test ./... +cd util && go test ./... # Or test a specific package cd server && go test -v