Refactor and optimize archive and manifest handling with benchmarks

.gitignore

@@ -1,7 +1,44 @@
+# If you prefer the allow list template instead of the deny list, see community template:
+# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
+#
+# Binaries for programs and plugins
+*.exe
+*.exe~
+*.dll
+*.so
+*.dylib
+
+# Test binary, built with `go test -c`
+*.test
+
+# Code coverage profiles and other test artifacts
+*.out
+coverage.*
+*.coverprofile
+profile.cov
+
+# Dependency directories (remove the comment below to include it)
+# vendor/
+
+# Go workspace file
+go.work
+go.work.sum
+
+# env file
+.env
+
+# Editor/IDE
+# .idea/
+# .vscode/
+
+
 Editing/
 debugging/
 manifests/
+unusual/
 norm/
 *.exe
 *.json
 output/
+/extracted/
+bin/

Makefile

@@ -0,0 +1,56 @@
+.PHONY: help build test bench clean install fmt lint check
+
+# Default target - show help
+.DEFAULT_GOAL := help
+
+# Show available targets
+help:
+	@echo "evrFileTools - EVR package/manifest tool"
+	@echo ""
+	@echo "Usage: make [target]"
+	@echo ""
+	@echo "Targets:"
+	@echo "  build         Build the CLI tool to bin/evrtools"
+	@echo "  test          Run all tests"
+	@echo "  bench         Run benchmarks"
+	@echo "  bench-compare Run benchmarks with multiple iterations"
+	@echo "  clean         Remove build artifacts"
+	@echo "  install       Install CLI tool via go install"
+	@echo "  fmt           Format code"
+	@echo "  lint          Run go vet"
+	@echo "  check         Run fmt, lint, and test"
+
+# Build the CLI tool
+build:
+	go build -o bin/evrtools ./cmd/evrtools
+
+# Run all tests
+test:
+	go test -v ./pkg/...
+
+# Run benchmarks
+bench:
+	go test -bench=. -benchmem -benchtime=1s ./pkg/...
+
+# Run benchmarks with comparison
+bench-compare:
+	go test -bench=. -benchmem -count=5 ./pkg/...
+
+# Clean build artifacts
+clean:
+	rm -rf bin/
+
+# Install the CLI tool
+install:
+	go install ./cmd/evrtools
+
+# Format code
+fmt:
+	go fmt ./...
+
+# Lint code
+lint:
+	go vet ./...
+
+# Check for common issues
+check: fmt lint test

README.md

@@ -1,25 +1,155 @@
-thank you Exhibitmark for doing the hard work and making [carnation](https://github.com/Exhibitmark/carnation), saved me a lot of headache reversing the manifest format :)
+# evrFileTools
 
-tool i ~~not so~~ quickly threw together to modify any file(s) in an EVR manifest/package combo.
-Barely in a working state, please cut me some slack while i clean this up
+A Go library and CLI tool for working with EVR (Echo VR) package and manifest files.
 
-extracting files example:
+> Thanks to [Exhibitmark](https://github.com/Exhibitmark) for [carnation](https://github.com/Exhibitmark/carnation) which helped with reversing the manifest format!
+
+## Features
+
+- Extract files from EVR packages
+- Build new packages from extracted files
+- Read and write EVR manifest files
+- ZSTD compression/decompression with optimized context reuse
+
+## Installation
+
+```bash
+go install github.com/EchoTools/evrFileTools/cmd/evrtools@latest
+```
+
+Or build from source:
+
+```bash
+git clone https://github.com/EchoTools/evrFileTools.git
+cd evrFileTools
+make build
+```
+
+## Usage
+
+### Extract files from a package
+
+```bash
+evrtools -mode extract \
+    -data ./ready-at-dawn-echo-arena/_data/5932408047/rad15/win10 \
+    -package 48037dc70b0ecab2 \
+    -output ./extracted
 ```
-evrFileTools -mode extract -packageName 48037dc70b0ecab2 -dataDir ./ready-at-dawn-echo-arena/_data/5932408047/rad15/win10 -outputDir ./output/
+
+This extracts all files from the package. Output structure:
+- `./output/<typeSymbol>/<fileSymbol>`
+
+With `-preserve-groups`, frames are preserved:
+- `./output/<frameIndex>/<typeSymbol>/<fileSymbol>`
+
+### Build a package from files
+
+```bash
+evrtools -mode build \
+    -input ./files \
+    -output ./output \
+    -package mypackage
 ```
-this will extract and write out every file contained in the package to outputFolder.
-the names of the subfolders created in outputFolder are the filetype symbols, the files contained within are named with their respective symbols.
 
-If the `-outputPreserveGroups` flag is provided, there will be folders created to seperate each frame. This is currently the directory structure that `-mode build` expects.
+Expected input structure: `./input/<frameIndex>/<typeSymbol>/<fileSymbol>`
+
+### CLI Options
+
+| Flag | Description |
+|------|-------------|
+| `-mode` | Operation mode: `extract` or `build` |
+| `-data` | Path to _data directory containing manifests/packages |
+| `-package` | Package name (e.g., `48037dc70b0ecab2`) |
+| `-input` | Input directory for build mode |
+| `-output` | Output directory |
+| `-preserve-groups` | Preserve frame grouping in extract output |
+| `-force` | Allow non-empty output directory |
+
+## Library Usage
 
+```go
+package main
 
-replacing files example:
+import (
+    "log"
+    "github.com/EchoTools/evrFileTools/pkg/manifest"
+)
+
+func main() {
+    // Read a manifest
+    m, err := manifest.ReadFile("/path/to/manifests/packagename")
+    if err != nil {
+        log.Fatal(err)
+    }
+
+    log.Printf("Manifest: %d files in %d packages", m.FileCount(), m.PackageCount())
+
+    // Open the package files
+    pkg, err := manifest.OpenPackage(m, "/path/to/packages/packagename")
+    if err != nil {
+        log.Fatal(err)
+    }
+    defer pkg.Close()
+
+    // Extract all files
+    if err := pkg.Extract("./output"); err != nil {
+        log.Fatal(err)
+    }
+}
 ```
-echoFileTools -mode replace -outputDir ./output/ -packageName 48037dc70b0ecab2 -dataDir ./ready-at-dawn-echo-arena/_data/5932408047/rad15/win10 -inputDir ./input/
+
+## Project Structure
+
+```
+evrFileTools/
+├── cmd/
+│   └── evrtools/           # CLI application
+├── pkg/
+│   ├── archive/            # ZSTD archive format
+│   │   ├── header.go       # Archive header (24 bytes)
+│   │   ├── reader.go       # Streaming decompression
+│   │   └── writer.go       # Streaming compression
+│   └── manifest/           # EVR manifest/package handling
+│       ├── manifest.go     # Manifest types and binary encoding
+│       ├── package.go      # Multi-part package extraction
+│       ├── builder.go      # Package building from files
+│       └── scanner.go      # Input directory scanning
+├── Makefile
+└── go.mod
+```
+
+## Development
+
+```bash
+# Build
+make build
+
+# Run tests
+make test
+
+# Run benchmarks
+make bench
+
+# Format and lint
+make check
 ```
-Directory structure of inputDir while using `-mode replace` should be `./inputFolder/0/...`, where ... is the structure of `-mode extract` *without* the `-outputPreserveGroups` flag.
 
-e.g. if replacing the Echo VR logo DDS, the stucture would be as follows: `./input/0/-4707359568332879775/-3482028914369150717`
+## Performance
+
+The library uses several optimizations:
+
+- **Direct binary encoding** instead of reflection-based `binary.Read/Write`
+- **Pre-allocated buffers** for zero-allocation encoding paths
+- **ZSTD context reuse** for ~4x faster decompression with zero allocations
+- **Frame index maps** for O(1) file lookups during extraction
+- **Directory caching** to minimize syscalls
+
+Run benchmarks to see current performance:
+
+```bash
+go test -bench=. -benchmem ./pkg/...
+```
 
+## License
 
-if a file with the same filetype symbol & filename symbol exists in the manifest, it will edit the manifest & package file to match, and write out the contents of both to outputDir.
+MIT License - see LICENSE file

_data

@@ -0,0 +1 @@
+/mnt/c/OculusLibrary/Software/ready-at-dawn-echo-arena/_data/5932408047/rad15/win10/