diff --git a/AGENTS.md b/AGENTS.md index b3afd40e4..a6afd7f57 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -100,7 +100,9 @@ make clean-all # Also remove downloaded tools ```bash go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.0 go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0 -tools/protoc/osxm1/protoc *.proto --go-grpc_out=. --go_out=. + +# Generate (adjust protoc path for your platform) +vermeer/tools/protoc/linux64/protoc vermeer/apps/protos/*.proto --go-grpc_out=vermeer/apps/protos/. --go_out=vermeer/apps/protos/. # please note remove license header if any ``` ## Architecture @@ -174,6 +176,7 @@ tools/protoc/osxm1/protoc *.proto --go-grpc_out=. --go_out=. - Master scheduling: `vermeer/apps/master/tasks/tasks.go` - Worker management: `vermeer/apps/master/workers/workers.go` - HTTP endpoints: `vermeer/apps/master/services/http_master.go` +- Scheduler: `vermeer/apps/master/bl/scheduler_bl.go` ## Integration with HugeGraph @@ -213,7 +216,7 @@ tools/protoc/osxm1/protoc *.proto --go-grpc_out=. --go_out=. - Unit tests run in isolation without external dependencies **Vermeer:** -- Test scripts in `vermeer/test/` +- Test scripts in `vermeer/test/`,with `vermeer_test.go` and `vermeer_test.sh` - Configuration files in `vermeer/config/` (master.ini, worker.ini templates) ## CI/CD diff --git a/README.md b/README.md index aa06ce37e..2f86b4189 100644 --- a/README.md +++ b/README.md @@ -60,12 +60,13 @@ graph TB UI[Web UI Dashboard] end - subgraph Master["Master Node :6688"] - HTTP[HTTP Server] + subgraph Master["Master Node"] + HTTP[HTTP Server :6688] GRPC_M[gRPC Server :6689] GM[Graph Manager] TM[Task Manager] WM[Worker Manager] + SCH[Scheduler] end subgraph Workers["Worker Nodes"] @@ -145,6 +146,10 @@ For quick start and single-machine deployments, we recommend **Vermeer**: # Pull the image docker pull hugegraph/vermeer:latest +# Change config path in docker-compose.yml +volumes: + - ~/:/go/bin/config # Change here to your actual config path, e.g., vermeer/config + # Run with docker-compose docker-compose up -d ``` diff --git a/vermeer/AGENTS.md b/vermeer/AGENTS.md index 2fb2df04c..3a6f715cf 100644 --- a/vermeer/AGENTS.md +++ b/vermeer/AGENTS.md @@ -61,7 +61,9 @@ go test -tags=vermeer_test -v -mode=scheduler ```bash go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.0 go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0 -tools/protoc/osxm1/protoc *.proto --go-grpc_out=. --go_out=. + +# Generate (adjust protoc path for your platform) +vermeer/tools/protoc/linux64/protoc vermeer/apps/protos/*.proto --go-grpc_out=vermeer/apps/protos/. --go_out=vermeer/apps/protos/. # please note remove license header if any ``` ## Architecture @@ -81,6 +83,7 @@ vermeer/ │ │ ├── services/ # HTTP handlers │ │ ├── workers/ # Worker management (WorkerManager, WorkerClient) │ │ ├── tasks/ # Task scheduling +│ │ ├── schedules/ # Task scheduling strategies │ │ └── graphs/ # Graph metadata management │ ├── worker/ # Worker service entry │ ├── compute/ # Worker-side compute logic @@ -146,6 +149,7 @@ Algorithms implement the interface defined in `apps/compute/api.go`. Each algori - Master scheduling: `apps/master/tasks/tasks.go` - Worker management: `apps/master/workers/workers.go` - HTTP endpoints: `apps/master/services/http_master.go` +- Scheduler: `vermeer/apps/master/bl/scheduler_bl.go` ## Development Workflow @@ -167,7 +171,8 @@ Algorithms implement the interface defined in `apps/compute/api.go`. Each algori 1. Edit `.proto` files in `apps/protos/` 2. Regenerate Go code using protoc (adjust path for platform): ```bash - tools/protoc/osxm1/protoc apps/protos/*.proto --go-grpc_out=. --go_out=. + # Generate (adjust protoc path for your platform) + vermeer/tools/protoc/linux64/protoc vermeer/apps/protos/*.proto --go-grpc_out=vermeer/apps/protos/. --go_out=vermeer/apps/protos/. # please note remove license header if any ``` ## Configuration diff --git a/vermeer/README.md b/vermeer/README.md index a989b4ad2..c8ad8f606 100644 --- a/vermeer/README.md +++ b/vermeer/README.md @@ -78,6 +78,7 @@ vermeer/ │ ├── master/ # Master service │ │ ├── services/ # HTTP handlers │ │ ├── workers/ # Worker management +| | ├── schedules/ # Task scheduling strategies │ │ └── tasks/ # Task scheduling │ ├── compute/ # Worker-side compute logic │ ├── graphio/ # Graph I/O (HugeGraph, CSV, HDFS) @@ -117,7 +118,12 @@ docker run -v ~/vermeer-config:/go/bin/config hugegraph/vermeer --env=worker #### Docker Compose -Update `master_peer` in `~/worker.ini` to `172.20.0.10:6689`, then: +Update `master_peer` in `~/worker.ini` to `172.20.0.10:6689`, and edit ``docker-compose.yml`` to mount your config directory: + +```yaml + volumes: + - ~/:/go/bin/config # Change here to your actual config path +``` ```bash docker-compose up -d @@ -479,7 +485,7 @@ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.0 go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0 # Generate (adjust protoc path for your platform) -tools/protoc/osxm1/protoc *.proto --go-grpc_out=. --go_out=. +vermeer/tools/protoc/linux64/protoc vermeer/apps/protos/*.proto --go-grpc_out=vermeer/apps/protos/. --go_out=vermeer/apps/protos/. # please note remove license header if any ``` ## Performance Tuning