diff --git a/auto_dev/commands/augment.py b/auto_dev/commands/augment.py index 96551c83..e95e2187 100644 --- a/auto_dev/commands/augment.py +++ b/auto_dev/commands/augment.py @@ -405,7 +405,6 @@ def customs(ctx, component_type, auto_confirm, use_daos): write_to_file(handler_path, handler_code, FileType.PYTHON) logger.info(f"Handler code written to {handler_path}") - scaffolder.create_dialogues() if use_daos: scaffolder.create_exceptions() logger.info("OpenAPI3 scaffolding completed successfully.") diff --git a/auto_dev/data/openapi/openapi_specification.yaml b/auto_dev/data/openapi/openapi_specification.yaml index a2339963..fc05a2a1 100644 --- a/auto_dev/data/openapi/openapi_specification.yaml +++ b/auto_dev/data/openapi/openapi_specification.yaml @@ -4,7 +4,7 @@ info: version: 1.0.0 description: A simple API for testing the OpenAPI Handler Generator paths: - /users: + /api/users: get: summary: List users responses: @@ -31,7 +31,7 @@ paths: application/json: schema: $ref: '#/components/schemas/User' - /users/{userId}: + /api/users/{userId}: get: summary: Get a user parameters: diff --git a/docs/dao.md b/docs/dao.md index dadafd4e..edd88f82 100644 --- a/docs/dao.md +++ b/docs/dao.md @@ -1,204 +1,232 @@ -# Scaffolding a new Data Access Object (DAO) +# Data Access Object (DAO) Scaffolding -The tools within the `dao` subcommand are used to scaffold a new customs component Data Access Object (DAO) based on an OpenAPI 3 specification. This process automates the creation of Data Access Object classes, dummy data, and test scripts. +This guide explains how to scaffold Data Access Objects based on an OpenAPI 3 specification, automating the creation of database interaction classes. ## Prerequisites -1. An OpenAPI 3 specification file with components/schema models defined. -2. A `component.yaml` file in the current directory that references the OpenAPI specification using the `api_spec` field. - -## Steps to Create a Data Access Object - -1. Ensure you have the OpenAPI 3 specification file. You can view its contents using: - - ``` - cat auto_dev/data/openapi/openapi_specification.yaml - ``` - - Output: - - ```yaml - openapi: 3.0.0 - info: - title: Test API - version: 1.0.0 - description: A simple API for testing the OpenAPI Handler Generator - paths: - /users: - get: - summary: List users - responses: - '200': - description: Successful response - content: - application/json: - schema: - type: array - items: - $ref: '#/components/schemas/User' - post: - summary: Create a user - requestBody: - required: true - content: - application/json: - schema: +- An OpenAPI 3 specification file with defined components/schema models +- A `component.yaml` file referencing the OpenAPI specification via the `api_spec` field + +## Quick Start + +### 1. Define an OpenAPI 3 specification + +Your specification should include schema definitions: + +``` +cat auto_dev/data/openapi/openapi_specification.yaml +``` + +Output: + +```yaml +openapi: 3.0.0 +info: + title: Test API + version: 1.0.0 + description: A simple API for testing the OpenAPI Handler Generator +paths: + /api/users: + get: + summary: List users + responses: + '200': + description: Successful response + content: + application/json: + schema: + type: array + items: $ref: '#/components/schemas/User' - responses: - '201': - description: Created - content: - application/json: - schema: - $ref: '#/components/schemas/User' - /users/{userId}: - get: - summary: Get a user - parameters: - - name: userId - in: path - required: true + post: + summary: Create a user + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/User' + responses: + '201': + description: Created + content: + application/json: + schema: + $ref: '#/components/schemas/User' + /api/users/{userId}: + get: + summary: Get a user + parameters: + - name: userId + in: path + required: true + schema: + type: integer + responses: + '200': + description: Successful response + content: + application/json: schema: - type: integer - responses: - '200': - description: Successful response - content: - application/json: - schema: - $ref: '#/components/schemas/User' - components: - schemas: - User: - type: object - required: - - id - - name - properties: - id: - type: integer - name: - type: string - email: - type: string - ``` - -2. If not already done, scaffold a repo and a customs component. - - Initialize aea: - ```bash - aea init --remote --author xiuxiuxar --ipfs --reset - ``` - - Create a new repo: - ```bash - adev repo scaffold -t autonomy new_station - ``` - - ```bash - cd new_station - ``` - - Scaffold a customs component: - ```bash - aea scaffold -tlr custom simple_dao - ``` - -3. Create or update the `component.yaml` file to reference the OpenAPI specification: - - ```bash - cp ../auto_dev/data/openapi/openapi_specification.yaml packages/xiuxiuxar/customs/simple_dao/ - ``` - - ```bash - yq e '.api_spec = "openapi_specification.yaml"' -i packages/xiuxiuxar/customs/simple_dao/component.yaml - ``` - - ```bash - cat packages/xiuxiuxar/customs/simple_dao/component.yaml - ``` - - Output: - - ```yaml - name: simple_dao - author: xiuxiuxar - version: 0.1.0 - type: custom - description: The custom component package. - license: Apache-2.0 - aea_version: '>=1.0.0, <2.0.0' - fingerprint: - __init__.py: bafybeigjt2jaxismyp3hspvqefjrcp33mmen3pnmnkm4rhesh74io3vikm - fingerprint_ignore_patterns: [] - dependencies: {} - api_spec: openapi_specification.yaml - ``` - -4. Run the DAO scaffolding command from the customs component directory: - - ```bash - cd packages/xiuxiuxar/customs/simple_dao - ``` + $ref: '#/components/schemas/User' +components: + schemas: + User: + type: object + required: + - id + - name + properties: + id: + type: integer + name: + type: string + email: + type: string +``` + +### 2. Set Up Your Project Structure + +!!! note "Optional Step" + Skip this if you already have a project structure set up. + +Initialize aea: +```bash +aea init --remote --author xiuxiuxar --ipfs --reset +``` + +Create a new repo: +```bash +adev repo scaffold -t autonomy new_station +``` + +```bash +cd new_station +``` + +Scaffold a customs component: +```bash +aea scaffold -tlr custom simple_dao +``` + +### 3. Configure Your Component + +```bash +cp ../auto_dev/data/openapi/openapi_specification.yaml packages/xiuxiuxar/customs/simple_dao/ +``` + +```bash +yq e '.api_spec = "openapi_specification.yaml"' -i packages/xiuxiuxar/customs/simple_dao/component.yaml +``` +```bash +cat packages/xiuxiuxar/customs/simple_dao/component.yaml +``` + +Output: + +```yaml +name: simple_dao +author: xiuxiuxar +version: 0.1.0 +type: custom +description: The custom component package. +license: Apache-2.0 +aea_version: '>=1.0.0, <2.0.0' +fingerprint: + __init__.py: bafybeigjt2jaxismyp3hspvqefjrcp33mmen3pnmnkm4rhesh74io3vikm +fingerprint_ignore_patterns: [] +dependencies: {} +api_spec: openapi_specification.yaml +``` + +### 4. Generate DAOs + +```bash +cd packages/xiuxiuxar/customs/simple_dao +``` + +!!! tip We automatically confirm all actions, though you can omit the `--auto-confirm` flag to see the actions that will be taken. - ```bash - adev scaffold dao --auto-confirm - ``` +```bash +adev scaffold dao --auto-confirm +``` -## Generated Files +## Generated Structure -The scaffolding process generates the following files in your customs component: +The scaffolding process creates the following files: ``` daos/ ├── __init__.py ├── base_dao.py # Base Data Access Object class -├── _dao.py # Model-specific Data Access Object -├── _dao.py # Model-specific Data Access Object -├── aggregated_data.json # Sample data for testing -└── test_dao.py # Test script +├── _dao.py # Model-specific Data Access Object +├── _dao.py # Model-specific Data Access Object +├── aggregated_data.json # Sample test data +└── test_dao.py # Test script ``` -## Implementation Details +## Understanding the Generated Code -1. Base Data Access Object Class -2. Generating Data Access Object classes for each model -3. Test Script Generation +### Base DAO -## How It Works +The `base_dao.py` file contains: -The scaffolding process involves several steps: +- Common CRUD operations (Create, Read, Update, Delete) +- Transaction handling +- Error management -1. Loading and validating the OpenAPI specification (checking for required fields, etc.) -2. Generating DAO classes for each model -3. Creating dummy data for testing -4. Generating a test script +### Model-Specific DAOs -For more details on the implementation, refer to: -`auto_dev/dao/scaffolder.py` +Each model gets its own DAO class with: + +- Model-specific validation +- Custom query methods +- Type hints for your schema + +### Test Data + +The `aggregated_data.json` file contains: + +- Sample data for each model +- Valid test cases for CRUD operations ## Customization -The generated DAO classes use Jinja2 templates for customization. If you need to modify the structure of the generated classes, you can update the templates located in the `auto_dev/data/templates/dao` directory. +!!! tip "Template Customization" + The generated DAOs use Jinja2 templates. To customize: + + 1. Locate templates in `auto_dev/data/templates/dao` + 2. Modify them to suit your requirements + 3. Re-run the scaffolding command + +## Best Practices -## Error Handling +### Error Handling The scaffolding process validates: -- `component.yaml` exists and has `api_spec` field -- OpenAPI spec file exits and is valid YAML -- OpenAPI spec contains components/schemas -- OpenAPI spec follows OpenAPI 3.0 format -Detailed error messages are logged for troubleshooting. +- Component configuration +- OpenAPI specification format +- Schema definitions + +!!! warning "Schema Requirements" + Ensure your OpenAPI spec includes proper schema definitions with required fields and property types. -## Next Steps +### Keeping DAOs in Sync -After scaffolding your Data Access Objects: +!!! success "Regenerate After Changes" + Always regenerate your DAOs after modifying your OpenAPI specification to maintain consistency. + +## Testing Your DAOs + +Run the generated test script to verify functionality: + +``` +python -m packages.your_author.customs.simple_dao.daos.test_dao +``` -1. Review the generated Data Access Object classes in the `daos/` directory. -2. Customize the generated classes as needed. -3. Run the `test_dao.py` script to ensure the basic functionality of your Data Access Objects. +## Integration with Handlers -Remember to regenerate the Data Access Objects if you make changes to your OpenAPI specification to keep them in sync. \ No newline at end of file +DAOs work seamlessly with OpenAPI handlers. See the [OpenAPI Handler Integration](openapi.md) guide for details on connecting your DAOs to API endpoints. \ No newline at end of file diff --git a/docs/openapi.md b/docs/openapi.md index 9ae54414..82c23068 100644 --- a/docs/openapi.md +++ b/docs/openapi.md @@ -1,92 +1,274 @@ -# Scaffolding a new API Handler agent. +# OpenAPI Handler Integration -The tools within the `handler` subcommand are used to scaffold a new agent. +This guide explains how to augment components with handler methods based on an OpenAPI 3. -We start with a simple openapi; +## Prerequisites -```bash +- An OpenAPI 3 specification file with defined paths and operationIds +- A `component.yaml` file referencing the OpenAPI specification via the `api_spec` field. +- For DAO integration: Data Access Objects for each schema in the specification. See [Data Access Object](dao.md). + +## Basic Handler Augmentation + +### 1. Define an OpenAPI 3 Specification. + +``` cat auto_dev/data/openapi/openapi_specification.yaml ``` -We now scaffold the agent and cd in. +Output: + +```yaml +openapi: 3.0.0 +info: + title: Test API + version: 1.0.0 + description: A simple API for testing the OpenAPI Handler Generator +paths: + /api/users: + get: + summary: List users + responses: + '200': + description: Successful response + content: + application/json: + schema: + type: array + items: + $ref: '#/components/schemas/User' + post: + summary: Create a user + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/User' + responses: + '201': + description: Created + content: + application/json: + schema: + $ref: '#/components/schemas/User' + /api/users/{userId}: + get: + summary: Get a user + parameters: + - name: userId + in: path + required: true + schema: + type: integer + responses: + '200': + description: Successful response + content: + application/json: + schema: + $ref: '#/components/schemas/User' +components: + schemas: + User: + type: object + required: + - id + - name + properties: + id: + type: integer + name: + type: string + email: + type: string +``` + +### 2. Set Up Your Project Structure + +!!! note "Optional Step" + Skip this if you already have a project structure set up. + +Initialise aea: + +```bash +aea init --remote --author xiuxiuxar --ipfs --reset +``` + +Create a new repo: ```bash -aea create new_agent && cd new_agent +adev repo scaffold --no-install -t autonomy new_station ``` -This creates a new agent without any real skills. +```bash +cd new_station +``` -Once we have a new agent, we can now use the cli to scaffold the skill using the CORE autonomy libraries and the OpenAPI specification. +Create a new agent based on the frontend_agent template: -This reduces the amount of code we need to write to get a skill up and means that we have no need to write any code to re-implement the wheel. +```bash +adev create xiuxiuxar/agent -t eightballer/frontend_agent --no-clean-up --force +``` -## Scaffolding a new skill +```bash +cd agent +``` -Use the --new-skill flag to scaffold a new skill. +Eject a custom component to use as a base handler template: ```bash -adev scaffold handler ../auto_dev/data/openapi/openapi_specification.yaml --output my_api_skill --new-skill +adev eject custom eightballer/simple_html xiuxiuxar/new_handler ``` -The skill will be created in the skills directory. The user will be prompted whether to rename MyModel.py to strategy.py, and whether to remove the dialogues.py file. THe scaffolding step will also install the http protocol, and fingerprint the skill. At the completion, the user can now run the agent. +Push to the local packages directory: ```bash -aea run +adev publish xiuxiuxar/agent --force ``` -## Augmenting with an OpenAPI Handler +```bash +cd .. +``` -The tools within the `openapi` subcommand are used to augment a customs component with a new handler based on an OpenAPI 3 specification. This process automates the creation of endpoints methods. -## Prerequisites +### 3. Configure Your Component -1. An OpenAPI 3 specification file with paths, operationIds, and if augmenting with DAOs, schemas defined. -2. A `component.yaml` file in the current directory that references the OpenAPI specification using the `api_spec` field. -3. If augmenting with DAOs, DAOs for each schema in the OpenAPI specification (see dao docs for how to scaffold these). +Add your OpenAPI specification to the component: -## Steps to Augment a Handler +```bash +cp ../auto_dev/data/openapi/openapi_specification.yaml packages/xiuxiuxar/customs/new_handler/openapi3_spec.yaml +``` -1. Ensure you have the OpenAPI 3 specification file. You can view its contents using: +```bash +yq e '.api_spec = "openapi3_spec.yaml"' -i packages/xiuxiuxar/customs/new_handler/component.yaml +``` ```bash -cat auto_dev/data/openapi/openapi_specification.yaml +cat packages/xiuxiuxar/customs/new_handler/component.yaml ``` -2. Create or update the `component.yaml` file to reference the OpenAPI specification using the `api_spec` field. +Output: ```yaml -api_spec: +version: 0.1.0 +type: custom +description: Simple HTML frontend to act as a template for the frontend ABCI loader. +license: Apache-2.0 +aea_version: '>=1.0.0, <2.0.0' +fingerprint: + __init__.py: bafybeidwwtpfuhelmwdkgh2ayk6hls62xpxns33f27qsryzd4vh2kw6hxq + behaviours.py: bafybeiadbbujl23thcggr55up7t7jo5nxjk5qty7uqf75xgw57nzrq3dwy + build/index.html: bafybeigyjfv75r7fnd7mw63vbidusohzopzyz47c5nmc3tiak5yh22gbfi + handlers.py: bafybeibnco7jqjsfwamcek64cvltcr3wvvw73bs74wzbbhwqggy4p4hlsi + openapi3_spec.yaml: bafybeibdvlfbr4ghin75dyqjyd3zbusq45uekmuoftmj74k4cvnq2rapci + tests/__init__.py: bafybeiayjxaqrjpu7poz56eozqbi4ln7uene4r27bnq5qeely3ambcmggm + tests/test_simple_html.py: bafybeiaqu3mw6kq5yq7blk5vjrd5mtw5ej55vs32ftxdf225k3semjyn4u +fingerprint_ignore_patterns: [] +dependencies: {} +api_spec: openapi3_spec.yaml +frontend_dir: build +behaviours: +- class_name: LogReadingBehaviour + args: {} +handlers: +- class_name: HttpHandler + args: {} +``` + +### 4. Run the Augmentation Command + +```bash +cd packages/xiuxiuxar/customs/new_handler +``` + +We automatically confirm all actions, though you can omit the `--auto-confirm` flag to see the actions that will be taken. + +```bash +adev augment customs openapi3 --auto-confirm +``` + +### 5. Update Configuration + +```bash +cd ../../../.. +``` + +We ensure the component yaml references the handler class: + +```bash +yq e '.handlers[].class_name = "ApiHttpHandler"' -i packages/xiuxiuxar/customs/new_handler/component.yaml ``` -3. Run the Handler augmenting command, optionally with the `--use-daos` flag if you are augmenting with DAOs: +Modify the agent to load the new custom handler method: + +```bash +yq e '(select(.public_id == "eightballer/trader_abci:0.1.0") | .models.params.args.user_interface.custom_component) = "xiuxiuxar/new_handler"' -i packages/xiuxiuxar/agents/agent/aea-config.yaml +``` ```bash -adev augment customs openapi3 --use-daos +autonomy packages lock +``` + +### 6. Run Your Agent + +```shell +adev run dev xiuxiuxar/agent --force +``` + +## Advanced: Integration with Data Access Objects (DAOs) + +### 1. Complete Basic Setup + +Follow steps 1-3 from the basic handler augmentation. + +### 2. Scaffold DAOs + +!!! tip + We automatically confirm all actions, though you can omit the `--auto-confirm` flag to see the actions that will be taken. + + +```shell +adev scaffold dao --auto-confirm +``` + +### 3. Augment with DAO Integration + +!!! tip + We automatically confirm all actions, though you can omit the `--auto-confirm` flag to see the actions that will be taken. + +```shell +adev augment customs openapi3 --use-daos --auto-confirm ``` -The augmenting process creates the following: +### 4. Complete Configuration -1. Handler methods: For each path defined in the OpenAPI specification, a corresponding handler method is generated, along with a general handler and resolver method. -2. Dialogues.py: A boilerplate dialogues file is generated. +Follow steps 5-6 from the basic handler augmentation. ## How It Works -The augmentation process involves several steps: +The augmentation process: -1. Loading and validating the OpenAPI specification -2. Generating Handler methods for each path +1. Loads and validates the OpenAPI specification +2. Generates handler methods for each defined path +3. Creates resolver methods to handle requests +4. When using DAOs, connects endpoints to database operations -For more details on the implementation, refer to: -`auto_dev/handler/scaffolder.py` +!!! tip "Implementation Details" + For more details on the implementation, see `auto_dev/handler/scaffolder.py` ## Customization -The generated Handler methods use Jinja2 templates for customization. If you need to modify the structure of the generated classes, you can update the templates located in the `JINJA_TEMPLATE_FOLDER`. +The generated handlers use Jinja2 templates. To customize the structure: -## Next Steps +1. Locate the templates in the `JINJA_TEMPLATE_FOLDER` +2. Modify them to suit your specific requirements +3. Re-run the augmentation command -After augmenting your handler: +## Best Practices -- Review the generated handler methods in the `handlers.py` file. +!!! warning "Keep in Sync" + Always regenerate handlers after modifying your OpenAPI specification to maintain consistency. -Remember to regenerate the Handlers if you make changes to your OpenAPI specification to keep them in sync. \ No newline at end of file +!!! success "Review Generated Code" + After augmentation, review the generated code in `handlers.py` to ensure it meets your requirements. diff --git a/tests/test_docs.py b/tests/test_docs.py index b8ee7285..aa902e2f 100644 --- a/tests/test_docs.py +++ b/tests/test_docs.py @@ -34,7 +34,7 @@ def extract_code_blocks(doc): # we test the documents works. -documentation = ["docs/fsm.md", "docs/dao.md"] +documentation = ["docs/fsm.md", "docs/dao.md", "docs/openapi.md"] logger = getLogger()