Protocol Scaffolder #618

Karrenbelt · 2025-03-22T16:08:09Z

A re-implementation of the protocol scaffolder

no longer based on Python’s ast, but on a protobuf ast (protodantic.py, test cases)
custom primitives that match the .proto definition one-to-one, enabling precise bidirectional translation
minimalistic dialogue and message test scaffolds
automated hypothesis strategy generation for both pydantic models derived from protobuf and protocol performatives
a protocol scaffolder with minimal class usage and maximal functional composition 🦾
an integration test that mimics Inception’s layered architecture: it scaffolds protocols and hypothesis tests in an isolated filesystem, then launches a subprocess to execute the generated tests via pytest

…l values

…d values

…num values

Karrenbelt · 2025-04-11T22:41:44Z

auto_dev/protocols/primitives.py

+        return super().__new__(cls, to_float32(float(value)))
+
+
+class Int32(BaseConstrainedInt):


we need this, not to constrain the data per se (pydantic can deal with that via i.e. conint), but to encode to the correct proto type; the information from i.e. conint does not carry over to the python int generated, so in many cases you cannot discern which type of proto integer it was / needs to be converted into

Karrenbelt · 2025-04-12T13:08:44Z

auto_dev/protocols/formatter.py

+class ResolvedType(NamedTuple):
+    fully_qualified_name: str
+    ast_node: MessageAdapter | ast.Enum | None = None
+
+    @property
+    def is_enum(self):
+        return isinstance(self.ast_node, ast.Enum)
+
+    @property
+    def is_message(self):
+        return isinstance(self.ast_node, MessageAdapter)
+
+    def __str__(self):
+        return self.fully_qualified_name


This facilitates checking, as we traverse parents, what type we're dealing with

…est_scaffold_protocol

Karrenbelt · 2025-04-12T22:10:39Z

auto_dev/data/templates/protocols/performatives.jinja

+for cls in BaseModel.__subclasses__():
+    cls.model_rebuild()


The sames goes for this module: the performative models (generated under tests/performatives.py) also need to update forward references in order for hypothesis to generate strategies from them correctly using st.from_type

Karrenbelt · 2025-04-12T22:15:58Z

tests/test_protocol.py

+    # Point PYTHONPATH to the temporary project root so generated modules are discoverable
+    env = os.environ.copy()
+    env["PYTHONPATH"] = str(repo_root)
+
+    command = ["adev", "-v", "scaffold", "protocol", str(protocol_spec)]
+    result = subprocess.run(command, env=env, check=False, text=True, capture_output=True)
+    if result.returncode != 0:
+        msg = f"Protocol scaffolding failed: {result.stderr}"
+        raise ValueError(msg)
+
+    assert protocol_outpath.exists()
+
+    test_dir = protocol_outpath / "tests"
+    command = ["pytest", str(test_dir), "-vv", "-s", "--tb=long", "-p", "no:warnings"]
+    result = subprocess.run(
+        command,
+        env=env,
+        check=False,
+        text=True,
+        capture_output=True,
+    )
+
+    assert result.returncode == 0, f"Failed pytest on generated protocol: {result.stderr}"


before, i passed the env with custom PYTHONPATH only to the last subprocess call and tests passed. However, no code was generated in the proper location (!) as I didn't pass the custom env to the adev -v scaffold protocol call. This resulted in a situation where no tests were ran in the subprocess, causing this test to pass without actually running any tests. We should find a better way to assert this, i.e. by checking output of the subprocess call and comparing that reported output (parsing stdout) to expected result. For now, tho, this assert protocol_outpath.exists() ensures that the expected outpath was generated (which fails when one does not pass the env)

We have the subprocess task thing for this quite extensively used throughout the repo

Which gives you access to all the env var stuff etc and provides the abstraction

I think there’s a small mix-up in how loading and executing (newly generated) code occurs in the context of an isolated_filesystem:

We run the codegen inside an isolated temp directory (e.g. /tmp/...) that isn’t on the default PYTHONPATH.

Spawning a separate python process guarantees zero cached imports; otherwise you’ll end up with stale code (cache invalidation truly is one of the two hard things in computer science - more details in isolated_filesystem contextmanager issues #609)

When you switch into the temp directory, you must also update PYTHONPATH (or use -m with the correct module path) so that the newly written files become importable in that subprocess.

…ProtocolSpec.template_context` cached property

tests/data/protocols/protobuf/map_scalar_keys.proto

Makefile

8ball030 · 2025-04-07T16:07:17Z

auto_dev/connections/scaffolder.py

+        protocol_name = self.protocol.metadata.name
+        protocol_author = self.protocol.metadata.author
+        speech_acts = list(self.protocol.metadata.speech_acts)
+        roles = list(self.protocol.interaction_model.roles)


This isnt a key on the original protocol specs?

well this is the custom data model for a protocol spec consisting of 3 yaml parts. I've simply named these individual sections (as per the pydantic model here)

auto_dev/protocols/primitives.py

8ball030 · 2025-04-07T16:09:43Z

pyproject.toml

 exclude = "/(\n    \\.eggs\n  | \\.git\n  | \\.hg\n  | \\.mypy_cache\n  | \\.tox\n  | \\.venv\n  | _build\n  | buck-out\n  | build\n  | dist\n)/\n"

 [tool.poetry.dependencies]
-python = ">=3.10,<3.14"


why change this?

we do no want to drop this support

please read the comments i leave on my PRs

Bro there is like 200 commits here.

It doesnt even show on this page.

8ball030 · 2025-04-07T16:09:58Z

pyproject.toml

 openapi-spec-validator = "0.2.8"
 disutils = "^1.4.32.post2"
 setuptools = "^75.8.0"
+proto-schema-parser = "^1.5.0"


this enforces less than 3.13< ?

No. I think the bump to include 3.13 previously was an error in a previous PR of mine (see git blame)

8ball030 · 2025-04-29T09:53:05Z

tests/conftest.py



+@pytest.fixture(scope="module")
+def module_scoped_dummy_agent_tim() -> Path:


so now there is like 3 different agents?

Why?

because of the time it takes to create this "isolated_filesystem". See the difference in run times (1 and 3) in this comment

why not remove the unused ones?

Karrenbelt added 30 commits March 16, 2025 14:26

feat: templates/protocols/dialogues.jinja

09b67c6

tests: add test cases for primitive types and basic enums

46d869b

tests: add map-related test cases including nested and variant maps

cb848e9

tests: add nested message and oneof-related test cases

1cc6687

tests: add recursive structure test cases

a89ab5c

feat: first protodantic.jinja draft

78ae9dc

feat: add proto-schema-parser to pyproject.toml

a44ac42

feat: first protodantic.py draft

ac02f85

feat: protocols/hypothesis.jinja

5dd5c53

feat: create and write code and tests in protodantic.py

7153cb8

chore: cleanup protodantic.py

537073d

tests: add integration tests for protodantic.create

ed75c9a

fix: import path from repo_root

f6cb91c

feat: add .encode and .decode to protodantic.jinja

f8e286a

tests: add .encode and .decode invocation to hypothesis.jinja

2d116d0

feat: add messages_pb2 generation via protoc

77bd673

tests: add optional_primitives.proto

e75f51c

feat: update protodantic.jinja and hypothesis.jinja to handle optiona…

c4eba2f

…l values

refactor: simplify protodantic.jinja .decode and .encode

9444d85

chore: rename fields repeated_primitives.proto

8cc27f5

tests: add repeated_primitives.proto

17599f0

feat: update protodantic.jinja and hypothesis.jinja to handle repeate…

c7d848c

…d values

tests: add basic_enum.proto

510581d

feat: update protodantic.jinja and hypothesis.jinja to handle basic e…

00b0b33

…num values

tests: add optional_enum.proto

5d82d19

feat: update protodantic.jinja to handle optional enum values

1393402

chore: make enum .decode strict

8792f1a

tests: add repeated_enum.proto

94352db

feat: update protodantic.jinja to handle repeated enum values

91547e3

fix: enum strategies based on field cardinality in hypothesis.jinja

0aa9e2e

Karrenbelt commented Apr 11, 2025

View reviewed changes

Karrenbelt added 4 commits April 12, 2025 11:22

test: add map_nested.proto

860c670

fix: qualified_type via introduction of ResolvedType

0fd5c03

refactor: renaming and property -> cachedproperty

850150d

tests: add map_of_map.proto

b15a79b

Karrenbelt commented Apr 12, 2025

View reviewed changes

Karrenbelt added 8 commits April 12, 2025 20:58

refactor: explicit ast. reference in adapters.py

96ef64d

chore: make fmt lint

5e3f23f

tests: add map_scalar_keys.proto

76f48d5

fix: point PYTHONPATH to tmp dir before adev scaffold protocol in t…

bff04dd

…est_scaffold_protocol

feat: protocols/performatives.jinja

fdda705

refactor: import from tests/performatives.py in other tests

9dceeac

chore: remove empty protocols/tests.jinja

e79c40b

chore: make fmt lint

d5b9281

Karrenbelt commented Apr 12, 2025

View reviewed changes

Karrenbelt added 7 commits April 13, 2025 13:00

refactor: unify template rendering with TemplateContext model and `…

46efb90

…ProtocolSpec.template_context` cached property

refactor: remove initial_performative_types

830b52d

refactor: moved hardcoded template paths into JinjaTemplates model

e872187

refactor: protodantic.py

b42b11a

refactor: TemplateContext in protodantic.py

2f0ff1b

refactor: _prepare_pb2 in protodantic.py

2965899

refactor: ImportPaths in protodantic.py

87e7004

Karrenbelt commented Apr 13, 2025

View reviewed changes

tests/data/protocols/protobuf/map_scalar_keys.proto Show resolved Hide resolved

8ball030 reviewed Apr 29, 2025

View reviewed changes

8ball030 merged commit c2abb54 into main Apr 29, 2025
7 checks passed

8ball030 deleted the refactor/protocol_scaffolder branch April 29, 2025 15:39

		return super().__new__(cls, to_float32(float(value)))


		class Int32(BaseConstrainedInt):



		@pytest.fixture(scope="module")
		def module_scoped_dummy_agent_tim() -> Path:

Protocol Scaffolder #618

Protocol Scaffolder #618

Uh oh!

Conversation

Karrenbelt commented Mar 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Karrenbelt Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Karrenbelt commented Mar 22, 2025 •

edited

Loading

Karrenbelt Apr 11, 2025 •

edited

Loading