diff --git a/WEAVE-REFERENCE-FIXES-SUMMARY.md b/WEAVE-REFERENCE-FIXES-SUMMARY.md new file mode 100644 index 0000000000..61128c0b41 --- /dev/null +++ b/WEAVE-REFERENCE-FIXES-SUMMARY.md @@ -0,0 +1,159 @@ +# Weave Reference Generation Script Fixes + +This document summarizes the fixes applied to the Weave reference documentation generation scripts based on PR #1888 feedback. + +## Issues Fixed + +### 1. Models Reference Files Being Renamed (CRITICAL BUG) +**Problem**: `fix_casing.py` was incorrectly targeting `models/ref/python/public-api` files instead of Weave reference docs. + +**Fix**: Updated `fix_casing.py` to only target `weave/reference/python-sdk` files. +- Changed path from `models/ref/python/public-api` to `weave/reference/python-sdk` +- Removed the logic that was renaming Models API files (ArtifactCollection, etc.) +- Added clear comments indicating this should NEVER touch Models reference docs + +**Files Modified**: +- `scripts/reference-generation/weave/fix_casing.py` + +### 2. TypeScript SDK Using PascalCase Filenames +**Problem**: TypeScript SDK files were being generated with PascalCase filenames (e.g., `Dataset.mdx`, `WeaveClient.mdx`), which causes Git case-sensitivity issues. + +**Fix**: Updated generation scripts to use lowercase filenames throughout. +- Modified `generate_typescript_sdk_docs.py` to convert filenames to lowercase when creating `.mdx` files +- Updated function and type-alias extraction to use lowercase filenames +- Updated internal links to use lowercase paths + +**Files Modified**: +- `scripts/reference-generation/weave/generate_typescript_sdk_docs.py` (lines 259, 319-320, 369-370, 379) +- `scripts/reference-generation/weave/fix_casing.py` (simplified to just convert to lowercase) + +### 3. H1 in service-api/index.mdx +**Problem**: The generated `service-api/index.mdx` had both a frontmatter title and an H1, which is redundant in Mintlify. + +**Fix**: Removed the H1 heading since Mintlify uses the frontmatter title. + +**Files Modified**: +- `scripts/reference-generation/weave/generate_service_api_spec.py` (line 31) + +### 4. Duplicate H3 Headings in service-api.mdx +**Problem**: The `service-api.mdx` file had duplicate category sections (e.g., "### Calls" appeared on both line 23 and line 158), listing the same endpoints twice. + +**Fix**: Added deduplication logic to prevent duplicate categories and duplicate endpoints. +- Track which categories have been written to prevent duplicate H3 headings +- Deduplicate endpoints within each category by (method, path) tuple +- This prevents the same endpoint from being listed multiple times if it appears in the OpenAPI spec with duplicate tags + +**Files Modified**: +- `scripts/reference-generation/weave/update_service_api_landing.py` (lines 99-118) + +### 5. Markdown Table Formatting Errors (------ lines) +**Problem**: Python SDK docs contained standalone lines with just dashes (`------`) which break markdown parsing. + +**Example**: In `trace_server_interface.mdx`, lines like 22, 30, 39, etc. had `------` that created invalid table structures. + +**Fix**: Added regex pattern to remove these malformed table separators. +- Pattern: `\n\s*------+\s*\n` → `\n\n` +- This removes lines that are just dashes with optional whitespace + +**Files Modified**: +- `scripts/reference-generation/weave/generate_python_sdk_docs.py` (lines 258-260) + +## Testing Recommendation + +Before merging, test the fixes by running the reference generation locally: + +```bash +# From the docs repository root +cd scripts/reference-generation/weave +python generate_weave_reference.py +``` + +Then verify: +1. No files in `models/ref/python/public-api` were modified +2. All TypeScript SDK files in `weave/reference/typescript-sdk/` have lowercase filenames +3. `weave/reference/service-api/index.mdx` has no H1 heading +4. `weave/reference/service-api.mdx` has no duplicate H3 category headings +5. No `------` lines in `weave/reference/python-sdk/trace_server/trace_server_interface.mdx` +6. In `docs.json`, modules under `weave/reference/python-sdk/trace/` are grouped as "Core" (not "Other") +7. In `docs.json`, the Service API `openapi` configuration uses the local spec (not a GitHub URL) if sync_openapi_spec.py was run with `--use-local` + +### 6. Incorrect Section Grouping ("Core" → "Other") +**Problem**: Python SDK modules in the `trace/` directory were being incorrectly grouped as "Other" instead of "Core" in docs.json navigation. + +**Root Cause**: The path checking logic in `update_weave_toc.py` was checking `if parts[0] == "weave"`, but paths are relative to `python-sdk/`, so `parts[0]` is actually the module subdirectory (`trace`, `trace_server`, etc.), not `weave`. + +**Fix**: Corrected the path checking logic to check the actual first path component. +- Changed from checking `parts[0] == "weave"` then `parts[1] == "trace"` +- To directly checking `parts[0] == "trace"`, `parts[0] == "trace_server"`, etc. + +**Files Modified**: +- `scripts/reference-generation/weave/update_weave_toc.py` (lines 33-45) + +### 7. OpenAPI Configuration Being Overwritten +**Problem**: `update_weave_toc.py` was unconditionally overwriting the OpenAPI spec configuration in docs.json to use a remote URL, ignoring the local spec that `sync_openapi_spec.py` downloads and configures. + +**Impact**: Even though `sync_openapi_spec.py` downloads the OpenAPI spec locally and can configure docs.json to use it, `update_weave_toc.py` would immediately overwrite it with a remote GitHub URL, defeating the purpose of the local spec. + +**Fix**: Removed the Service API OpenAPI configuration code from `update_weave_toc.py`. This script should only manage Python/TypeScript SDK navigation, not the OpenAPI spec source. +- Deleted lines 209-224 that were setting `page["openapi"]` to remote URLs +- Added comment noting that OpenAPI configuration is managed by `sync_openapi_spec.py` + +**Files Modified**: +- `scripts/reference-generation/weave/update_weave_toc.py` (lines 206-207) + +### 8. Missing Root Module Documentation (CRITICAL - WEAVE PACKAGE REGRESSION) +**Problem**: The generated `python-sdk.mdx` file is only 8 lines (just frontmatter), completely missing all the important API documentation for functions like `init()`, `publish()`, `ref()`, `get()`, etc. + +**Expected**: The current version (Weave 0.52.10) has 2074 lines documenting all the core Weave functions and classes. + +**Root Cause**: **This is a WEAVE PACKAGE REGRESSION, not a script bug.** + +Something changed in Weave between versions **0.52.10** (current docs) and **0.52.16** (PR version) that broke documentation generation for the root `weave` module. The generation scripts haven't changed, and lazydocs hasn't changed - so this is an upstream issue in the Weave package itself. + +Possible causes: +1. Changes to `weave/__init__.py` that affect how the module exports its public API +2. Module structure refactoring that lazydocs can't handle +3. New import patterns or lazy loading that breaks introspection + +**Status**: **CRITICAL UPSTREAM BUG** - This makes the Python SDK documentation completely unusable for version 0.52.16. + +**Action Required**: Report this to the Weave team immediately: +1. File an issue: https://github.com/wandb/weave/issues +2. Include: "Documentation generation broken in 0.52.16 - root module exports not discoverable by lazydocs" +3. Mention: "Works fine in 0.52.10, broken in 0.52.16" +4. Tag: @dbrian57 or relevant Weave maintainers + +**Recommendation**: +- **DO NOT MERGE PR #1888** - it will break Python SDK documentation +- Either: Fix the Weave package and regenerate docs +- Or: Stay on 0.52.10 documentation until the Weave package is fixed + +**Files to Investigate** (in Weave repo): +- `weave/__init__.py` between versions 0.52.10 and 0.52.16 +- Any structural changes to the weave package in that version range + +### 9. OpenAPI Spec Validation (New Feature) +**Enhancement**: Added validation to detect issues in the OpenAPI spec itself, which can help identify upstream problems. + +**Features**: +- Detects duplicate endpoint definitions (same method+path defined multiple times) +- Identifies endpoints appearing in multiple categories/tags +- Warns when critical issues like duplicate endpoints are found +- Suggests reporting issues to the Weave team when spec problems are detected + +**Files Modified**: +- `scripts/reference-generation/weave/sync_openapi_spec.py` (added `validate_spec()` function and integration in `main()`) + +This will help identify if duplicate H3s or other issues originate from the OpenAPI spec rather than our generation scripts. + +## Files Modified Summary + +1. `scripts/reference-generation/weave/fix_casing.py` +2. `scripts/reference-generation/weave/generate_typescript_sdk_docs.py` +3. `scripts/reference-generation/weave/generate_service_api_spec.py` +4. `scripts/reference-generation/weave/update_service_api_landing.py` +5. `scripts/reference-generation/weave/generate_python_sdk_docs.py` +6. `scripts/reference-generation/weave/update_weave_toc.py` +7. `scripts/reference-generation/weave/sync_openapi_spec.py` (new validation feature) + +All fixes are backward compatible and will take effect on the next reference documentation generation run. diff --git a/scripts/reference-generation/weave/fix_casing.py b/scripts/reference-generation/weave/fix_casing.py index b1225629c5..7971d5475d 100755 --- a/scripts/reference-generation/weave/fix_casing.py +++ b/scripts/reference-generation/weave/fix_casing.py @@ -12,108 +12,47 @@ from pathlib import Path def fix_typescript_casing(base_path): - """Fix TypeScript SDK file casing.""" - print("Fixing TypeScript SDK file casing...") + """Fix TypeScript SDK file casing - ensure all files use lowercase.""" + print("Fixing TypeScript SDK file casing to lowercase...") - ts_base = Path(base_path) / "weave/reference/typescript-sdk/weave" + ts_base = Path(base_path) / "weave/reference/typescript-sdk" if not ts_base.exists(): print(f" TypeScript SDK path not found: {ts_base}") return - # Define correct names for each directory - casing_rules = { - "classes": { - "dataset": "Dataset", - "evaluation": "Evaluation", - "weaveclient": "WeaveClient", - "weaveobject": "WeaveObject", - }, - "interfaces": { - "callschema": "CallSchema", - "callsfilter": "CallsFilter", - "weaveaudio": "WeaveAudio", - "weaveimage": "WeaveImage", - }, - "functions": { - # Functions should be lowercase/camelCase - "init": "init", - "login": "login", - "op": "op", - "requirecurrentcallstackentry": "requireCurrentCallStackEntry", - "requirecurrentchildsummary": "requireCurrentChildSummary", - "weaveaudio": "weaveAudio", - "weaveimage": "weaveImage", - "wrapopenai": "wrapOpenAI", - }, - "type-aliases": { - "op": "Op", # Type alias Op is uppercase - "opdecorator": "OpDecorator", - "messagesprompt": "MessagesPrompt", - "stringprompt": "StringPrompt", - } - } + # All TypeScript SDK files should use lowercase filenames for consistency + # This applies to classes, functions, interfaces, and type-aliases + subdirs_to_check = ["classes", "functions", "interfaces", "type-aliases"] - for dir_name, rules in casing_rules.items(): - dir_path = ts_base / dir_name + for subdir in subdirs_to_check: + dir_path = ts_base / subdir if not dir_path.exists(): continue for file in dir_path.glob("*.mdx"): - basename = file.stem.lower() - if basename in rules: - correct_name = rules[basename] - if file.stem != correct_name: - new_path = file.parent / f"{correct_name}.mdx" - print(f" Renaming: {file.name} → {correct_name}.mdx") - shutil.move(str(file), str(new_path)) + # Convert filename to lowercase + lowercase_name = file.stem.lower() + if file.stem != lowercase_name: + new_path = file.parent / f"{lowercase_name}.mdx" + print(f" Renaming: {file.name} → {lowercase_name}.mdx") + shutil.move(str(file), str(new_path)) def fix_python_casing(base_path): - """Fix Python SDK file casing.""" - print("Fixing Python SDK file casing...") + """Fix Python SDK file casing for WEAVE reference docs only.""" + print("Fixing Weave Python SDK file casing...") - py_base = Path(base_path) / "models/ref/python/public-api" + # IMPORTANT: This should ONLY touch Weave reference docs, never Models reference docs + py_base = Path(base_path) / "weave/reference/python-sdk" if not py_base.exists(): - print(f" Python SDK path not found: {py_base}") + print(f" Weave Python SDK path not found: {py_base}") return - # Python class files that should be uppercase - uppercase_files = { - "artifactcollection": "ArtifactCollection", - "artifactcollections": "ArtifactCollections", - "artifactfiles": "ArtifactFiles", - "artifacttype": "ArtifactType", - "artifacttypes": "ArtifactTypes", - "betareport": "BetaReport", - "file": "File", - "member": "Member", - "project": "Project", - "registry": "Registry", - "run": "Run", - "runartifacts": "RunArtifacts", - "sweep": "Sweep", - "team": "Team", - "user": "User", - } - - # Files that should remain lowercase - lowercase_files = ["api", "artifacts", "automations", "files", "projects", - "reports", "runs", "sweeps", "_index"] + # For Weave Python SDK, we generally want lowercase filenames + # Only specific files might need special casing - currently none known + # Most Weave modules use lowercase with underscores (e.g., weave_client.mdx) - for file in py_base.glob("*.mdx"): - basename = file.stem.lower() - - if basename in uppercase_files: - correct_name = uppercase_files[basename] - if file.stem != correct_name: - new_path = file.parent / f"{correct_name}.mdx" - print(f" Renaming: {file.name} → {correct_name}.mdx") - shutil.move(str(file), str(new_path)) - elif basename in lowercase_files: - # Ensure these stay lowercase - if file.stem != basename: - new_path = file.parent / f"{basename}.mdx" - print(f" Renaming: {file.name} → {basename}.mdx") - shutil.move(str(file), str(new_path)) + print(f" Weave Python SDK files are generated with correct casing") + print(f" No casing changes needed for Weave reference documentation") def main(): """Main function to fix all casing issues.""" diff --git a/scripts/reference-generation/weave/generate_python_sdk_docs.py b/scripts/reference-generation/weave/generate_python_sdk_docs.py index 46f6d0e9bd..63f22ee16b 100755 --- a/scripts/reference-generation/weave/generate_python_sdk_docs.py +++ b/scripts/reference-generation/weave/generate_python_sdk_docs.py @@ -255,6 +255,10 @@ def generate_module_docs(module, module_name: str, src_root_path: str, version: # Remove ` at the start of lines that don't have a closing content = re.sub(r'^- `([^`\n]*?)$', r'- \1', content, flags=re.MULTILINE) + # Remove malformed table separators that lazydocs sometimes generates + # These appear as standalone lines with just dashes (------) which break markdown parsing + content = re.sub(r'\n\s*------+\s*\n', '\n\n', content) + # Fix parameter lists that have been broken by lazydocs # Strategy: Parse all parameters into a structured format, then reconstruct them properly def fix_parameter_lists(text): diff --git a/scripts/reference-generation/weave/generate_service_api_spec.py b/scripts/reference-generation/weave/generate_service_api_spec.py index ad7bb0f456..6f078d713c 100755 --- a/scripts/reference-generation/weave/generate_service_api_spec.py +++ b/scripts/reference-generation/weave/generate_service_api_spec.py @@ -28,8 +28,6 @@ def main(): description: "REST API endpoints for the Weave service" --- -# Weave Service API - The Weave Service API provides REST endpoints for interacting with the Weave tracing service. ## Available Endpoints diff --git a/scripts/reference-generation/weave/generate_typescript_sdk_docs.py b/scripts/reference-generation/weave/generate_typescript_sdk_docs.py index d980f999ff..7d189e06b8 100755 --- a/scripts/reference-generation/weave/generate_typescript_sdk_docs.py +++ b/scripts/reference-generation/weave/generate_typescript_sdk_docs.py @@ -255,8 +255,9 @@ def convert_to_mintlify_format(docs_dir): # Just ensure .md extension is removed (already done above) pass - # Write as .mdx file - mdx_file = md_file.with_suffix('.mdx') + # Write as .mdx file with lowercase filename (avoid Git case sensitivity issues) + lowercase_stem = md_file.stem.lower() + mdx_file = md_file.parent / f"{lowercase_stem}.mdx" mdx_file.write_text(content) # Remove original .md file @@ -314,17 +315,18 @@ def extract_members_to_separate_files(docs_path): {func_content.replace(f'### {func_name}', f'# {func_name}')}""" - # Write the function file - func_file = functions_dir / f"{func_name}.mdx" + # Write the function file with lowercase filename (avoid Git case sensitivity issues) + func_filename = func_name.lower() + func_file = functions_dir / f"{func_filename}.mdx" func_file.write_text(func_file_content) - functions_found.append(func_name) - print(f" ✓ Extracted {func_name}.mdx") + functions_found.append(func_filename) + print(f" ✓ Extracted {func_filename}.mdx") if functions_found: # Remove the detailed function documentation from index content = function_pattern.sub('', content) - # Update the Functions section with links + # Update the Functions section with links (functions_found already has lowercase names) functions_section = "\n### Functions\n\n" for func in functions_found: functions_section += f"- [{func}](functions/{func})\n" @@ -363,17 +365,18 @@ def extract_members_to_separate_files(docs_path): {alias_content.replace(f'### {alias_name}', f'# {alias_name}')}""" - # Write the type alias file - alias_file = type_aliases_dir / f"{alias_name}.mdx" + # Write the type alias file with lowercase filename (avoid Git case sensitivity issues) + alias_filename = alias_name.lower() + alias_file = type_aliases_dir / f"{alias_filename}.mdx" alias_file.write_text(alias_file_content) - print(f" ✓ Extracted {alias_name}.mdx") + print(f" ✓ Extracted {alias_filename}.mdx") # Remove all extracted type aliases from index content = type_alias_pattern.sub('', content) - # Update Type Aliases section with links to all extracted type aliases + # Update Type Aliases section with links to all extracted type aliases (use lowercase filenames) if type_aliases: - type_aliases_links = [f"- [{name}](type-aliases/{name})" for _, name in type_aliases if _.startswith(f"### {name}\n\nƬ ")] + type_aliases_links = [f"- [{name}](type-aliases/{name.lower()})" for _, name in type_aliases if _.startswith(f"### {name}\n\nƬ ")] if type_aliases_links: type_aliases_section = "\n### Type Aliases\n\n" + "\n".join(sorted(type_aliases_links)) + "\n" diff --git a/scripts/reference-generation/weave/sync_openapi_spec.py b/scripts/reference-generation/weave/sync_openapi_spec.py index fff938e415..d42690e639 100755 --- a/scripts/reference-generation/weave/sync_openapi_spec.py +++ b/scripts/reference-generation/weave/sync_openapi_spec.py @@ -44,6 +44,62 @@ def spec_hash(spec: dict) -> str: return hashlib.sha256(spec_str.encode()).hexdigest() +def validate_spec(spec: dict) -> list: + """ + Validate the OpenAPI spec for potential issues. + Returns list of warning messages about spec issues. + """ + warnings = [] + + paths = spec.get("paths", {}) + + # Track endpoint definitions to detect duplicates + endpoint_map = {} # (method, path) -> [operation_ids] + tag_endpoint_map = {} # tag -> [(method, path)] + + for path, path_item in paths.items(): + for method in ["get", "post", "put", "delete", "patch"]: + if method not in path_item: + continue + + operation = path_item[method] + operation_id = operation.get("operationId", "") + tags = operation.get("tags", []) + + # Check for duplicate endpoint definitions + endpoint_key = (method.upper(), path) + if endpoint_key not in endpoint_map: + endpoint_map[endpoint_key] = [] + endpoint_map[endpoint_key].append(operation_id) + + # Track endpoints by tag to detect if endpoints appear in multiple tags + for tag in tags: + if tag not in tag_endpoint_map: + tag_endpoint_map[tag] = [] + tag_endpoint_map[tag].append(endpoint_key) + + # Check for actual duplicates (same endpoint with different operation IDs) + for endpoint_key, operation_ids in endpoint_map.items(): + if len(operation_ids) > 1: + method, path = endpoint_key + warnings.append(f" ⚠ Duplicate endpoint: {method} {path} defined {len(operation_ids)} times with operation IDs: {operation_ids}") + + # Check for endpoints appearing in multiple categories (tags) + endpoint_tag_count = {} + for tag, endpoints in tag_endpoint_map.items(): + for endpoint in endpoints: + if endpoint not in endpoint_tag_count: + endpoint_tag_count[endpoint] = [] + endpoint_tag_count[endpoint].append(tag) + + for endpoint, tags in endpoint_tag_count.items(): + if len(tags) > 1: + method, path = endpoint + warnings.append(f" ℹ Endpoint {method} {path} appears in multiple categories: {tags}") + + return warnings + + def compare_specs(local_spec: dict, remote_spec: dict) -> Tuple[bool, list]: """ Compare local and remote specs. @@ -131,6 +187,20 @@ def main(): print(" ✗ No local spec and couldn't fetch remote spec") return 1 + # Validate the remote spec for issues + print("\n Validating OpenAPI spec...") + spec_warnings = validate_spec(remote_spec) + if spec_warnings: + print(" ⚠ OpenAPI spec validation warnings:") + for warning in spec_warnings: + print(warning) + if any("Duplicate endpoint" in w for w in spec_warnings): + print("\n ⚠ CRITICAL: Duplicate endpoint definitions found!") + print(" This may indicate an issue in the upstream OpenAPI spec.") + print(" Consider reporting this to the Weave team: https://github.com/wandb/weave/issues") + else: + print(" ✓ OpenAPI spec validation passed") + # Load local spec local_spec = load_local_spec(local_spec_path) diff --git a/scripts/reference-generation/weave/update_service_api_landing.py b/scripts/reference-generation/weave/update_service_api_landing.py index 77abeb8052..bcfd038378 100755 --- a/scripts/reference-generation/weave/update_service_api_landing.py +++ b/scripts/reference-generation/weave/update_service_api_landing.py @@ -96,13 +96,24 @@ def generate_endpoints_section(endpoints: Dict[str, List[Tuple[str, str, str, st if tag not in category_order: category_order.append(tag) + # Track which categories we've already written to prevent duplicate H3s + written_categories = set() + for category in category_order: - if category not in endpoints: + if category not in endpoints or category in written_categories: continue - + + written_categories.add(category) lines.append(f"\n### {category}\n\n") + # Deduplicate endpoints within the category by (method, path) to avoid listing the same endpoint twice + seen_endpoints = set() for method, path, operation_id, summary in endpoints[category]: + endpoint_key = (method, path) + if endpoint_key in seen_endpoints: + continue + seen_endpoints.add(endpoint_key) + url = generate_endpoint_url(operation_id, category) lines.append(f"- **[{method} {path}]({url})** - {summary}\n") diff --git a/scripts/reference-generation/weave/update_weave_toc.py b/scripts/reference-generation/weave/update_weave_toc.py index aef71eda26..b08883f521 100755 --- a/scripts/reference-generation/weave/update_weave_toc.py +++ b/scripts/reference-generation/weave/update_weave_toc.py @@ -30,17 +30,15 @@ def get_generated_python_modules(): parts = list(rel_path.parts) parts[-1] = parts[-1].replace('.mdx', '') - # Determine the group based on path - if len(parts) >= 2: - if parts[0] == "weave": - if parts[1] == "trace": - group = "Core" - elif parts[1] == "trace_server": - group = "Trace Server" - elif parts[1] == "trace_server_bindings": - group = "Trace Server Bindings" - else: - group = "Other" + # Determine the group based on path structure + # Paths are relative to python-sdk/, so parts[0] is the module directory + if len(parts) >= 1: + if parts[0] == "trace": + group = "Core" + elif parts[0] == "trace_server": + group = "Trace Server" + elif parts[0] == "trace_server_bindings": + group = "Trace Server Bindings" else: group = "Other" else: @@ -205,26 +203,8 @@ def update_docs_json(python_modules, typescript_items, service_endpoints): print(f"✓ Updated TypeScript SDK with {sum(len(items) for items in typescript_items.values())} items") updated = True - # Update Service API - for page in reference_pages: - if isinstance(page, dict) and page.get("group") == "Service API": - # Point to the versioned OpenAPI spec in the Weave repo - # This spec already has the necessary filters applied - import os - version = os.environ.get('WEAVE_VERSION', 'latest') - - # The version should already be resolved by the Python SDK generation - # which converts "latest" to an actual version number - if version and version != "latest": - # Use the specific version tag - openapi_url = f"https://raw.githubusercontent.com/wandb/weave/v{version}/sdks/node/weave.openapi.json" - else: - # Fallback to master if somehow we don't have a version - openapi_url = "https://raw.githubusercontent.com/wandb/weave/master/sdks/node/weave.openapi.json" - - page["openapi"] = openapi_url - print(f"✓ Updated Service API to use versioned OpenAPI spec: {openapi_url}") - updated = True + # Note: Service API OpenAPI configuration is managed by sync_openapi_spec.py + # We don't modify it here to preserve the local vs remote spec choice break break