Plan MeshCore Python application with serial interface #1

jinglemansweep · 2025-11-24T22:38:08Z

This pull request introduces the initial implementation of the MeshCore Sidekick application, providing a companion service for MeshCore devices with event collection, persistence, and a REST API. The changes include a comprehensive configuration system, a main application entry point with support for both real and mock MeshCore devices, database integration, and REST API scaffolding. The documentation has also been significantly expanded to cover features, usage, and development details.

Application Core and Configuration:

Added the main application entry point in src/meshcore_sidekick/__main__.py, implementing startup, shutdown, event subscription, database cleanup, and signal handling logic for both real and mock MeshCore operation.
Implemented a flexible configuration system in src/meshcore_sidekick/config.py, supporting CLI arguments, environment variables, and defaults for all major runtime options.

Database Layer:

Added database initialization and model imports in src/meshcore_sidekick/database/__init__.py, enabling event persistence and query capabilities.

API and Documentation:

Added REST API scaffolding in src/meshcore_sidekick/api/__init__.py and src/meshcore_sidekick/api/routes/__init__.py, preparing for FastAPI-based endpoints. [1] [2]
Significantly expanded the README.md with feature overview, quick start guides for development and production (including Docker), configuration details, database querying instructions, API documentation, and development workflow.

Project Metadata:

Added project metadata and dependency management with Poetry in pyproject.toml, specifying dependencies for both runtime and development.
Added package versioning and module docstring in src/meshcore_sidekick/__init__.py.

Implemented core functionality: - Project structure with Python 3.11+ support - SQLite database with SQLAlchemy ORM (14 tables) - MeshCore interface abstraction layer - Real MeshCore implementation (meshcore_py wrapper) - Mock MeshCore with random events and scenario playback - Event subscriber and persistence layer - Configuration management (CLI > env > defaults) - Prometheus metrics collectors - Structured logging (JSON/text formats) - Database cleanup with configurable retention Features: - Support for all MeshCore event types - Node tracking with prefix-based queries - Message persistence (contact and channel) - Advertisement storage with GPS coordinates - Telemetry, trace paths, and acknowledgments - Mock scenarios: simple_chat, trace_path_test, telemetry_collection, network_stress Tested and verified: - Mock MeshCore random event generation - Scenario playback with "simple_chat" - Database persistence of all event types - Configuration system with CLI arguments

- Add INFO level logging for event subscriptions - Add INFO level logging for event processing - Enable start_auto_message_fetching() if available - Add exception tracebacks for better debugging - Log number of event handlers and subscriptions

- Await start_auto_message_fetching() coroutine properly - Add MESSAGES_WAITING and RX_LOG_DATA to handler map (informational) - Improve logging for unknown vs informational event types

Features: - Full database report with all tables and statistics - Query specific data types (messages, nodes, advertisements, etc.) - Activity timeline for last N hours - Event breakdown by type - Flexible output options via CLI arguments - Support for custom database paths Usage examples: python -m meshcore_sidekick.query # Full report python -m meshcore_sidekick.query --summary # Summary only python -m meshcore_sidekick.query --messages 20 # Recent messages python -m meshcore_sidekick.query --nodes 10 # Discovered nodes python -m meshcore_sidekick.query --activity 6 # Last 6 hours Updated README with query tool documentation.

Copilot

Pull request overview

This PR introduces the foundational implementation of the MeshCore Sidekick application, a companion service for MeshCore mesh networking devices. The application provides event collection, persistence to SQLite, and scaffolding for a REST API. It supports both real hardware via serial interface and a sophisticated mock implementation for development without hardware.

Key highlights:

Comprehensive event handling system with 14 database tables covering all MeshCore event types
Flexible configuration system supporting CLI arguments, environment variables, and defaults
Mock MeshCore implementation with 5 predefined scenarios and random event generation
Database cleanup with configurable retention policies
Prometheus metrics infrastructure (defined but not fully wired)
Command-line query tool for database exploration

Reviewed changes

Copilot reviewed 26 out of 27 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
pyproject.toml	Poetry project configuration with Python 3.11+ and all runtime/dev dependencies
requirements.txt	Pinned runtime dependencies for pip-based installations
README.md	User-facing documentation with quick start guides and usage examples
PLAN.md	Detailed implementation plan documenting completed and planned features
src/meshcore_sidekick/init.py	Package initialization with version metadata
src/meshcore_sidekick/main.py	Main application entry point with lifecycle management and signal handling
src/meshcore_sidekick/main.py	Convenience wrapper for running as a module
src/meshcore_sidekick/config.py	Configuration management with CLI/env/default priority handling
src/meshcore_sidekick/database/models.py	SQLAlchemy 2.0 models for 14 event/data tables
src/meshcore_sidekick/database/engine.py	Database engine with session management and SQLite optimizations
src/meshcore_sidekick/database/cleanup.py	Retention policy enforcement with configurable data cleanup
src/meshcore_sidekick/database/init.py	Database module exports
src/meshcore_sidekick/meshcore/interface.py	Abstract base class defining MeshCore interface contract
src/meshcore_sidekick/meshcore/real.py	Real hardware implementation wrapping meshcore_py library
src/meshcore_sidekick/meshcore/mock.py	Mock implementation with random and scenario-based event generation
src/meshcore_sidekick/meshcore/scenarios.py	Five predefined test scenarios with dynamic value placeholders
src/meshcore_sidekick/meshcore/init.py	MeshCore module exports
src/meshcore_sidekick/subscriber/event_handler.py	Event processing and database persistence logic
src/meshcore_sidekick/subscriber/metrics.py	Prometheus metrics collector definitions
src/meshcore_sidekick/subscriber/init.py	Subscriber module exports
src/meshcore_sidekick/utils/address.py	Public key validation, normalization, and prefix utilities
src/meshcore_sidekick/utils/logging.py	Custom JSON and colored text logging formatters
src/meshcore_sidekick/utils/init.py	Utilities module exports
src/meshcore_sidekick/query.py	Command-line tool for querying captured MeshCore data
src/meshcore_sidekick/api/init.py	API module scaffolding (placeholder for Phase 2)
src/meshcore_sidekick/api/routes/init.py	API routes scaffolding (placeholder for Phase 2)
tests/init.py	Test package initialization

Copilot · 2025-11-24T22:42:08Z

pyproject.toml

+
+[tool.black]
+line-length = 100
+target-version = ['py312']


The target-version in the [tool.black] section is set to 'py312' but the minimum Python version specified in [tool.poetry.dependencies] is "^3.11". These should be consistent. Either update black's target-version to ['py311'] or update the minimum Python version requirement to 3.12.

Copilot · 2025-11-24T22:42:09Z

pyproject.toml

+
+[tool.ruff]
+line-length = 100
+target-version = "py312"


The target-version in the [tool.ruff] section is set to "py312" but the minimum Python version specified in [tool.poetry.dependencies] is "^3.11". These should be consistent. Either update ruff's target-version to "py311" or update the minimum Python version requirement to 3.12.

Suggested change

target-version = "py312"

target-version = "py311"

Copilot · 2025-11-24T22:42:09Z

src/meshcore_sidekick/meshcore/mock.py

+                    "text": random.choice(messages),
+                    "snr": random.uniform(-5, 30),
+                    "rssi": random.uniform(-110, -50),
+                    "timestamp": datetime.utcnow().isoformat() + "Z",


Using datetime.utcnow() is deprecated in Python 3.12+ in favor of datetime.now(timezone.utc). This appears in multiple locations throughout the mock.py file (lines 214, 230). Consider updating to the recommended approach for future compatibility.

Copilot · 2025-11-24T22:42:09Z

src/meshcore_sidekick/database/cleanup.py

+        Returns:
+            Dictionary with counts of deleted records per table
+        """
+        cutoff_date = datetime.utcnow() - timedelta(days=self.retention_days)


Using datetime.utcnow() is deprecated in Python 3.12+ in favor of datetime.now(timezone.utc). This appears in the cleanup.py file (line 43). Consider updating to the recommended approach for future compatibility.

Copilot · 2025-11-24T22:42:10Z

src/meshcore_sidekick/database/engine.py

+        @event.listens_for(Engine, "connect")
+        def set_sqlite_pragma(dbapi_conn, connection_record):
+            cursor = dbapi_conn.cursor()
+            cursor.execute("PRAGMA foreign_keys=ON")
+            cursor.execute("PRAGMA journal_mode=WAL")  # Write-Ahead Logging for better concurrency
+            cursor.close()


The SQLite event listener is defined inside the initialize() method but decorated with @event.listens_for(Engine, "connect"), which registers it globally. This means each call to initialize() will register another listener, potentially causing duplicate pragma executions. Consider either: 1) defining this listener at module level outside the method, or 2) using @event.listens_for(self.engine, "connect") instead to bind to the specific engine instance.

Suggested change

@event.listens_for(Engine, "connect")

def set_sqlite_pragma(dbapi_conn, connection_record):

cursor = dbapi_conn.cursor()

cursor.execute("PRAGMA foreign_keys=ON")

cursor.execute("PRAGMA journal_mode=WAL") # Write-Ahead Logging for better concurrency

cursor.close()

def set_sqlite_pragma(dbapi_conn, connection_record):

cursor = dbapi_conn.cursor()

cursor.execute("PRAGMA foreign_keys=ON")

cursor.execute("PRAGMA journal_mode=WAL") # Write-Ahead Logging for better concurrency

cursor.close()

event.listen(self.engine, "connect", set_sqlite_pragma)

Copilot · 2025-11-24T22:42:10Z

src/meshcore_sidekick/utils/logging.py

+    def format(self, record: logging.LogRecord) -> str:
+        """Format log record as JSON."""
+        log_data: Dict[str, Any] = {
+            "timestamp": datetime.utcnow().isoformat() + "Z",


Using datetime.utcnow() is deprecated in Python 3.12+ in favor of datetime.now(timezone.utc). While the minimum Python version is 3.11, consider updating to the recommended approach for future compatibility: datetime.now(timezone.utc).replace(tzinfo=None) if naive datetime is required.

Copilot · 2025-11-24T22:42:11Z

src/meshcore_sidekick/subscriber/metrics.py

+
+
+# Global metrics collector instance
+_metrics: MetricsCollector = None


The global variable _metrics is initialized as None without a type annotation. For better type safety and consistency with the rest of the codebase, consider adding a type annotation: _metrics: Optional[MetricsCollector] = None.

Copilot · 2025-11-24T22:42:11Z

src/meshcore_sidekick/subscriber/event_handler.py

+                    for key, value in kwargs.items():
+                        if value is not None:
+                            setattr(node, key, value)
+                    node.last_seen = datetime.utcnow()


Using datetime.utcnow() is deprecated in Python 3.12+ in favor of datetime.now(timezone.utc). This appears in multiple locations in this file (lines 122, 129, 379). Consider updating to the recommended approach for future compatibility.

Copilot · 2025-11-24T22:42:11Z

src/meshcore_sidekick/meshcore/scenarios.py

+    for key, value in data.items():
+        if isinstance(value, str):
+            if value == "{{now}}":
+                result[key] = datetime.utcnow().isoformat() + "Z"


Using datetime.utcnow() is deprecated in Python 3.12+ in favor of datetime.now(timezone.utc). This appears in multiple locations in scenarios.py (line 32). Consider updating to the recommended approach for future compatibility.

Copilot · 2025-11-24T22:42:11Z

src/meshcore_sidekick/query.py

+        print(f"ACTIVITY TIMELINE (last {hours} hours)")
+        print("-" * 80)
+
+        cutoff = datetime.now() - timedelta(hours=hours)


The datetime.now() function is called without a timezone argument, which may use local time depending on the system. For consistency with other parts of the codebase that use UTC time, consider using datetime.now(timezone.utc) or documenting that local time is intentional here.

claude added 6 commits November 24, 2025 22:05

Add comprehensive implementation plan (PLAN.md)

9b21af5

Fix async call and add handlers for meshcore_py internal events

eab125a

- Await start_auto_message_fetching() coroutine properly - Add MESSAGES_WAITING and RX_LOG_DATA to handler map (informational) - Improve logging for unknown vs informational event types

Add NO_MORE_MSGS to informational events

aca9473

Copilot AI review requested due to automatic review settings November 24, 2025 22:38

jinglemansweep merged commit 7afecd1 into main Nov 24, 2025

Copilot started reviewing on behalf of jinglemansweep November 24, 2025 22:38 View session

Copilot finished reviewing on behalf of jinglemansweep November 24, 2025 22:40

Copilot AI reviewed Nov 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Plan MeshCore Python application with serial interface #1

Plan MeshCore Python application with serial interface #1

Uh oh!

jinglemansweep commented Nov 24, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Copilot AI Nov 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		# Global metrics collector instance
		_metrics: MetricsCollector = None

Plan MeshCore Python application with serial interface #1

Plan MeshCore Python application with serial interface #1

Uh oh!

Conversation

jinglemansweep commented Nov 24, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 24, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants