feat: Web Monitor Plugin - Track mentions and opportunities across platforms

[lane711]

Summary

A SonicJS plugin that monitors the web for relevant posts, discussions, and mentions where users can engage to promote their product/project. This is a marketing automation tool built into SonicJS.

Motivation

Currently, finding relevant discussions to participate in (Reddit, HN, DEV.to, etc.) requires manual searching. A web monitor plugin would:

• Automate discovery of relevant discussions
• Send timely notifications when opportunities arise
• Track engagement history
• Help SonicJS users grow their projects organically

Supported Platforms

Platform	API Method	Notes
Reddit	JSON API (no auth needed)	/r/{subreddit}/new.json
Hacker News	Firebase API	hacker-news.firebaseio.com
DEV.to	REST API	dev.to/api/articles
Stack Overflow	Stack Exchange API	Requires API key for higher limits
Twitter/X	API v2	Requires paid API access
GitHub Discussions	GraphQL API	Requires token
Lobsters	RSS/API	lobste.rs/newest.json
Product Hunt	GraphQL API	For launch monitoring

Plugin Architecture

File Structure

```
packages/core/src/plugins/core-plugins/web-monitor/
├── index.ts # PluginBuilder setup
├── services/
│ ├── monitor-service.ts # Core monitoring orchestration
│ ├── keyword-matcher.ts # Relevance scoring
│ └── adapters/
│ ├── base-adapter.ts # Abstract adapter interface
│ ├── reddit-adapter.ts
│ ├── hackernews-adapter.ts
│ ├── devto-adapter.ts
│ └── stackoverflow-adapter.ts
├── routes/
│ ├── admin.ts # Admin UI endpoints
│ └── api.ts # REST API
├── models/
│ ├── monitor-config.ts # Configuration schema
│ ├── found-post.ts # Discovered posts
│ └── engagement.ts # Tracking responses
├── admin/
│ └── components/
│ ├── dashboard.tsx # Main dashboard
│ ├── config-panel.tsx # Settings
│ └── post-list.tsx # Found posts view
└── scheduled/
└── monitor-task.ts # Background job logic
```

Database Schema

```sql
-- Monitor configurations
CREATE TABLE web_monitor_configs (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
platform TEXT NOT NULL, -- 'reddit', 'hackernews', 'devto', etc.
target TEXT NOT NULL, -- subreddit, tag, search query
keywords TEXT NOT NULL, -- JSON array
enabled INTEGER DEFAULT 1,
check_interval INTEGER DEFAULT 3600, -- seconds
last_checked_at INTEGER,
created_at INTEGER,
updated_at INTEGER
);

-- Discovered posts
CREATE TABLE web_monitor_posts (
id TEXT PRIMARY KEY,
config_id TEXT REFERENCES web_monitor_configs(id),
platform TEXT NOT NULL,
external_id TEXT NOT NULL, -- Platform-specific ID
title TEXT NOT NULL,
url TEXT NOT NULL,
author TEXT,
score INTEGER DEFAULT 0,
comment_count INTEGER DEFAULT 0,
content_preview TEXT,
relevance_score REAL, -- 0.0 - 1.0
status TEXT DEFAULT 'new', -- 'new', 'viewed', 'responded', 'ignored'
found_at INTEGER,
external_created_at INTEGER,
UNIQUE(platform, external_id)
);

-- Engagement tracking
CREATE TABLE web_monitor_engagements (
id TEXT PRIMARY KEY,
post_id TEXT REFERENCES web_monitor_posts(id),
response_url TEXT,
notes TEXT,
created_at INTEGER
);
```

Platform Adapter Interface

```typescript
interface PlatformAdapter {
platform: string;

// Fetch new posts from the platform
fetchPosts(config: MonitorConfig): Promise<RawPost[]>;

// Platform-specific rate limiting
getRateLimitDelay(): number;

// Normalize platform-specific data
normalizePost(raw: any): NormalizedPost;
}

interface NormalizedPost {
externalId: string;
title: string;
url: string;
author: string;
score: number;
commentCount: number;
contentPreview: string;
createdAt: Date;
}
```

Keyword Matching & Relevance Scoring

```typescript
interface KeywordMatcher {
// Calculate relevance score based on:
// - Keyword matches in title (weighted higher)
// - Keyword matches in body
// - Negative keywords (exclusions)
// - Recency bonus
// - Engagement potential (low comments = better opportunity)
calculateRelevance(post: NormalizedPost, config: MonitorConfig): number;
}
```

Admin Dashboard Features

Configuration Panel

• Add/edit/delete monitor configs
• Select platform and target (subreddit, HN front page, DEV.to tag, etc.)
• Define keywords with AND/OR logic
• Set negative keywords (exclusions)
• Configure check frequency
• Enable/disable individual monitors

Posts Dashboard

• List of found posts sorted by relevance
• Filter by platform, status, date range
• Quick actions: Mark as responded, Ignore, Open in new tab
• Bulk actions for multiple posts

Analytics

• Posts found over time
• Response rate
• Top performing keywords
• Platform breakdown

Notifications

• Email digest (configurable frequency)
• In-app notifications
• Webhook support for external integrations

API Endpoints

```
GET /api/web-monitor/configs # List all configs
POST /api/web-monitor/configs # Create config
PUT /api/web-monitor/configs/:id # Update config
DELETE /api/web-monitor/configs/:id # Delete config

GET /api/web-monitor/posts # List found posts (with filters)
PUT /api/web-monitor/posts/:id/status # Update post status
POST /api/web-monitor/posts/:id/engage # Log engagement

POST /api/web-monitor/trigger # Manually trigger check
GET /api/web-monitor/stats # Analytics data
```

Scheduled Task Integration

The plugin registers with SonicJS's scheduled task system:

```typescript
PluginBuilder.create({
name: 'web-monitor',
version: '1.0.0'
})
.addScheduledTask('web-monitor-check', {
cron: '0 * * * *', // Default: hourly
handler: async (ctx) => {
const configs = await getEnabledConfigs(ctx.db);
for (const config of configs) {
if (shouldCheck(config)) {
await runMonitor(config, ctx);
}
}
}
})
.build()
```

Configuration Example

```yaml

Example monitor configs via admin UI or API

monitors:

• name: "Reddit Headless CMS"
  platform: reddit
  targets:

  • webdev
  • nextjs
  • node
  • javascript
    keywords:
  • "headless CMS"
  • "Strapi alternative"
  • "lightweight CMS"
  • "API framework"
    negative_keywords:
  • "WordPress"
  • "hiring"
    check_interval: 3600 # 1 hour

• name: "Hacker News Show HN"
  platform: hackernews
  targets:

  • show # Show HN
  • ask # Ask HN
    keywords:
  • "CMS"
  • "Cloudflare Workers"
  • "headless"
    check_interval: 1800 # 30 minutes

• name: "DEV.to JavaScript"
  platform: devto
  targets:

  • javascript
  • webdev
  • node
    keywords:
  • "CMS comparison"
  • "best headless CMS"
    check_interval: 7200 # 2 hours
    ```

Email Notification Template

```
Subject: 🔥 5 New Opportunities Found - Web Monitor

Found 5 new posts matching your monitors:

High Relevance (Score > 0.8)

1. [Reddit] "Looking for lightweight headless CMS for my project"
   r/webdev • 12 comments • Score: 45
   https://reddit.com/r/webdev/...

2. [HN] "Ask HN: Best CMS for Cloudflare Workers?"
   Ask HN • 8 comments • Score: 23
   https://news.ycombinator.com/...

Medium Relevance (Score 0.5-0.8)

3. [DEV.to] "Comparing Modern Headless CMS Options"
   #javascript • 5 comments
   https://dev.to/...

---

Configure monitors: https://your-site.com/admin/web-monitor
```

Implementation Phases

Phase 1: Core Infrastructure

• Plugin scaffold with PluginBuilder
• Database models and migrations
• Base adapter interface
• Reddit adapter (first platform)
• Keyword matcher with basic scoring
• Scheduled task registration

Phase 2: Additional Platforms

• Hacker News adapter
• DEV.to adapter
• Stack Overflow adapter

Phase 3: Admin UI

• Configuration management
• Posts dashboard
• Status management

Phase 4: Notifications

• Email integration (via email plugin)
• Configurable digest frequency
• Webhook support

Phase 5: Analytics & Polish

• Analytics dashboard
• Engagement tracking
• AI-powered response suggestions (future)

Related Work

• Existing standalone Reddit monitor: tools/reddit-monitor/
• SonicJS scheduler service: packages/core/src/plugins/core-plugins/workflow-plugin/services/scheduler.ts
• Plugin system: packages/core/src/plugins/sdk/plugin-builder.ts

Open Questions

1. Should this be a core plugin or a separate installable package?
2. Rate limiting strategy across platforms?
3. Should AI response drafting be part of this plugin or separate?
4. Support for authenticated APIs (Twitter, GitHub) vs. public only?

---

Labels: enhancement, plugin, marketing