Add dashboard for explore data #5
Conversation
- update minio endpoint url in: - readme - dashboard/data/prepare_data.ipynb - dashboard/streamlit-css.py - notebooks/footprints/footprints.ipynb - notebooks/how_to.ipynb - notebooks/modified-gravity-tests/01_lastberu_cosmo_ground.ipynb - notebooks/proposals/proposals.ipynb
- remove dashboard directory and all its contents - this includes streamlit app, data files, and requirements
- scaffold basic React/TypeScript application with Vite - configure ESLint for code quality and formatting - set up basic UI layout with MUI components - add React Query for data fetching - implement Recoil for state management - create initial components: App, DataTables, SkyMap, Gallery, FiltersDrawer, ObjectsTable - implement data loading from static JSON files - implement basic filtering and selection of objects - add basic sky map visualization with object positions - add cutout image gallery with placeholder images - implement basic UI theme and styling - add .gitignore file to exclude node_modules and other unnecessary files - add public directory with placeholder data and images - add types file for data structures
- read data from minio - process data - save data to public/data directory
- configures ci to automatically build and deploy the react dashboard to github pages - sets up python and node.js environments, installs dependencies, prepares data from minio, and builds the project - deploys the built dashboard to github pages on pushes to the main or streamlit branches
- install pyarrow and fastparquet for enhanced data processing capabilities
WalkthroughIntroduces a React + TypeScript dashboard under dashboard/ with data loading, filtering, sky map, and cutout gallery. Adds data preparation script pulling artifacts from MinIO. Sets up Vite, ESLint, theme, and CI to build/deploy to GitHub Pages. Adds a devcontainer. Updates notebook MinIO endpoints and a README link. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Dev as Developer
participant GH as GitHub
participant WF as Actions: deploy.yml
participant MinIO as MinIO (secrets)
participant Pages as GitHub Pages
Dev->>GH: push to dashboard/**
GH-->>WF: trigger build job
WF->>WF: setup Python/Node
WF->>MinIO: run prepare_data.py (env secrets)
MinIO-->>WF: data files (JSON)
WF->>WF: npm ci && vite build (BASE_PATH)
WF->>Pages: upload-pages-artifact
Note over WF,Pages: On main/streamlit only
GH-->>WF: trigger deploy job
WF->>Pages: deploy artifact
Pages-->>Dev: page URL output
sequenceDiagram
autonumber
actor User
participant App as Dashboard App
participant RQ as React Query
participant FS as /public/data/*.json
participant MinIO as MinIO Gateway
User->>App: open dashboard
App->>RQ: fetch db, consolidated, dictionary, cutouts
RQ->>FS: GET /data/database.json
RQ->>FS: GET /data/consolidated_database.json
RQ->>FS: GET /data/dictionary.json
RQ->>FS: GET /data/cutouts.json
FS-->>RQ: JSON datasets
RQ-->>App: resolved data
App->>User: render tables, sky map, filters
User->>App: open Cutouts/Gallery
App->>MinIO: resolve cutout URL (ngrok-aware)
MinIO-->>App: image or blob URL
App->>User: show thumbnails/modal
sequenceDiagram
autonumber
participant Script as prepare_data.py
participant MinIO as MinIO
participant FS as dashboard/public/data/
Script->>MinIO: download Processed_Cutouts.parquet
Script->>MinIO: download FITS.parquet
Script->>Script: merge/normalize cutouts (bands, paths)
Script->>FS: write cutouts.json
Script->>MinIO: download Consolidated_Data.csv
Script->>Script: type normalize
Script->>FS: write consolidated_database.json
Script->>MinIO: download Database.csv
Script->>Script: build dictionary, expand multi-values
Script->>FS: write dictionary.json
Script->>FS: write database.json
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Poem
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- trigger deployment workflow only when changes occur in the dashboard directory
- introduce features, quick start guide, data files, theming, and scripts
oliveirara
had a problem deploying
to
github-pages
GitHub Actions
Failure
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 29
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
notebooks/how_to.ipynb (1)
64-72: Remove hardcoded MinIO credentials from all notebooks
HardcodedMINIO_ENDPOINT_URL,ACCESS_KEY, andSECRET_KEYare present in:
- notebooks/how_to.ipynb (lines 64–72)
- notebooks/proposals/proposals.ipynb (lines 38–44)
- notebooks/modified-gravity-tests/01_LaStBeRu_cosmo_ground.ipynb (lines 85–91)
- notebooks/footprints/footprints.ipynb (lines 54–60)
Load these via
os.getenv("MINIO_ENDPOINT_URL") os.getenv("MINIO_ACCESS_KEY") os.getenv("MINIO_SECRET_KEY")and raise an error if any are unset. Rotate exposed credentials and audit access.
rg -n -C1 -S '(MINIO_.*(KEY|SECRET)|slcomp)' notebooks/**/*.ipynbnotebooks/proposals/proposals.ipynb (3)
38-46: Remove hard-coded MinIO endpoint and credentials (secrets in repo).Endpoint, access key, and secret are committed in plain text. Move to environment variables and fail fast if missing.
Apply this diff:
- MINIO_ENDPOINT_URL = "nonarithmetically-undeliberating-janelle.ngrok-free.app" - ACCESS_KEY = "slcomp" - SECRET_KEY = "slcomp@data" - client = Minio( - MINIO_ENDPOINT_URL, - access_key=ACCESS_KEY, - secret_key=SECRET_KEY, - secure=True, - ) + MINIO_ENDPOINT_URL = os.environ["MINIO_ENDPOINT_URL"] + ACCESS_KEY = os.environ["MINIO_ACCESS_KEY"] + SECRET_KEY = os.environ["MINIO_SECRET_KEY"] + MINIO_SECURE = os.getenv("MINIO_SECURE", "1") not in {"0", "false", "False"} + client = Minio( + MINIO_ENDPOINT_URL, + access_key=ACCESS_KEY, + secret_key=SECRET_KEY, + secure=MINIO_SECURE, + )
534-536: Fix RA range logic (6h–20h → degrees).Current expression uses
8*20(160°), not20*15(300°). This incorrectly filters RA.Apply this diff:
- Database = Database.query( - "(RA>=6*15 and RA <=8*20) and (DEC >=-75 and DEC <= 15)" - ).reset_index(drop=True) + Database = Database.query( + "(RA >= 6*15 and RA <= 20*15) and (DEC >= -75 and DEC <= 15)" + ).reset_index(drop=True)
727-731: NaN handling in filter doesn’t work withisin([np.nan, ...]).
NaNnever matches viaisin. Useisna()for missing System_Type.Apply this diff:
- data_with_match = data_with_match[ - data_with_match.System_Type.isin([np.nan, "Single Lens Galaxy"]) - ].reset_index(drop=True) + data_with_match = data_with_match[ + data_with_match["System_Type"].isna() | (data_with_match["System_Type"] == "Single Lens Galaxy") + ].reset_index(drop=True)
🧹 Nitpick comments (62)
dashboard/src/components/VirtualizedList.tsx (7)
3-9: Ensure stable React keys; optionally add keyExtractor.As written, keys depend on renderItem providing them. To avoid “Each child should have a unique key” warnings, accept an optional keyExtractor and apply it where items are rendered.
interface VirtualizedListProps<T = unknown> { items: T[]; renderItem: (item: T, index: number) => React.ReactNode; itemHeight: number; containerHeight: number; overscan?: number; + keyExtractor?: (item: T, index: number) => React.Key; } @@ - overscan = 5 + overscan = 5, + keyExtractor }: VirtualizedListProps<T>) => { @@ - {items.map((item, index) => renderItem(item, index))} + {items.map((item, index) => { + const node = renderItem(item, index); + const key = keyExtractor ? keyExtractor(item, index) : index; + return React.isValidElement(node) + ? React.cloneElement(node, { key }) + : <div key={key}>{node}</div>; + })} @@ - {visibleItems.map(({ item, index }) => - renderItem(item, index) - )} + {visibleItems.map(({ item, index }) => { + const node = renderItem(item, index); + const key = keyExtractor ? keyExtractor(item, index) : index; + return React.isValidElement(node) + ? React.cloneElement(node, { key }) + : <div key={key}>{node}</div>; + })}Also applies to: 12-17, 50-52, 75-77
27-29: Clamp overscan to a non-negative integer.Guards against negative or fractional overscan values.
- const start = Math.max(0, startIndex - overscan); - const end = Math.min(items.length - 1, endIndex + overscan); + const safeOverscan = Math.max(0, Math.floor(overscan)); + const start = Math.max(0, startIndex - safeOverscan); + const end = Math.min(items.length - 1, endIndex + safeOverscan);
41-43: Reduce scroll jank with rAF batching.Scrolling can fire dozens of events per frame; batch setState with requestAnimationFrame.
+ const rafRef = React.useRef<number | null>(null); - const handleScroll = React.useCallback((e: React.UIEvent<HTMLDivElement>) => { - setScrollTop(e.currentTarget.scrollTop); - }, []); + const handleScroll = React.useCallback((e: React.UIEvent<HTMLDivElement>) => { + const st = e.currentTarget.scrollTop; + if (rafRef.current !== null) cancelAnimationFrame(rafRef.current); + rafRef.current = requestAnimationFrame(() => setScrollTop(st)); + }, []); + + React.useEffect(() => { + return () => { + if (rafRef.current !== null) cancelAnimationFrame(rafRef.current); + }; + }, []);
45-47: Clamp scrollTop when data or sizes shrink to avoid stale window.If totalHeight decreases below current scrollTop, visible window can be off until the next user scroll.
const totalHeight = items.length * itemHeight; + React.useEffect(() => { + const maxScroll = Math.max(0, totalHeight - containerHeight); + setScrollTop((s) => Math.min(s, maxScroll)); + }, [totalHeight, containerHeight]);
61-62: Remove unused positioning on outer container.position: 'relative' isn’t needed here; the absolutely positioned layer is relative to its own parent container below.
- overflowY: 'auto', - position: 'relative' + overflowY: 'auto'
67-73: Hint GPU acceleration for smoother scrolling.Small win: willChange helps browsers plan for frequent transforms.
style={{ transform: `translateY(${visibleIndices.start * itemHeight}px)`, position: 'absolute', top: 0, left: 0, - right: 0 + right: 0, + willChange: 'transform' }}
47-54: Make the “20 items” heuristic configurable or data-driven.Consider a prop like minVirtualizeCount = 20 or auto-enable when items.length * itemHeight > containerHeight.
dashboard/public/data/.gitkeep (1)
1-2: Make the generator path explicit (minor clarity).Reference the exact script path to avoid ambiguity when browsing the repo.
-# Files are generated by prepare_data.py which fetches data from MinIO +# Files are generated by dashboard/prepare_data.py which fetches data from MinIOnotebooks/how_to.ipynb (7)
23-27: Importdisplayexplicitly for portability.Fixes the Ruff F821 and makes the notebook runnable after export.
import io +from IPython.display import display
333-338: MinIO client: prefer streaming/read() over.data(API compatibility + memory).
.dataisn’t guaranteed across minio client versions and loads entire objects into memory. Useread()or stream.Example refactor (apply similarly to other occurrences):
-Database_object = client.get_object("slcomp", "Data/Database.csv").data -Database = pd.read_csv(io.StringIO(Database_object.decode("utf-8")), low_memory=False, dtype=object) +resp = client.get_object("slcomp", "Data/Database.csv") +try: + Database = pd.read_csv(io.StringIO(resp.read().decode("utf-8")), low_memory=False, dtype=object) +finally: + resp.close() + resp.release_conn()Also applies to: 752-761, 1098-1107, 1436-1445, 1691-1694, 2008-2015
1882-1886: Ensure directories exist beforefget_objectwrites (robustness).Current code may fail if nested directories aren’t present.
-[ - client.fget_object("slcomp", "Cutouts/" + file_path, file_path) - for file_path in Cutouts_Catalog.query('JNAME=="J114833.1+193003.2"').file_path -] +for file_path in Cutouts_Catalog.query('JNAME=="J114833.1+193003.2"').file_path: + os.makedirs(os.path.dirname(file_path), exist_ok=True) + client.fget_object("slcomp", f"Cutouts/{file_path}", file_path)
2260-2266: Same: create parent dirs before saving processed cutouts.-[ - client.fget_object("slcomp", "Cutouts/" + file_path, file_path) - for file_path in Processed_Cutouts_Catalog.query('JNAME=="J114833.1+193003.2"').file_path -] +for file_path in Processed_Cutouts_Catalog.query('JNAME=="J114833.1+193003.2"').file_path: + os.makedirs(os.path.dirname(file_path), exist_ok=True) + client.fget_object("slcomp", f"Cutouts/{file_path}", file_path)
2978-2980: Boolean filter should not compare to string literal.If
is_rgbis boolean, compare as a boolean or filter directly.-len(Processed_Cutouts_Catalog.query('is_rgb=="True"')) +Processed_Cutouts_Catalog["is_rgb"].sum() # or: len(Processed_Cutouts_Catalog[Processed_Cutouts_Catalog.is_rgb])
2403-2404: Avoidnp.hstackon mixed list/scalar cells; useexplode.
hstackrisks splitting strings into characters if a cell isn’t a list.-pd.DataFrame(np.hstack(Database.Lens_Type), columns=["Lens_Type"]).value_counts() +tmp = Database[["Lens_Type"]].copy() +tmp["Lens_Type"] = tmp["Lens_Type"].apply(lambda v: v if isinstance(v, list) else [v]) +tmp.explode("Lens_Type")["Lens_Type"].value_counts() -pd.DataFrame(np.hstack(Database.Source_Type), columns=["Source_Type"]).value_counts() +tmp = Database[["Source_Type"]].copy() +tmp["Source_Type"] = tmp["Source_Type"].apply(lambda v: v if isinstance(v, list) else [v]) +tmp.explode("Source_Type")["Source_Type"].value_counts()Also applies to: 2437-2438
1-3045: Strip heavy notebook outputs before committing (repo hygiene).Outputs bloat diffs and repo size; keep notebooks lightweight or use nbdime/LFS.
README.md (1)
52-55: Endpoint drift: README MinIO link differs from notebook endpoint.README uses ruggedly-quaky-maricruz…, notebooks use nonarithmetically-undeliberating-janelle… Consolidate to a single canonical URL or document that endpoints are ephemeral.
Suggestion:
- Define MINIO_ENDPOINT_URL once (e.g., in an .env example) and reference it in README and notebooks.
- Consider a stable subdomain (e.g., minio.slcomp.org) that CNAMEs to the current tunnel.
dashboard/.gitignore (2)
15-15: Typo: TypeScript build info pattern.The file is typically
tsconfig.tsbuildinfo. Use a broader pattern.-tsbuildinfo.tsbuildinfo +*.tsbuildinfo
12-12: Consider ignoring parquet artifacts if added later.Future-proof the generated data rule.
public/data/*.json +public/data/*.parquetnotebooks/footprints/footprints.ipynb (2)
54-54: Make endpoint configurable; avoid hardcoding ephemeral ngrok URLBind MINIO_ENDPOINT_URL from env (optionally support a local default) to avoid future breakage and simplify CI/CD and local dev.
-import os +import os -MINIO_ENDPOINT_URL = "nonarithmetically-undeliberating-janelle.ngrok-free.app" +MINIO_ENDPOINT_URL = os.getenv( + "MINIO_ENDPOINT_URL", + # Optional local/dev fallback (adjust as appropriate) + "nonarithmetically-undeliberating-janelle.ngrok-free.app" +)Consider centralizing this in a small config module used by notebooks and dashboard/prepare_data.py, and document required env vars in README and GH Actions (use repository Secrets).
57-57: Be explicit about TLS when initializing Minio clientGiven the ngrok endpoint is HTTPS, pass secure=True explicitly to avoid ambiguity across client versions.
client = Minio( MINIO_ENDPOINT_URL, access_key=ACCESS_KEY, secret_key=SECRET_KEY, + secure=True, )notebooks/proposals/proposals.ipynb (1)
996-1058: Close image resources and simplify subplot loop.Use context managers to avoid file/resource leaks and potential warnings.
Apply this diff:
- data_io = io.BytesIO( - client.get_object("slcomp", "Cutouts/" + data.iloc[k].file_path).data - ) - # im = plt.imread(data_io) - im = Image.open(data_io) - im.load() - ax = plt.subplot(gs[i, j]) + data_io = io.BytesIO( + client.get_object("slcomp", "Cutouts/" + data.iloc[k].file_path).data + ) + ax = plt.subplot(gs[i, j]) + with Image.open(data_io) as im: + im.load() if data.iloc[k].survey == "CS82": - ax.imshow(im, interpolation="lanczos", aspect="auto", cmap="gray") + ax.imshow(im, interpolation="lanczos", aspect="auto", cmap="gray") else: - ax.imshow(im, interpolation="lanczos", aspect="auto") + ax.imshow(im, interpolation="lanczos", aspect="auto") ax.set_xticklabels([]) ax.set_yticklabels([]) ax.set_xticks([]) ax.set_yticks([]) @@ - im = Image.open(f"mosaics/{jname}.png") - im2 = im.crop(im.getbbox()) - im2.save(f"mosaics/{jname}.png") + with Image.open(f"mosaics/{jname}.png") as _im: + im2 = _im.crop(_im.getbbox()) + im2.save(f"mosaics/{jname}.png")dashboard/tsconfig.json (1)
2-21: Tighten TS config for Vite/React.Add Vite client types and convenient import aliasing.
Apply this diff:
"compilerOptions": { @@ - "jsx": "react-jsx" + "jsx": "react-jsx", + "types": ["vite/client"], + "baseUrl": ".", + "paths": { + "@/*": ["src/*"] + } },dashboard/src/hooks/useDebounce.ts (1)
6-14: Guard zero/negative delays.Immediate update when
delay <= 0avoids unnecessary timers.Apply this diff:
useEffect(() => { - const handler = setTimeout(() => { - setDebouncedValue(value); - }, delay); + if (delay <= 0) { + setDebouncedValue(value); + return; + } + const handler = setTimeout(() => { + setDebouncedValue(value); + }, delay);dashboard/package.json (2)
6-11: Add a dedicated typecheck and fix script.Keeps build fast and enables quick autofixes.
Apply this diff:
"scripts": { "dev": "vite", - "build": "tsc && vite build", + "build": "tsc --noEmit && vite build", "preview": "vite preview", - "lint": "eslint src --ext .ts,.tsx" + "lint": "eslint src --ext .ts,.tsx", + "lint:fix": "eslint src --ext .ts,.tsx --fix", + "typecheck": "tsc --noEmit" },
1-5: Add engines for predictable CI/dev behavior.Pin Node to the version Vite/TS expect.
Apply this diff:
{ "name": "slcomp-dashboard", "version": "0.1.0", "private": true, "type": "module", + "engines": { + "node": ">=18.18.0" + },.devcontainer/devcontainer.json (1)
20-20: Security note: disabling CORS/XSRF.Okay for local dev, but avoid committing insecure defaults; add a comment or guard via env flag.
Apply this diff:
- "updateContentCommand": "[ -f packages.txt ] && sudo apt update && sudo apt upgrade -y && sudo xargs apt install -y <packages.txt; [ -f requirements.txt ] && pip3 install --user -r requirements.txt; pip3 install --user streamlit; echo '✅ Packages installed and Requirements met'", + "updateContentCommand": "[ -f packages.txt ] && sudo apt update && sudo apt upgrade -y && sudo xargs apt install -y <packages.txt; [ -f requirements.txt ] && pip3 install --user -r requirements.txt; pip3 install --user streamlit; echo '✅ Packages installed and Requirements met' # dev-only",dashboard/vite.config.ts (2)
4-7: Normalize BASE_PATH for safety.Ensures leading/trailing slashes so GH Pages routing is correct regardless of input.
Apply this diff:
-// For GitHub Pages deployment: set BASE_PATH environment variable to your repo name -// Example: export BASE_PATH=/slcomp/ before building -const basePath = process.env.BASE_PATH || '/'; +// For GitHub Pages deployment: set BASE_PATH to your repo name (with or without slashes) +const rawBase = process.env.BASE_PATH || '/'; +const basePath = (() => { + let b = rawBase.trim(); + if (!b.startsWith('/')) b = '/' + b; + if (!b.endsWith('/')) b = b + '/'; + return b; +})();
23-31: Consider strict port to avoid collision during dev.Small DX improvement.
Apply this diff:
server: { - port: 5173, + port: 5173, + strictPort: true,dashboard/src/types.ts (3)
1-4: Avoidanyin data records.Prefer
unknownand narrow at use sites to keep type-safety.Apply this diff:
export interface DataRecord { JNAME: string; - [key: string]: any; // dynamic attributes + [key: string]: unknown; // dynamic attributes }
6-9: Same here for ConsolidatedRecord.Apply this diff:
export interface ConsolidatedRecord { JNAME: string; - [key: string]: any; + [key: string]: unknown; }
18-21: Avoidanyin DictionaryEntry.Keeps downstream code safer.
Apply this diff:
export interface DictionaryEntry { JNAME?: string[] | string; - [key: string]: any; + [key: string]: unknown; }dashboard/eslint.config.js (2)
3-3: Ignore pattern may miss other build artifacts.If you want to fully avoid scanning generated/static files, consider also ignoring coverage/, .vite/, and public/data/*.json (though files: only targets src/, so this is optional).
21-26: Consider enabling unused-disable reporting.Helps keep config tidy by flagging stale // eslint-disable lines.
{ files: ['src/**/*.{ts,tsx}'], + linterOptions: { reportUnusedDisableDirectives: true }, languageOptions: {dashboard/prepare_data.py (2)
24-27: Make output path robust to working directory.Resolve DATA_PATH relative to this script so CI/local runs are consistent.
-# Data will be saved to public/data directory -DATA_PATH = "public/data" +# Data will be saved to dashboard/public/data (relative to this file) +DATA_PATH = str((pathlib.Path(__file__).parent / "public" / "data").resolve())
74-76: Consider ordered categories for band.If band sorting matters in the UI, set ordered=True to enforce intended order.
-cutouts.band = pd.Categorical( - cutouts.band, categories=["u", "g", "r", "i", "z", "y", "trilogy", "lsb"] -) +cutouts.band = pd.Categorical( + cutouts.band, + categories=["u", "g", "r", "i", "z", "y", "trilogy", "lsb"], + ordered=True, +).github/workflows/deploy.yml (2)
23-23: Clean trailing spaces to satisfy linters.Minor formatting to appease YAML linters and keep diffs clean.
- uses: actions/checkout@v4 - + uses: actions/checkout@v4 @@ - + @@ - + @@ - + @@ - + @@ - + @@ - +Also applies to: 28-28, 35-35, 40-40, 49-49, 54-54, 57-57, 63-63
36-54: Optional: set working-directory instead of cd for clarity.Reduces repetition and chances of path mistakes.
- - name: Install Python dependencies - run: | - cd dashboard - pip install pandas numpy minio pyarrow fastparquet + - name: Install Python dependencies + working-directory: dashboard + run: pip install pandas numpy minio pyarrow fastparquet @@ - - name: Prepare data from MinIO + - name: Prepare data from MinIO env: MINIO_ENDPOINT_URL: ${{ secrets.MINIO_ENDPOINT_URL }} MINIO_ACCESS_KEY: ${{ secrets.MINIO_ACCESS_KEY }} MINIO_SECRET_KEY: ${{ secrets.MINIO_SECRET_KEY }} - run: | - cd dashboard - python prepare_data.py + working-directory: dashboard + run: python prepare_data.py @@ - - name: Install dependencies - run: | - cd dashboard - npm ci + - name: Install dependencies + working-directory: dashboard + run: npm ci @@ - - name: Build - run: | - cd dashboard - export BASE_PATH=/slcomp/ - npm run build + - name: Build + working-directory: dashboard + env: + BASE_PATH: /slcomp/ + run: npm run build @@ - - name: Upload artifact + - name: Upload artifact uses: actions/upload-pages-artifact@v3 with: path: 'dashboard/dist'dashboard/src/main.tsx (2)
8-24: Harden ErrorBoundary types and add a lightweight resetTighten types and provide a simple “try again” without a full reload.
-class ErrorBoundary extends React.Component<{children: React.ReactNode}, {error: any}> { - constructor(props:any){ +class ErrorBoundary extends React.Component<React.PropsWithChildren, { error: Error | null }> { + constructor(props: React.PropsWithChildren){ super(props); this.state = { error: null }; } - static getDerivedStateFromError(error:any){ return { error }; } - componentDidCatch(err:any, info:any){ console.error('App crashed:', err, info); } + static getDerivedStateFromError(error: Error){ return { error }; } + componentDidCatch(err: Error, info: React.ErrorInfo){ console.error('App crashed:', err, info); } render(){ if(this.state.error){ return <div style={{padding:24,fontFamily:'monospace',color:'#eee'}}> <h2>Application Error</h2> - <pre>{String(this.state.error)}</pre> - <p>Check console for stack trace.</p> + <pre>{String(this.state.error.message || this.state.error)}</pre> + <p>Check console for stack trace.</p> + <button onClick={() => this.setState({ error: null })} style={{marginTop:12}}>Try again</button> </div>; } return this.props.children; } }
37-46: Guard #root existence and wrap in StrictModePrevents a cryptic null deref and enables extra checks in dev.
-ReactDOM.createRoot(document.getElementById('root') as HTMLElement).render( - <QueryClientProvider client={qc}> - <ThemeProvider theme={darkAquaTheme}> - <CssBaseline /> - <ErrorBoundary> - <App /> - </ErrorBoundary> - </ThemeProvider> - </QueryClientProvider> -); +const rootEl = document.getElementById('root'); +if (!rootEl) throw new Error('Root element #root not found'); +ReactDOM.createRoot(rootEl).render( + <React.StrictMode> + <QueryClientProvider client={qc}> + <ThemeProvider theme={darkAquaTheme}> + <CssBaseline /> + <ErrorBoundary> + <App /> + </ErrorBoundary> + </ThemeProvider> + </QueryClientProvider> + </React.StrictMode> +);dashboard/README.md (2)
18-24: Fix markdownlint MD058: add blank lines around the tableAdds the required blank lines for table blocks.
-## Data Files (place in `public/data/`) -| File | Purpose | +## Data Files (place in `public/data/`) + +| File | Purpose | ... -| `dictionary.json` | Reference dictionary | +| `dictionary.json` | Reference dictionary | +
11-15: Optional: call out data preparation step to avoid 404sIf data is generated by prepare_data.py in CI, mention how to populate public/data locally.
I can propose a short “Prepare data locally” subsection if you confirm the intended local workflow.
dashboard/src/env.d.ts (1)
3-12: Add missing type for VITE_MINIO_SCHEME (if referenced)AI summary mentions usage; add it (optional) to avoid TS errors if used.
interface ImportMetaEnv { readonly VITE_MINIO_ENDPOINT: string; + readonly VITE_MINIO_SCHEME?: 'http' | 'https'; readonly VITE_MINIO_ACCESS_KEY: string; readonly VITE_MINIO_SECRET_KEY: string; readonly VITE_MINIO_BUCKET: string; }If you remove secrets per above, keep only ENDPOINT/BUCKET (and optional SCHEME).
dashboard/src/theme.ts (2)
1-1: Avoidas anyon theme.shadows; type it asShadowsKeeps TS safety while satisfying MUI’s 25-shadow tuple.
-import { createTheme, alpha } from '@mui/material/styles'; +import { createTheme, alpha } from '@mui/material/styles'; +import type { Shadows } from '@mui/material/styles'; @@ -const customShadows = [ +const customShadows = [ 'none', '0 2px 4px -2px rgba(0,0,0,0.6), 0 0 0 1px rgba(255,255,255,0.02)', '0 4px 12px -2px rgba(0,0,0,0.65), 0 0 0 1px rgba(255,255,255,0.02)', '0 6px 18px -4px rgba(0,0,0,0.65), 0 0 0 1px rgba(255,255,255,0.025)', '0 10px 28px -6px rgba(0,0,0,0.7), 0 0 0 1px rgba(255,255,255,0.025)', ...Array(20).fill('0 0 0 1px rgba(0,0,0,0.4)') as string[] -] as const; +] as unknown as Shadows; @@ - shadows: customShadows as any, + shadows: customShadows,Also applies to: 3-10, 38-38
42-52: Add Firefox scrollbar styling (WebKit-only at the moment)Small cross-browser polish.
body: { backgroundColor: '#03060a', backgroundImage: [ 'radial-gradient(circle at 20% 15%, rgba(0,200,255,0.09), rgba(0,0,0,0) 45%)', 'radial-gradient(circle at 80% 75%, rgba(120,60,255,0.10), rgba(0,0,0,0) 50%)', 'linear-gradient(135deg, #020409 0%, #040b14 45%, #020409 100%)' ].join(','), backgroundAttachment: 'fixed', overscrollBehavior: 'none', - WebkitFontSmoothing: 'antialiased' + WebkitFontSmoothing: 'antialiased', + scrollbarWidth: 'thin', + scrollbarColor: '#145566 #03060a' },dashboard/src/components/DataTables.tsx (1)
1-1: Memoize column definitions to avoid DataGrid state resets and extra work.Apply:
-import React from 'react'; +import React, { useMemo } from 'react';And:
- const dbCols = autoCols(database); - const consCols = autoCols(consolidated); + const dbCols = useMemo(() => autoCols(database), [database]); + const consCols = useMemo(() => autoCols(consolidated), [consolidated]);dashboard/src/components/Gallery.tsx (3)
90-96: Don’t open modal for missing images; add alt text.Improves UX and accessibility.
Apply:
- <Grid item key={img.key} xs={6} sm={4} md={3} lg={2} onClick={()=> openModal(idx)} style={{ cursor: img.url ? 'pointer':'default' }}> + <Grid item key={img.key} xs={6} sm={4} md={3} lg={2} onClick={()=> img.url && openModal(idx)} style={{ cursor: img.url ? 'pointer':'default' }}> @@ - {img.url ? (<img src={img.url} style={{ maxWidth:'100%', maxHeight:120, objectFit:'contain' }} />) : (<Typography variant="caption" color="text.secondary">No image</Typography>)} + {img.url ? (<img src={img.url} alt={`${img.survey} ${img.band} cutout for ${img.key}`} style={{ maxWidth:'100%', maxHeight:120, objectFit:'contain' }} />) : (<Typography variant="caption" color="text.secondary">No image</Typography>)}
120-123: Alt text for modal image.Apply:
- <img src={current.url} style={{ maxWidth:'100%', maxHeight:'70vh', objectFit:'contain', borderRadius:4 }} /> + <img src={current.url} alt={`${current.survey} ${current.band} cutout for ${current.key}`} style={{ maxWidth:'100%', maxHeight:'70vh', objectFit:'contain', borderRadius:4 }} />
37-51: Optional: Fetch in parallel with limited concurrency for large sets.Sequential
awaitper item will be slow; consider a small pool (e.g., 6–8) and update progress as promises settle.If helpful, I can provide a p-limit based diff.
dashboard/src/components/CutoutGrid.tsx (2)
1-1: ImportuseEffectfor cleanup.Apply:
-import React, { memo } from 'react'; +import React, { memo, useEffect } from 'react';
42-49: Add alt text for accessibility.Apply:
- {!isLoading && data && <img - src={data} + {!isLoading && data && <img + src={data} + alt={`${record.survey} ${record.band} cutout for ${record.JNAME}`} loading="lazy" decoding="async" crossOrigin="anonymous" style={{ width:'100%', height:'100%', objectFit:'contain', imageRendering:'auto' }}dashboard/src/components/SkyMap.tsx (4)
231-233: Keep marker radius constant in screen px under anisotropic scaling.Use the larger axis scale for px→world conversion; current code uses only baseScaleX, producing ellipses with zoom/aspect changes.
- const pxToWorld = 1 / (baseScaleX*zoom); - const rWorld = targetPx * pxToWorld; + const invS = 1 / (Math.max(baseScaleX, baseScaleY) * zoom); + const rWorld = targetPx * invS;and
- const pxToWorld = 1 / (baseScaleX*zoom); - const rWorld = targetPx * pxToWorld; + const invS = 1 / (Math.max(baseScaleX, baseScaleY) * zoom); + const rWorld = targetPx * invS;Also applies to: 246-247
362-377: Add pointercancel handler to avoid stuck “grabbing” state on gesture interruptions.canvas.addEventListener('pointerup', onUp); + canvas.addEventListener('pointercancel', onUp); canvas.addEventListener('pointerleave', onUp); @@ - canvas.removeEventListener('pointerup', onUp as any); + canvas.removeEventListener('pointerup', onUp as any); + canvas.removeEventListener('pointercancel', onUp as any); canvas.removeEventListener('pointerleave', onUp as any);
97-106: Remove unused refs (dead state).lastPtsRef, lastSelectedRef, lastPanRef, lastZoomRef are assigned but never read. Trim to reduce churn.
307-312: Redundant redraw effects.Both effects call draw(); a single effect keyed on draw is sufficient.
- useEffect(()=>{ draw(); }, [draw]); - - // Redraw selection change - useEffect(()=>{ draw(); }, [selected, draw]); + useEffect(()=>{ draw(); }, [draw]);dashboard/src/components/ObjectsTable.tsx (1)
31-33: Tighten types for handlers and row-className.Avoid any; use DataGrid types for better safety and editor help.
-import { DataGrid, GridColDef } from '@mui/x-data-grid'; +import { DataGrid, GridColDef, GridRowParams, GridPaginationModel, GridRowClassNameParams } from '@mui/x-data-grid'; @@ - const handleRowClick = useCallback((params: any) => { + const handleRowClick = useCallback((params: GridRowParams) => { onSelect(params.row.JNAME); }, [onSelect]); @@ - const handlePaginationChange = useCallback((model: any) => { + const handlePaginationChange = useCallback((model: GridPaginationModel) => { setPage(model.page); setPageSize(model.pageSize); }, []); @@ - getRowClassName: (params: any) => params.row.JNAME === selected ? 'selected-row' : '', + getRowClassName: (params: GridRowClassNameParams) => (params.row as any).JNAME === selected ? 'selected-row' : '',Also applies to: 35-38, 65-66, 22-24
dashboard/src/App.tsx (6)
170-171: Avoid mutating dependencies with in-place sort.
references.sort()mutates the array and can create subtle ordering bugs. Sort a copy instead.- const allReferences = useMemo(()=> references.sort(), [references]); + const allReferences = useMemo(()=> [...references].sort(), [references]);
53-66: Stream min/max instead of building large arrays.
Math.min(...vals)with spreads is memory-heavy for big datasets. Compute min/max in one pass.- const domain = useMemo(()=>{ - const acc: Record<string,{min:number;max:number}> = {}; - numericFields.forEach(f=>{ - const vals: number[] = []; - for(const r of database){ - if(!r) continue; - let v: unknown = r[f.key]; - if(typeof v === 'string'){ const parsed = parseFloat(v); if(!isNaN(parsed)) v = parsed; } - if(typeof v === 'number' && !isNaN(v)) vals.push(v); - } - if(vals.length){ acc[f.key] = { min: Math.min(...vals), max: Math.max(...vals) }; } - }); - return acc; - }, [database, numericFields]); + const domain = useMemo(()=>{ + const acc: Record<string,{min:number;max:number}> = {}; + for (const f of numericFields) { + let min = Infinity, max = -Infinity; + for (const r of database) { + if(!r) continue; + let v: unknown = r[f.key]; + if (typeof v === 'string') { const p = parseFloat(v); if (!isNaN(p)) v = p; } + if (typeof v === 'number' && !isNaN(v)) { if (v < min) min = v; if (v > max) max = v; } + } + if (min !== Infinity) acc[f.key] = { min, max }; + } + return acc; + }, [database, numericFields]);
150-156: Reset should also clear current selection (optional).When resetting filters, keeping a stale JNAME selected can confuse users. Clear selection too.
- const resetFilters = useCallback(() => { - setFilters(initialFilters); - }, [initialFilters]); + const resetFilters = useCallback(() => { + setFilters(initialFilters); + setJName(''); + }, [initialFilters]);
28-33: Static JSON: make queries non-stale and avoid retries.These endpoints are static assets. Consider disabling retries and marking data perpetually fresh.
- const { data: database = [], isLoading: dbLoading, error: dbError } = useQuery({ queryKey: ['db'], queryFn: loadDatabase }); + const { data: database = [], isLoading: dbLoading, error: dbError } = + useQuery({ queryKey: ['db'], queryFn: loadDatabase, staleTime: Infinity, retry: false }); - const { data: consolidated = [], isLoading: consLoading, error: consError } = useQuery({ queryKey: ['cons'], queryFn: loadConsolidated }); + const { data: consolidated = [], isLoading: consLoading, error: consError } = + useQuery({ queryKey: ['cons'], queryFn: loadConsolidated, staleTime: Infinity, retry: false }); - const { data: dictionary = {} as Record<string, unknown>, isLoading: dictLoading, error: dictError } = useQuery({ queryKey: ['dict'], queryFn: loadDictionary }); + const { data: dictionary = {} as Record<string, unknown>, isLoading: dictLoading, error: dictError } = + useQuery({ queryKey: ['dict'], queryFn: loadDictionary, staleTime: Infinity, retry: false }); - const { data: cutouts = [], isLoading: cutoutsLoading, error: cutoutsError } = useQuery({ queryKey: ['cutouts'], queryFn: loadCutouts }); + const { data: cutouts = [], isLoading: cutoutsLoading, error: cutoutsError } = + useQuery({ queryKey: ['cutouts'], queryFn: loadCutouts, staleTime: Infinity, retry: false });
184-188: Disable “Reset Filters” based on actual state, not reference equality.
filters === initialFiltersonly catches the “just-reset” reference. Consider a shallow check on fields to better reflect user changes.Example:
- disabled={filters === initialFilters} + disabled={ + filters.jnameSearch === '' && + filters.references.length === 0 && + Object.values(filters.numeric).every(v => v == null) + }
198-203: Error panel: avoid leaking raw errors to end users.Render a friendly message and log details to console for diagnostics.
- <Typography variant="body2" color="text.secondary">{String(anyError)}</Typography> + <Typography variant="body2" color="text.secondary"> + One or more datasets failed to load. Please retry or check hosting configuration. + </Typography> + {console.error('[Explorer] Data load error:', anyError)}dashboard/src/components/FiltersDrawer.tsx (1)
31-31: Format numeric labels.Long floats can be noisy. Format to a few decimals.
-const pct = (min: number, max: number) => `${min} – ${max}`; +const fmt = (n: number) => (Number.isInteger(n) ? n : +n.toFixed(4)); +const pct = (min: number, max: number) => `${fmt(min)} – ${fmt(max)}`;
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (2)
dashboard/package-lock.jsonis excluded by!**/package-lock.jsondashboard/public/telescope.pngis excluded by!**/*.png
📒 Files selected for processing (30)
.devcontainer/devcontainer.json(1 hunks).github/workflows/deploy.yml(1 hunks)README.md(1 hunks)dashboard/.gitignore(1 hunks)dashboard/README.md(1 hunks)dashboard/eslint.config.js(1 hunks)dashboard/index.html(1 hunks)dashboard/package.json(1 hunks)dashboard/prepare_data.py(1 hunks)dashboard/public/data/.gitkeep(1 hunks)dashboard/src/App.tsx(1 hunks)dashboard/src/api.ts(1 hunks)dashboard/src/components/CutoutGrid.tsx(1 hunks)dashboard/src/components/DataTables.tsx(1 hunks)dashboard/src/components/FiltersDrawer.tsx(1 hunks)dashboard/src/components/Gallery.tsx(1 hunks)dashboard/src/components/ObjectsTable.tsx(1 hunks)dashboard/src/components/SkyMap.tsx(1 hunks)dashboard/src/components/VirtualizedList.tsx(1 hunks)dashboard/src/env.d.ts(1 hunks)dashboard/src/hooks/useDebounce.ts(1 hunks)dashboard/src/main.tsx(1 hunks)dashboard/src/theme.ts(1 hunks)dashboard/src/types.ts(1 hunks)dashboard/tsconfig.json(1 hunks)dashboard/vite.config.ts(1 hunks)notebooks/footprints/footprints.ipynb(1 hunks)notebooks/how_to.ipynb(1 hunks)notebooks/modified-gravity-tests/01_LaStBeRu_cosmo_ground.ipynb(1 hunks)notebooks/proposals/proposals.ipynb(1 hunks)
🔥 Files not summarized due to errors (1)
- notebooks/modified-gravity-tests/01_LaStBeRu_cosmo_ground.ipynb: Error: Server error: no LLM provider could handle the message
👮 Files not reviewed due to content moderation or server errors (1)
- notebooks/modified-gravity-tests/01_LaStBeRu_cosmo_ground.ipynb
🧰 Additional context used
🧬 Code graph analysis (6)
dashboard/src/api.ts (1)
dashboard/src/types.ts (4)
DataRecord(1-4)ConsolidatedRecord(6-9)Dictionary(23-23)CutoutRecord(11-16)
dashboard/src/main.tsx (1)
dashboard/src/theme.ts (1)
darkAquaTheme(13-137)
dashboard/src/components/Gallery.tsx (2)
dashboard/src/types.ts (1)
CutoutRecord(11-16)dashboard/src/api.ts (1)
getCutoutObject(52-104)
dashboard/src/App.tsx (8)
dashboard/src/components/DataTables.tsx (1)
DataTables(32-87)dashboard/src/components/CutoutGrid.tsx (1)
CutoutGrid(12-25)dashboard/src/api.ts (4)
loadDatabase(5-8)loadConsolidated(10-13)loadDictionary(15-20)loadCutouts(22-25)dashboard/src/components/FiltersDrawer.tsx (2)
FiltersState(12-16)FiltersDrawer(33-148)dashboard/src/hooks/useDebounce.ts (1)
useDebounce(3-17)dashboard/src/components/ObjectsTable.tsx (1)
ObjectsTable(18-109)dashboard/src/components/SkyMap.tsx (1)
SkyMap(41-398)dashboard/src/types.ts (1)
CutoutRecord(11-16)
dashboard/src/components/CutoutGrid.tsx (2)
dashboard/src/types.ts (1)
CutoutRecord(11-16)dashboard/src/api.ts (1)
getCutoutObject(52-104)
dashboard/src/components/DataTables.tsx (1)
dashboard/src/types.ts (2)
DataRecord(1-4)ConsolidatedRecord(6-9)
🪛 markdownlint-cli2 (0.17.2)
dashboard/README.md
18-18: Tables should be surrounded by blank lines
(MD058, blanks-around-tables)
🪛 Ruff (0.12.2)
notebooks/how_to.ipynb
64-64: Undefined name display
(F821)
dashboard/prepare_data.py
245-245: Do not use bare except
(E722)
🪛 actionlint (1.7.7)
.github/workflows/deploy.yml
25-25: the runner of "actions/setup-python@v4" action is too old to run on GitHub Actions. update the action's version to fix this issue
(action)
🪛 YAMLlint (1.37.1)
.github/workflows/deploy.yml
[error] 23-23: trailing spaces
(trailing-spaces)
[error] 28-28: trailing spaces
(trailing-spaces)
[error] 35-35: trailing spaces
(trailing-spaces)
[error] 40-40: trailing spaces
(trailing-spaces)
[error] 49-49: trailing spaces
(trailing-spaces)
[error] 54-54: trailing spaces
(trailing-spaces)
[error] 57-57: trailing spaces
(trailing-spaces)
[error] 63-63: trailing spaces
(trailing-spaces)
🔇 Additional comments (11)
dashboard/src/components/VirtualizedList.tsx (2)
1-82: Solid, minimal virtualization component.Clean math, correct index clamping, and nice fallback for small lists. Good reuse potential.
6-7: Validate heights > 0.Guard against itemHeight <= 0 or containerHeight <= 0 to prevent division edge cases.
Do you want me to add a small runtime assert (and story/test) to enforce positive heights?
notebooks/proposals/proposals.ipynb (1)
875-881: Verify and correct boolean comparison foris_rgb
- Comparing to string
"True"may mis-filter; confirmCutouts["is_rgb"].dtypeis boolean.- If it’s not, convert first:
Cutouts["is_rgb"] = Cutouts["is_rgb"].astype("boolean")- Then update the query:
- .query('cutout_size == "20asec" and is_rgb == "True"') + .query('cutout_size == "20asec" and is_rgb == True')dashboard/src/types.ts (1)
11-16: Align CutoutRecord with the JSON shape from prepare_dataRemove any fields dropped in data prep (cutout_size, file_name) and keep band required; add the two columns still present in
/data/cutouts.json:dashboard/src/types.ts export interface CutoutRecord { JNAME: string; survey: string; band: string; file_path: string; + processing: string; + is_rgb: string; }Likely an incorrect or invalid review comment.
dashboard/eslint.config.js (1)
10-19: ESLint top-level await is supported; no changes required. dashboard/package.json specifies eslint ^9.9.0 and CI uses Node 18, satisfying the flat-config ESM and top-level-await requirements. Optional: switch to static imports for broader environment compatibility.dashboard/prepare_data.py (1)
12-21: Validate MinIO vars early, strip scheme, and fail fast
Add a guard at the top of dashboard/prepare_data.py to ensure all three env vars are set, remove anyhttp(s)://scheme before passing to Minio, and raise a clear error. If you support Python < 3.9, replacestr.removeprefixwithurlparseor a regex. Also confirm that yourendpointis inhost[:port]form:MINIO_ENDPOINT_URL = os.getenv("MINIO_ENDPOINT_URL") ACCESS_KEY = os.getenv("MINIO_ACCESS_KEY") SECRET_KEY = os.getenv("MINIO_SECRET_KEY") +if not (MINIO_ENDPOINT_URL and ACCESS_KEY and SECRET_KEY): + raise RuntimeError( + "MINIO_ENDPOINT_URL, MINIO_ACCESS_KEY, and MINIO_SECRET_KEY must be set" + ) +# strip any URL scheme (use urlparse or removeprefix on Python ≥3.9) +endpoint = MINIO_ENDPOINT_URL.removeprefix("https://").removeprefix("http://") client = Minio( - MINIO_ENDPOINT_URL, + endpoint, access_key=ACCESS_KEY, secret_key=SECRET_KEY, secure=True, )dashboard/src/main.tsx (2)
26-35: React Query defaults look goodSane cache policy for an interactive dashboard; no changes needed.
26-35: No changes needed: @tanstack/react-query v5.56.2 supportsgcTime.dashboard/src/components/CutoutGrid.tsx (1)
30-35: Optional: avoid caching blob URLs in React Query.If using @tanstack/react-query v5, set
gcTime: 0; if v4, setcacheTime: 0to reduce holding blob URLs in memory after unmount.Want me to propose the exact option based on your installed version?
dashboard/src/components/SkyMap.tsx (1)
380-397: Nice UX polish.Controls, zoom bounds, and DPR handling look solid. The selection glow is tasteful and performant with conditional RAF.
dashboard/src/components/ObjectsTable.tsx (1)
40-48: Auto-scroll to selection is a nice touch.Keeping the selected row visible improves ergonomics with large datasets.
Summary by CodeRabbit
New Features
Deployment
Documentation
Chores