Skip to content

feat(storage): add indexing, pagination, and lazy loading to JSON TaskStorage#54

Open
premchand11 wants to merge 1 commit intoTheDevOpsBlueprint:mainfrom
premchand11:feature/json-optimizations
Open

feat(storage): add indexing, pagination, and lazy loading to JSON TaskStorage#54
premchand11 wants to merge 1 commit intoTheDevOpsBlueprint:mainfrom
premchand11:feature/json-optimizations

Conversation

@premchand11
Copy link

PR Checklist

  • Follows single-purpose principle
  • Tests pass locally
  • Documentation updated (not needed for this PR)

What does this PR do?

This PR optimizes the JSON storage backend to improve performance and scalability.

Changes

  • Added in-memory indexing for O(1) lookups (get_task).
  • Implemented pagination with list_tasks(page, page_size) to support the ls command.
  • Added lazy loading via iter_tasks(start, count) for large task lists.
  • Added tests .

Related Issue

Closes #20

Type of change

  • Bug fix
  • New feature
  • Configuration
  • Documentation

@Valentin-v-Todorov
Copy link
Contributor

@premchand11

Critical problems:

No CLI integration - The ls command still loads ALL tasks. Pagination methods were added but never called.
Data consistency risk - In-memory index can desync from file, especially during upgrades from old format
Buggy _next_id calculation - save_tasks() recalculates IDs incorrectly after deletions
Backward compatibility broken - Requires calling private _load_index() method manually

This PR needs major rework. It adds complexity without solving the actual performance issue since the CLI doesn't use the new pagination features. Consider either:

The current implementation is incomplete and introduces bugs while not delivering the promised performance benefits.

Try starting from scratch and create smaller more focused PRs. Add the PRs to the issue so we can keep track.


If you feel like you don't have the mood for this issue no worries! You can drop it or take another. We are not at work here ! :D

HAVE FUN !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize performance

2 participants