feat(database): add SQLite storage backend #201
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Implement SQLite integration with SQLAlchemy/SQLModel - Add support for JSON-serialized embeddings (no native vector type) - Implement brute-force in-memory vector search for SQLite - Add SQLite repositories for Resources, Items, Categories, and Relations - Add documentation and integration tests Signed-off-by: Jintao Zhang <[email protected]>
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request adds SQLite as a lightweight, file-based storage backend for MemU, providing an alternative to the existing in-memory and PostgreSQL backends. The implementation follows the existing repository pattern and includes comprehensive documentation.
Key Changes
- SQLite storage backend with SQLAlchemy/SQLModel integration using JSON-serialized embeddings
- Brute-force in-memory vector search implementation for SQLite (no native vector support)
- Complete repository implementations for Resources, Items, Categories, and Relations
- Integration tests and comprehensive documentation
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sqlite.py | Integration test demonstrating SQLite backend with memorize/retrieve operations |
| src/memu/database/sqlite/sqlite.py | Core SQLiteStore implementation managing repositories and database lifecycle |
| src/memu/database/sqlite/session.py | Session manager handling SQLite engine and connection lifecycle |
| src/memu/database/sqlite/schema.py | Schema builder with model caching and dynamic scope field support |
| src/memu/database/sqlite/repositories/base.py | Base repository class with shared filtering, embedding serialization utilities |
| src/memu/database/sqlite/repositories/resource_repo.py | Resource repository with CRUD operations and caching |
| src/memu/database/sqlite/repositories/memory_item_repo.py | Memory item repository with vector search and update operations |
| src/memu/database/sqlite/repositories/memory_category_repo.py | Category repository with get-or-create pattern |
| src/memu/database/sqlite/repositories/category_item_repo.py | Relation repository managing item-category links |
| src/memu/database/sqlite/repositories/init.py | Repository package exports |
| src/memu/database/sqlite/models.py | SQLModel definitions with JSON embedding properties and table configurations |
| src/memu/database/sqlite/init.py | Package entry point with database builder function |
| src/memu/database/factory.py | Updated factory to support SQLite provider selection |
| src/memu/database/init.py | Added sqlite to package exports |
| src/memu/app/settings.py | Extended configuration to support sqlite provider and updated DSN description |
| docs/sqlite.md | Comprehensive documentation covering setup, configuration, migration, and troubleshooting |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Cast json.loads result to list explicitly to satisfy mypy - Ensure strict type checking passes for new sqlite module
- Rename cat_id to category_id for consistency - Use slice assignment for in-place cache update in unlink_item_category - Add logging for embedding JSON parse failures - Use specific exception types instead of bare Exception - Remove incorrect Raises section from docstring
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
close #170