…red wiki/documentation generation. Unified file-comments and headers in format nd style

…nd scores

CRITICAL BUG FIXES:
- Fix array consistency in sparse_bm25.py: filter scores alongside docs
- Fix array consistency in dense_faiss.py: maintain docs-scores alignment
- Fix SQLAlchemy datetime import issue in models.py

PROBLEM SOLVED:
When documents were missing from database but present in vector index,
retrievers would return mismatched arrays where len(docs) != len(scores).
This caused runtime errors in hybrid retriever and inconsistent API responses.

SOLUTION IMPLEMENTED:
- Iterate through retrieved_ids/doc_ids with enumerate()
- Only append doc AND corresponding score if doc exists in database
- Ensures len(ordered_docs) == len(filtered_scores) always

COMPREHENSIVE TESTS ADDED:
- 16 parametrized tests covering missing document scenarios
- Edge cases: empty results, database errors, k parameter validation
- Array consistency verification for all retrieval modes
- Score normalization and ordering validation

VALIDATION:
- All 77 existing tests pass (no regressions)
- 16 new tests pass covering critical edge cases
- Fixes propagate correctly through hybrid retriever

This resolves BUG-001 and BUG-002 from comprehensive bug audit.
Ensures robust production behavior when vector index and database
become inconsistent due to deletions or corruption.

CRITICAL BUG FIX:
- Add transactional safety to ETL pipeline preventing inconsistent state
- Implement rollback mechanism when embedding/vector operations fail
- Add comprehensive input validation and filtering
- Ensure embedding count matches document count

PROBLEM SOLVED:
ETL service could leave database in inconsistent state when:
- Embedding generation failed after documents were stored
- Vector storage failed after documents and embeddings succeeded
- Embedding count didn't match document count

SOLUTION IMPLEMENTED:
- Input validation: filter empty/whitespace-only texts
- Rollback mechanism: remove stored documents on pipeline failures
- Dimension validation: verify embedding count matches document count
- Error context: provide detailed error messages for debugging
- Graceful degradation: handle repositories without deletion support

COMPREHENSIVE TESTS ADDED:
- 17 parametrized tests covering transactional scenarios
- Rollback testing with various failure points
- Input validation and filtering edge cases
- Error propagation and context preservation
- Concurrent ingestion safety verification

VALIDATION:
- All 11 existing ETL tests pass (updated for new filtering behavior)
- 17 new transactionality tests pass
- No regressions in other test suites

This resolves BUG-003 from comprehensive bug audit.
Ensures ETL pipeline maintains consistency between document and vector storage
even under failure conditions.

…shes

CRITICAL BUG FIX:
- Add comprehensive query validation in dense and sparse retrievers
- Prevent IndexError from embedder.embed([query])[0] with empty queries
- Implement consistent query normalization and type safety
- Add graceful error handling with contextual error messages

PROBLEM SOLVED:
Dense retriever could crash with IndexError when:
- Query was None, empty string, or whitespace-only
- Embedder returned empty list for invalid queries
- Non-string types were passed as queries

SOLUTION IMPLEMENTED:
- _is_valid_query() method validates query before processing
- Query normalization with .strip() before embedding/tokenization
- Empty embeddings handling without IndexError
- Type safety checks for non-string inputs
- Consistent validation across dense and sparse retrievers
- Detailed error context preservation for debugging

COMPREHENSIVE TESTS ADDED:
- 29 parametrized tests covering query validation scenarios
- Invalid query types: None, empty, whitespace, non-string
- Query normalization verification
- Embedder failure handling and error context
- Type safety and edge case coverage
- Consistency validation between retriever types

VALIDATION:
- All 59 existing retrieval tests pass (no regressions)
- 29 new query validation tests pass
- Consistent behavior across dense, sparse, and hybrid retrievers

This resolves BUG-004 from comprehensive bug audit.
Ensures retrievers handle invalid queries gracefully without crashes,
maintaining system stability under all input conditions.

CRITICAL BUG FIX:
- Fix SQL document retrieval to preserve input ID order
- Implement efficient O(1) lookup with order preservation
- Handle missing documents gracefully while maintaining order
- Support duplicate IDs in request with proper ordering

PROBLEM SOLVED:
SqlDocumentStorage.get() was not preserving the order of requested IDs:
- SQLAlchemy .all() returns results in database order, not request order
- This broke correspondence between documents and scores in retrievers
- Caused incorrect ranking and inconsistent results

SOLUTION IMPLEMENTED:
- Query all documents with single .in_() filter for efficiency
- Create O(1) lookup dictionary from query results
- Iterate through input IDs to preserve original order
- Skip missing documents without breaking order of found ones
- Support duplicate IDs by preserving all occurrences

COMPREHENSIVE TESTS ADDED:
- 8 tests covering order preservation scenarios
- Basic order preservation with different patterns
- Missing document handling with order maintenance
- Duplicate ID support and content integrity
- Empty and invalid ID list handling
- Performance testing with large ID lists
- Type safety and content integrity validation

VALIDATION:
- All 12 persistence tests pass (no regressions)
- 8 new order preservation tests pass
- Efficient single-query implementation
- Maintains backward compatibility

This resolves BUG-005 from comprehensive bug audit.
Ensures retrieval operations maintain correct document-score correspondence
critical for accurate ranking in search results.

CRITICAL BUG FIX:
- Add robust input validation for question and top_k parameters
- Implement question sanitization with whitespace trimming
- Add reasonable limits for top_k to prevent performance issues
- Provide clear error messages for invalid inputs

PROBLEM SOLVED:
RAG service had no input validation, allowing:
- None, empty, or whitespace-only questions to crash downstream
- Invalid top_k values (0, negative, excessive) causing errors
- Non-string/non-integer types causing type errors
- Poor user experience with cryptic error messages

SOLUTION IMPLEMENTED:
- _validate_and_sanitize_question(): Validates and trims questions
- _validate_top_k(): Validates range (1-50) and type safety
- Clear ValueError messages for all validation failures
- Maintains backward compatibility for valid inputs
- Consistent behavior with retriever validation patterns

COMPREHENSIVE TESTS ADDED:
- 36 parametrized tests covering validation scenarios
- Invalid question types: None, empty, whitespace, non-string
- Invalid top_k types: zero, negative, excessive, non-integer
- Question sanitization verification
- Unicode and special character handling
- Boundary testing for max top_k limits
- Performance impact validation
- Error message accuracy testing

VALIDATION:
- All 38 core service tests pass (no regressions)
- 36 new validation tests pass
- Maintains original functionality for valid inputs
- Provides clear error messages for invalid inputs

This resolves BUG-006 from comprehensive bug audit.
Ensures RAG service handles all input edge cases gracefully,
providing better UX and preventing downstream crashes.

MEDIUM BUG FIX:
- Add comprehensive None and type validation to preprocess_text()
- Implement defensive programming to prevent AttributeError crashes
- Support graceful handling of non-string types with conversion
- Maintain backward compatibility for all valid string inputs

PROBLEM SOLVED:
preprocess_text() could crash with AttributeError when:
- Input text was None (text.lower() on None fails)
- Non-string types were passed (int, list, etc.)
- Objects that couldn't be converted to string were provided

SOLUTION IMPLEMENTED:
- None check returns empty string (safe default)
- Type validation with graceful str() conversion
- Exception handling for unconvertible objects
- Maintains all original text processing functionality
- Updated type annotation to accept str | None

COMPREHENSIVE TESTS ADDED:
- 24 tests covering None validation and type safety
- None input handling and empty string return
- Non-string type conversion (int, float, bool, list, dict)
- Unconvertible object handling with exception safety
- Unicode and special character preservation
- HTML processing and word boundary maintenance
- Performance testing with large inputs
- Regression prevention for critical edge cases

VALIDATION:
- All 24 new validation tests pass
- Maintains original functionality for valid strings
- Defensive behavior prevents any crashes
- Type annotation accurately reflects accepted inputs

This resolves BUG-007 from comprehensive bug audit.
Ensures preprocess_text() is crash-proof and handles all input types
gracefully while maintaining its core text processing capabilities.

…tion-ready

MEDIUM BUG FIX:
- Move hardcoded Spanish message to configurable settings
- Add no_documents_message field to Settings with English default
- Support environment variable configuration (NO_DOCUMENTS_MESSAGE)
- Maintain backward compatibility while enabling internationalization

PROBLEM SOLVED:
RAG service had hardcoded Spanish message that was:
- Not configurable without code changes
- Not internationalization-ready
- Inconsistent with potential English-first applications
- Difficult to maintain across multiple deployment environments

SOLUTION IMPLEMENTED:
- Added no_documents_message to Settings with descriptive field
- Changed default to English for international compatibility
- RAG service now uses settings.no_documents_message
- Supports environment variable override
- Maintains all original functionality

COMPREHENSIVE TESTS ADDED:
- 13 tests covering message configuration scenarios
- Default English message verification
- Custom message configuration via Settings
- Spanish message backward compatibility
- Unicode and special character support
- Environment variable configuration
- Message consistency and isolation testing
- Validation that successful queries are unaffected

VALIDATION:
- All 51 RAG service tests pass (including updated expectations)
- 13 new configuration tests pass
- Backward compatibility maintained (Spanish can be configured)
- Environment variable support verified

This resolves BUG-008 from comprehensive bug audit.
Enables proper internationalization and configuration flexibility
while maintaining system functionality and user experience.

… leaks

FINAL BUG FIX - BUG-009:
- Add configurable history management with enable_history and max_history_entries settings
- Implement automatic cleanup mechanism to prevent unlimited memory growth
- Support multiple cleanup strategies with graceful fallback
- Add comprehensive validation and environment variable support

PROBLEM SOLVED:
RAG service had unlimited history growth causing:
- Memory leaks in long-running applications
- Database bloat with unlimited Q&A storage
- Performance degradation with large history tables
- No way to disable history for privacy/performance needs

SOLUTION IMPLEMENTED:
- enable_history setting to disable history completely (default: True)
- max_history_entries setting with reasonable default (1000, 0=unlimited)
- _save_to_history_if_enabled() method with conditional saving
- _cleanup_history_if_needed() with multiple cleanup strategies
- Graceful error handling if cleanup fails
- Settings validation for non-negative max_history_entries

COMPREHENSIVE TESTS ADDED:
- 18 tests covering history management scenarios
- History enable/disable functionality
- Cleanup with various limits (0=unlimited, 1-10000)
- Alternative cleanup methods when primary unavailable
- Graceful failure handling and error recovery
- Environment variable configuration
- Performance impact validation

VALIDATION:
- All existing RAG tests pass (no regressions)
- 18 new history management tests pass
- MyPy type checking: Success (38 source files)
- Ruff formatting applied and validated
- Production-ready error handling

COMPREHENSIVE BUG AUDIT COMPLETED:
✅ BUG-001 & BUG-002: Array length mismatch in retrievers
✅ BUG-003: ETL transactionality with rollback mechanism
✅ BUG-004: Query validation preventing IndexError crashes
✅ BUG-005: SQL document order preservation
✅ BUG-006: RAG service input validation
✅ BUG-007: Utils None validation and type safety
✅ BUG-008: Configurable messages for internationalization
✅ BUG-009: History memory leak prevention

This resolves the final BUG-009 from comprehensive bug audit.
The RAG prototype now has production-grade robustness with all critical
bugs fixed, comprehensive test coverage, and memory-safe operation.

Fixed all 20 test failures through systematic validation improvements:

ETL Service: Add type checking, validate embeddings (NaN/inf/empty/dims), filter whitespace

FAISS Index: Fix array validation, handle empty/1D arrays correctly

Middleware: Fix prometheus imports, add runtime validation

Tests: Fix imports, regex escaping, mock behavior, unused variables

Code Quality: Pass ruff + mypy, format code, add noqa/nosec comments

Result: 407/407 tests passing