Haystack Diagnostics Engine

A dedicated diagnostics and observability engine built specifically for Haystack 2.x RAG pipelines and document stores.

It connects the dots between pipeline introspection, native tracing, and validation to solve ingestion-layer anomalies and retrieval failure modes before they compromise downstream applications.

While Haystack already offers robust pipeline graphing, OpenTelemetry tracing, and evaluator wrappers (like RagasEvaluator), there has been no unified tool to:

1. Validate Document Store Health: Catching silent ingestion bugs (such as content=None which crashes downstream language classifiers or prompt builders) before queries run.
2. Automate Retrieval Failure Classification: Systematically categorizing why a query returned a wrong answer or low-quality documents.
3. Unify the Debugging Workflow: Bringing Inspect → Validate → Diagnose into a single library and a lightweight Model Context Protocol (MCP) server.

Evaluates the health of documents written to a Haystack document store. It checks for:

• Tenant-Scoped/Filter Isolation: Supports a filters parameter to isolate document analysis (e.g. scoping health checks by user_id or other metadata in multi-tenant environments).
• content=None (blob-only documents causing pipeline crashes)
• Empty or whitespace-only documents (content="" or consisting only of space characters)
• Very short chunks (character count below threshold)
• Duplicate chunks (detected using MD5 content hash) and duplicate document IDs
• Missing metadata fields needed by pipeline filters
• Null embeddings
• Embedding dimension mismatch (both store-wide consistency and against an expected dimension)
• Optional RAG Lineage Metadata validation (checks for presence of source_id, chunk_id, content_hash, embedding_model, index_version, and indexed_at keys, enabled via check_lineage_metadata=True)

Inspects a live Haystack pipeline and returns its structural metadata:

• All components and their class paths
• Input and output sockets (including types, mandatory/optional status, and default values)
• Component-to-component connections
• A native Mermaid.js graph representation of the pipeline topology (drawn using Haystack's native layout engine)

Classifies query-level retrieval and pipeline failures by running the pipeline and extracting intermediate retriever outputs. It classifies failures in the following sequence:

1. NO_RESULTS: The retriever returned 0 documents.
2. FILTER_EXCLUSION (requires relevant_doc_id + a document store): The expected document exists in the document store but was excluded by the query's active metadata filters.
3. SCORE_BELOW_CUTOFF / RANKING_FAILURE: The top result's score was below ranking_threshold.
   • With relevant_doc_id hint: classified as SCORE_BELOW_CUTOFF (expected doc was not retrieved and not excluded by filters) or CONTEXT_LOSS (see below).
   • Without hint: classified as RANKING_FAILURE (legacy, backwards-compatible).
4. CONTEXT_LOSS (requires relevant_doc_id + a reranker in the pipeline): The expected document was present in the retriever's raw output but was dropped by the reranker. Covers reranker demotion; prompt-builder truncation detection is a planned future extension.
5. EMPTY_CONTEXT: Documents were retrieved, but their combined content is empty or contains no usable text.
6. GENERATOR_FAILURE: Relevant context was supplied, but the generator returned an empty response, an LLM refusal (e.g., "I don't know"), or did not match the expected_answer (if provided).

Automated Discovery: If document_store is not explicitly passed to diagnose_retrieval_failure or collect_debug_bundle, the toolchain automatically attempts to discover it from the pipeline's retriever component. This works reliably across all standard database integrations (like Weaviate, Qdrant, Elasticsearch, and InMemory stores) by checking retriever attributes.

Runtime safety: If a reranker is detected in the pipeline but its intermediate output is missing from pre-computed pipeline_outputs, an explicit warning note is included in the return dict. There is no silent fallback to RANKING_FAILURE.

Captures the full state of a single query execution as a structured, diffable JSON file on disk. Designed for the debugging workflow described by production users: run one query, persist everything, compare against a previous known-good run.

Each bundle captures:

• Pipeline graph, Haystack version, and component init_parameters
• Raw retriever top-k (pre-reranker) — scores, metadata, content previews
• Reranked top-k (when a reranker is detected)
• Prompt snapshot and generated answer
• Failure classification
• Corpus health checks scoped only to the retrieved document IDs (not the full store)

Bundle filename: {query_slug}_{timestamp}.json — human-readable, sorts naturally across runs of the same query. The bundle_id UUID lives inside the JSON.

diff_debug_bundles(bundle_a, bundle_b, ignore_config_paths=...): Compares two persisted bundles and reports score deltas per document, docs that appeared or disappeared, component config changes, and a character-level answer diff (via difflib, no tokenizer dependency).

Config diffs filter out known volatile fields (e.g. InMemoryDocumentStore.index, a random UUID generated at instantiation) by default via DEFAULT_VOLATILE_CONFIG_PATHS. Pass ignore_config_paths=set() to disable filtering, or extend the default set with component-specific paths like "retriever.session_id".

CLI: python -m diagnostics.debug_bundler diff <bundle_a.json> <bundle_b.json>

To run the local diagnostics demo, execute:

python demo/sample_pipeline.py

This script populates an in-memory document store with a mix of healthy and corrupted documents, builds a RAG pipeline, and demonstrates the output of the validator, inspector, and all four failure diagnoses cases (Success, No Results, Generator Refusal, and Ranking Failure) with zero external API dependencies.

============================================================
             HAYSTACK DIAGNOSTICS DEMO RUN
============================================================

[Step 1] Initializing Document Store and Indexing Documents...
Indexed 7 documents.

[Step 2] Running validate_document_store()...
{
  "summary": {
    "total_documents": 7,
    "valid_documents": 0,
    "invalid_documents": 7,
    "total_issues_found": 12
  },
  "checks": {
    "content_none": {
      "status": "fail",
      "count": 1,
      "document_ids": [
        "0f5940589232f80dd5547fea9d88a13ff9f217b4fe124de4c24e13f57dd4ad4a"
      ]
    },
    "empty_content": {
      "status": "fail",
      "count": 1,
      "document_ids": [
        "5241c4b42c205fc2dc784e0d4700d76f70645eebb445c9368948c154d019ad92"
      ]
    },
    "short_chunks": {
      "status": "warning",
      "count": 1,
      "document_ids": [
        "b11b37cde4c884c418583ef1b3003e367ba24e80b12983f1188c88faab990918"
      ],
      "threshold": 20
    },
    "duplicate_chunks": {
      "status": "fail",
      "count": 1,
      "duplicates": [
        {
          "content_hash": "d509e5a2c3cca5a6ca1492380a149f80",
          "document_ids": [
            "1d48eac32b58322c065b46888b5b078e95c783b7d52a01ffa7c8aaa17234af70",
            "0c831f313e41d38cb742717723e5806227e9d1b96be54fe4f966a89fb91eb469"
          ]
        }
      ]
    },
    "missing_metadata": {
      "status": "pass",
      "count": 0,
      "details": []
    },
    "null_embeddings": {
      "status": "fail",
      "count": 7,
      "document_ids": [
        "335f72ebc7e221eb58b13f1e8b6a86e056cc79fc064c4539b461fe624f7b96d0",
        "1d48eac32b58322c065b46888b5b078e95c783b7d52a01ffa7c8aaa17234af70",
        "73572c7e0908450c20ebf64119f41d570a1dc7d0c53b91b9953698744051e048",
        "0f5940589232f80dd5547fea9d88a13ff9f217b4fe124de4c24e13f57dd4ad4a",
        "5241c4b42c205fc2dc784e0d4700d76f70645eebb445c9368948c154d019ad92",
        "b11b37cde4c884c418583ef1b3003e367ba24e80b12983f1188c88faab990918",
        "0c831f313e41d38cb742717723e5806227e9d1b96be54fe4f966a89fb91eb469"
      ]
    },
    "embedding_dimension_mismatch": {
      "status": "pass",
      "count": 0,
      "expected_dimension": null,
      "actual_dimensions": {},
      "details": []
    }
  }
}

[Step 3] Constructing RAG Pipeline...
Pipeline constructed and connected successfully.

[Step 4] Running inspect_pipeline()...
Pipeline Metadata: {}
Pipeline Components found: ['retriever', 'prompt_builder', 'generator']
Pipeline Connections count: 2

Generated Mermaid Diagram:
------------------------------------------------------------

%%{ init: {} }%%

graph TD;

retriever["<b>retriever</b><br><small><i>InMemoryBM25Retriever<br><br>Optional inputs:<ul style='text-align:left;'><li>filters (dict[str, Any] | None)</li><li>top_k (int | None)</li><li>scale_score (bool | None)</li></ul></i></small>"]:::component -- "documents -> documents<br><small><i>list[Document]</i></small>" --> prompt_builder["<b>prompt_builder</b><br><small><i>PromptBuilder<br><br>Optional inputs:<ul style='text-align:left;'><li>template (str | None)</li><li>template_variables (dict[str, Any] | None)</li></ul></i></small>"]:::component
prompt_builder["<b>prompt_builder</b><br><small><i>PromptBuilder<br><br>Optional inputs:<ul style='text-align:left;'><li>template (str | None)</li><li>template_variables (dict[str, Any] | None)</li></ul></i></small>"]:::component -- "prompt -> prompt<br><small><i>str</i></small>" --> generator["<b>generator</b><br><small><i>SimpleDemoGenerator</i></small>"]:::component
i{&ast;}--"query<br><small><i>str</i></small>"--> retriever["<b>retriever</b><br><small><i>InMemoryBM25Retriever<br><br>Optional inputs:<ul style='text-align:left;'><li>filters (dict[str, Any] | None)</li><li>top_k (int | None)</li><li>scale_score (bool | None)</li></ul></i></small>"]:::component
i{&ast;}--"query<br><small><i>Any</i></small>"--> prompt_builder["<b>prompt_builder</b><br><small><i>PromptBuilder<br><br>Optional inputs:<ul style='text-align:left;'><li>template (str | None)</li><li>template_variables (dict[str, Any] | None)</li></ul></i></small>"]:::component
generator["<b>generator</b><br><small><i>SimpleDemoGenerator</i></small>"]:::component--"replies<br><small><i>list</i></small>"--> o{&ast;}

classDef component text-align:center;


------------------------------------------------------------

[Step 5] Running failure diagnostics for different query cases...

--- CASE A: SUCCESS QUERY (Query: 'Paris') ---
Detected Failure Type: SUCCESS
Answer: The capital of France is Paris.
Triggered Checks: {'no_results': False, 'empty_context': False, 'generator_failure': False, 'ranking_failure': False}

--- CASE B: NO RESULTS FAILURE (Query: 'Tokyo') ---
Detected Failure Type: NO_RESULTS
Triggered Checks: {'no_results': True, 'empty_context': False, 'generator_failure': False, 'ranking_failure': False}

--- CASE C: GENERATOR FAILURE - REFUSAL (Query: 'Rome') ---
Detected Failure Type: GENERATOR_FAILURE
Answer: I'm sorry, I do not know the answer based on the provided context.
Triggered Checks: {'no_results': False, 'empty_context': False, 'generator_failure': True, 'ranking_failure': False}

--- CASE D: RANKING FAILURE (Query: 'Berlin', Threshold: 5.0) ---
Detected Failure Type: RANKING_FAILURE
Top Document Score: 1.1650555327475023
Triggered Checks: {'no_results': False, 'empty_context': False, 'generator_failure': False, 'ranking_failure': True}

============================================================
Demo completed successfully!
============================================================

Tested against a live RAG Studio (Vectornest AI) instance backed by Weaviate 1.25.10 with 823 ingested chunks (OpenAI 1536-dim embeddings).

• validate_document_store findings:
  • 195 duplicate chunks (23.7% of corpus)
  • 8 short chunks below minimum content threshold
  • 14 documents with missing metadata keys
  • Tenant-scoped validation via filters parameter correctly isolated 796 documents for a single user_id
• inspect_pipeline findings:
  • Reconstructed and mapped a 4-component RAG pipeline (OpenAITextEmbedder → WeaviateEmbeddingRetriever → PromptBuilder → OpenAIGenerator) with full Mermaid.js graph output.
• diagnose_retrieval_failure findings:
  • Correctly classified a gibberish query (xyzabcde123) as GENERATOR_FAILURE based on LLM refusal patterns.
• MCP Benchmark:
  • 15 concurrent inspect_pipeline_graph calls over stdio via asyncio.gather completed in ~0.95s with zero lock contention.

1. Clone the repository and navigate to it:

   cd haystack-diagnostics

2. Install the package and dependencies: You can install the package in editable mode along with all core dependencies:

   pip install -e .

   To install with development dependencies (e.g. pytest):

   pip install -e ".[dev]"

   Or replicate the exact conda/pip environment using pinned versions:

   pip install -r requirements.txt

   Contributors: install from requirements.txt before pip install -e .[dev]. pyproject.toml declares a haystack-ai>=2.29.0 lower bound so pip may otherwise resolve a newer release than the tested version. The requirements.txt is the canonical lockfile; keep it in sync when bumping dependencies.

from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack import Document, Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.builders import PromptBuilder

# Import the diagnostic tools
from diagnostics import (
    validate_document_store,
    inspect_pipeline,
    diagnose_retrieval_failure,
    collect_debug_bundle,
    diff_debug_bundles,
    DEFAULT_VOLATILE_CONFIG_PATHS,  # frozenset of known volatile config fields
)

# 1. Validate Document Store Ingestion Health
document_store = InMemoryDocumentStore()
# (Write documents to your store...)

report = validate_document_store(
    document_store=document_store,
    expected_metadata_keys=["source", "language"],
    expected_embedding_dim=1536,
    short_chunk_threshold=50,
    filters={"user_id": "tenant-123"},  # Optional scoping for tenant isolation
    check_lineage_metadata=True,        # Optional checking for RAG lineage tags
)
print("Store Health Report:", report["summary"])

# 2. Inspect Pipeline Structure
pipe = Pipeline()
# (Add components and connect them...)

structure = inspect_pipeline(pipe)
print("Mermaid graph:\n", structure["mermaid"])

# 3. Diagnose Retrieval Failures
# Basic usage (legacy RANKING_FAILURE bucket)
diagnostics = diagnose_retrieval_failure(
    pipeline=pipe,
    query="What is the capital of France?",
    expected_answer="Paris",
    ranking_threshold=0.6
)
print("Primary Failure Type:", diagnostics["failure_type"])

# Enhanced usage with relevant_doc_id hint for SCORE_BELOW_CUTOFF / CONTEXT_LOSS split
diagnostics = diagnose_retrieval_failure(
    pipeline=pipe,
    query="What is the capital of France?",
    relevant_doc_id="doc-france-wiki",   # expected doc ID
    reranker_component_name="ranker",     # optional, auto-discovered if None
    ranking_threshold=0.6
)
print("Failure Subtype:", diagnostics["diagnostics"]["ranking"]["failure_subtype"])
# -> SCORE_BELOW_CUTOFF if doc not retrieved and not excluded by filters
# -> FILTER_EXCLUSION if expected doc is in store but excluded by filters
# -> CONTEXT_LOSS if doc retrieved but reranker dropped it

# 4. Collect a Debug Bundle for a single query
bundle = collect_debug_bundle(
    query="What is the capital of France?",
    pipeline=pipe,
    document_store=document_store,    # optional; enables scoped corpus checks
    relevant_doc_id="doc-france-wiki", # optional; enables CONTEXT_LOSS detection
    output_dir="./debug_bundles",     # default; writes {query_slug}_{timestamp}.json
)
print("Bundle written to:", bundle["_bundle_path"])
print("Failure type:", bundle["failure_type"])
print("Raw top-k:", bundle["retrieval"]["raw_top_k"])

# 5. Diff two bundles to find what changed between runs
diff = diff_debug_bundles(
    "./debug_bundles/what_is_the_capital_20260601T120000Z.json",
    "./debug_bundles/what_is_the_capital_20260602T090000Z.json",
    # ignore_config_paths defaults to DEFAULT_VOLATILE_CONFIG_PATHS (suppresses UUIDs like
    # InMemoryDocumentStore.index that change every run but carry no semantic meaning).
    # To suppress additional volatile fields per-component:
    # ignore_config_paths=DEFAULT_VOLATILE_CONFIG_PATHS | {"retriever.session_id"}
    # To disable all filtering and see every diff:
    # ignore_config_paths=set()
)
print("Failure type changed:", diff["failure_type_change"]["changed"])
print("Score deltas:", diff["score_deltas"])
print("Answer diff:\n", diff["answer_diff"])

python -m diagnostics.debug_bundler diff \
    debug_bundles/what_is_the_capital_20260601T120000Z.json \
    debug_bundles/what_is_the_capital_20260602T090000Z.json

Deploy diagnose_retrieval_failure directly in your backend API to automatically classify and log RAG failures in production:

import logging
from fastapi import FastAPI
from pydantic import BaseModel
from diagnostics import diagnose_retrieval_failure
from my_project.pipeline import get_rag_pipeline

app = FastAPI()
logger = logging.getLogger("rag_diagnostics")

class QueryRequest(BaseModel):
    query: str
    expected_answer: str | None = None

@app.post("/query")
async def run_query(request: QueryRequest):
    pipeline = get_rag_pipeline()
    
    # Run the query once, capturing intermediate retriever outputs
    inputs = {
        "retriever": {"query": request.query},
        "prompt_builder": {"query": request.query}
    }
    result = pipeline.run(inputs, include_outputs_from={"retriever"})
    answer = result.get("generator", {}).get("replies", [None])[0]
    
    # If the answer is missing, too short, or indicates refusal, trigger diagnostics (zero-overhead)
    if not answer or any(w in answer.lower() for w in ["sorry", "don't know", "not mentioned"]):
        report = diagnose_retrieval_failure(
            pipeline=pipeline,
            query=request.query,
            pipeline_inputs=inputs,
            pipeline_outputs=result,  # <-- Pass pre-computed outputs (skips second pipeline.run!)
            expected_answer=request.expected_answer,
            ranking_threshold=0.65
        )
        # Log the classified failure type (NO_RESULTS, RANKING_FAILURE, etc.)
        logger.error(f"RAG Failure: {report['failure_type']} | Details: {report['diagnostics']}")
        
    return {"answer": answer}

Verify the state of your document store after bulk uploads or on a cron schedule to alert on malformed documents:

import sys
from diagnostics import validate_document_store
from my_project.db import get_document_store

def run_health_check():
    store = get_document_store()
    report = validate_document_store(
        document_store=store,
        expected_metadata_keys=["source", "author"],
        expected_embedding_dim=1536,
        batch_size=1000,
        validate_embeddings=True  # Disable to skip fetching high-dim vectors
    )
    
    if report["summary"]["invalid_documents"] > 0:
        print(f"ALERT: Detected {report['summary']['total_issues_found']} ingestion errors!")
        sys.exit(1)
        
if __name__ == "__main__":
    run_health_check()

Verify that architectural constraints are not violated by developers modifying pipeline components:

# test_architecture.py
from diagnostics import inspect_pipeline
from my_project.pipeline import build_pipeline

def test_pipeline_layout_constraints():
    pipe = build_pipeline()
    report = inspect_pipeline(pipe)
    
    # Assert structural layout: reranker must send documents to prompt_builder
    connections = report["connections"]
    assert any(
        c["sender"] == "reranker" and c["receiver"] == "prompt_builder"
        for c in connections
    ), "Architecture Error: The Reranker output is not connected to the PromptBuilder."

haystack-diagnostics/
│
├── diagnostics/
│   ├── __init__.py
│   ├── document_validator.py
│   ├── pipeline_inspector.py
│   ├── failure_diagnoser.py
│   └── debug_bundler.py          # collect_debug_bundle, diff_debug_bundles, CLI
│
├── mcp/
│   └── server.py                 # MCP server (validate_store, inspect_pipeline_graph,
│                                 #             diagnose_retrieval, collect_debug_bundle_tool)
│
├── demo/
│   └── sample_pipeline.py        # Out-of-the-box local demo run
│
├── tests/
│   ├── test_document_validator.py
│   ├── test_pipeline_inspector.py
│   ├── test_failure_diagnoser.py
│   ├── test_debug_bundler.py     # Bundle schema, filename, corpus check, diff, CONTEXT_LOSS
│   ├── test_mcp.py               # MCP tool registration and handler tests
│   ├── test_scalability.py       # Performance scalability validation
│   ├── smoke_mcp.py              # Standalone MCP end-to-end smoke test (ASCII status markers)
│   ├── verify_real_world.py      # E2E Weaviate integration validation layers script
│   └── mcp_benchmark.py          # MCP concurrency and latency benchmark script
│
├── pyproject.toml                # Package metadata and build system setup
├── requirements.txt              # Pinned dependencies for environment replication
└── README.md

The project includes a complete suite of unit tests verifying validator edge cases, Mermaid graph exports, sequential RAG query classification, debug bundle schema and diff behaviour, and MCP tool end-to-end smoke tests. All tests run fully offline and require zero external API keys.

54 tests across 6 test files:

• tests/test_document_validator.py: Verifies the document store validation checks (including whitespace, duplicate document IDs, and RAG lineage metadata) using mock documents.
• tests/test_pipeline_inspector.py: Validates component detail extraction, socket parsing, and Mermaid graph output.
• tests/test_failure_diagnoser.py: Verifies the failure classification engine (NO_RESULTS, FILTER_EXCLUSION, SCORE_BELOW_CUTOFF, CONTEXT_LOSS, EMPTY_CONTEXT, GENERATOR_FAILURE), including backwards compatibility, auto-discovery of document stores, and the reranker-detected-but-not-captured warning.
• tests/test_debug_bundler.py: Verifies bundle schema, {query_slug}_{timestamp} filename format (not UUID), scoped corpus checks, failure classification via bundle (including explicit document store pass-in), diff_debug_bundles() score/appearance/config/answer detection, and ignore_config_paths wildcard/component-specific/opt-out behaviour.
• tests/test_mcp.py: Verifies that MCP tools are correctly registered and their arguments/calls are handled properly.
• tests/test_scalability.py: Ensures that performance scale tests execute within constraints.

Standalone Scripts:

• tests/smoke_mcp.py: Standalone end-to-end MCP smoke test covering collect_debug_bundle_tool with inline content, file path, content > path precedence, and neither-provided error handling. Uses tempfile.gettempdir() for a portable output path (Windows-safe and ASCII-safe).
• tests/verify_real_world.py: Standalone real-world validation run that integrates with a live Weaviate backend, verifying all layers of bundle fidelity, taxonomy classification, runtime guardrails, diff engine, and scalability (ASCII-safe).
• tests/mcp_benchmark.py: Standalone concurrency benchmark verifying performance and responsiveness of the MCP server.

To run the full test suite:

pytest tests/

We resolved two critical production issues to ensure robust compatibility with live environments:

• UUID/Datetime Metadata Serialization: Fixed a TypeError when serializing retrieved document metadata containing non-JSON-primitive types (e.g., Weaviate UUID metadata values) by introducing a recursive metadata cleaner.
• PosixPath Stream Loading in MCP Server: Fixed a 'PosixPath' object has no attribute 'read' crash inside the MCP pipeline loader. The engine now correctly opens file-like streams when executing Pipeline.load() from YAML/JSON configs.

• Portable smoke test temp dir and ASCII status markers: tests/smoke_mcp.py previously hardcoded output_dir="/tmp/mcp_smoke_bundles", causing access errors on Windows. Fixed to use a platform-portable temp directory. Also replaced Unicode status glyphs (✓/✗) and em dashes (—) with ASCII-safe markers ([PASS]/[FAIL], -) in tests/smoke_mcp.py and tests/verify_real_world.py to prevent UnicodeEncodeError and rendering issues in Windows cp1252 consoles.
• Noisy config diffs from volatile InMemoryDocumentStore.index: diff_debug_bundles() previously reported config_changes for every InMemoryDocumentStore comparison because index is a random UUID generated at instantiation. Added ignore_config_paths parameter (default: DEFAULT_VOLATILE_CONFIG_PATHS = frozenset({"*.index"})) to suppress known volatile fields. Pass ignore_config_paths=set() to opt out of filtering entirely.
• pyproject.toml version lower bound: Tightened haystack-ai>=2.0.0 to haystack-ai>=2.29.0 to reflect the minimum tested version and prevent pip from silently resolving a newer, untested release when installing without requirements.txt.

The project includes an MCP server (mcp/server.py) that exposes the core diagnostics tools to LLM clients (like Claude Desktop or Cursor).

To start the MCP server:

mcp dev mcp/server.py

Add the following configuration to your claude_desktop_config.json:

{
  "mcpServers": {
    "haystack-diagnostics": {
      "command": "python",
      "args": [
        "/path/to/haystack-diagnostics/mcp/server.py"
      ]
    }
  }
}

Once connected, Claude can automatically validate document store health, query pipeline graphs, debug retrieval failures, and collect full debug bundles for individual queries.

All tools accept pipeline_config_content (inline YAML/JSON string) or pipeline_config_path (file path). pipeline_config_content takes precedence if both are provided.

Here is what an LLM client sends and receives when invoking the validate_store tool using either a local path or inline payloads:

The client instructs the MCP server to validate document store health (this example uses decoupled inline document payloads, bypassing filesystem dependencies):

{
  "name": "validate_store",
  "arguments": {
    "store_type": "in_memory",
    "documents_data": [
      {
        "content": "Paris is the capital of France.",
        "meta": {"source": "wiki"}
      },
      {
        "content": "Berlin is the capital of Germany.",
        "meta": {"source": "wiki"}
      }
    ],
    "expected_metadata_keys": ["source", "language"],
    "expected_embedding_dim": 1536
  }
}

The server returns a structured diagnostic report detailing ingestion issues, duplicates, and missing metadata:

{
  "summary": {
    "total_documents": 7,
    "valid_documents": 0,
    "invalid_documents": 7,
    "total_issues_found": 12
  },
  "checks": {
    "content_none": {
      "status": "fail",
      "count": 1,
      "document_ids": [
        "0f5940589232f80dd5547fea9d88a13ff9f217b4fe124de4c24e13f57dd4ad4a"
      ]
    },
    "empty_content": {
      "status": "fail",
      "count": 1,
      "document_ids": [
        "5241c4b42c205fc2dc784e0d4700d76f70645eebb445c9368948c154d019ad92"
      ]
    },
    "short_chunks": {
      "status": "warning",
      "count": 1,
      "document_ids": [
        "b11b37cde4c884c418583ef1b3003e367ba24e80b12983f1188c88faab990918"
      ],
      "threshold": 20
    },
    "duplicate_chunks": {
      "status": "fail",
      "count": 1,
      "duplicates": [
        {
          "content_hash": "d509e5a2c3cca5a6ca1492380a149f80",
          "document_ids": [
            "1d48eac32b58322c065b46888b5b078e95c783b7d52a01ffa7c8aaa17234af70",
            "0c831f313e41d38cb742717723e5806227e9d1b96be54fe4f966a89fb91eb469"
          ]
        }
      ]
    },
    "missing_metadata": {
      "status": "pass",
      "count": 0,
      "details": []
    },
    "null_embeddings": {
      "status": "fail",
      "count": 7,
      "document_ids": [
        "335f72ebc7e221eb58b13f1e8b6a86e056cc79fc064c4539b461fe624f7b96d0",
        "1d48eac32b58322c065b46888b5b078e95c783b7d52a01ffa7c8aaa17234af70"
      ]
    },
    "embedding_dimension_mismatch": {
      "status": "pass",
      "count": 0,
      "expected_dimension": null,
      "actual_dimensions": {},
      "details": []
    }
  }
}