fix: change default similarity_metric for milvus by e7217 · Pull Request #1000 · Marker-Inc-Korea/AutoRAG

e7217 · 2024-11-25T05:46:23Z

description
Milvus not supported similarity_metric:cosine and only L2 and IP are supported. Therefore, I updated the default similarity_metric.

log

[11/25/24 14:28:57] INFO     [evaluator.py:127] >>                                                                                                                  evaluator.py:127
                                             _        _____            _____                                                                                                        
                                  /\        | |      |  __ \     /\   / ____|                                                                                                       
                                 /  \  _   _| |_ ___ | |__) |   /  \ | |  __                                                                                                        
                                / /\ \| | | | __/ _ \|  _  /   / /\ \| | |_ |                                                                                                       
                               / ____ \ |_| | || (_) | | \ \  / ____ \ |__| |                                                                                                       
                              /_/    \_\__,_|\__\___/|_|  \_\/_/    \_\_____|                                                                                                       
                                                                                                                                                                                    
                                                                                                                                                                                    
                    INFO     [evaluator.py:128] >> Start Validation input data and config YAML file first. If you want to skip this, put the --skip_validation flag evaluator.py:128
                             or `skip_validation` at the start_trial function.                                                                                                      
                    WARNING  [validator.py:50] >> Minimal Requested sample size (5) is larger than available records (1). Sampling will be limited to 1 records.     validator.py:50
[11/25/24 14:28:58] INFO     [evaluator.py:228] >> Embedding BM25 corpus...                                                                                         evaluator.py:228
[11/25/24 14:29:10] INFO     [evaluator.py:248] >> BM25 corpus embedding complete.                                                                                  evaluator.py:248
                    INFO     [_client.py:1039] >> HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"                                          _client.py:1039
[11/25/24 14:29:11] ERROR    [decorators.py:147] >> RPC error: [create_index], <MilvusException: (code=0, message=metric type not found or not supported,          decorators.py:147
                             supported: [L2 IP])>, <Time:{'RPC start': '2024-11-25 14:29:11.011688', 'RPC error': '2024-11-25 14:29:11.017874'}>

vkehfdl1 · 2024-11-25T11:16:57Z

Hi, I thought Milvus is supporting cosine similarity metric according to the docs of Milvus.

vkehfdl1 · 2024-11-25T11:17:20Z

@e7217 But is it solved your Milvus issue? Then I will happy to merge it.

e7217 · 2024-11-25T11:30:23Z

@vkehfdl1 I made an assumption based on the log messages. From the log, it appears that it shows supported: [L2 IP], which led me to conclude that the cosine similarity method is not supported. My assumption could be incorrect, but the issue was resolved when I proceeded this way. To ensure proper functionality, the parameters need to be defined. #1001
Thank you!

https://github.com/langchain-ai/langchain-milvus/blob/288f5197f1e68d1cc98a416e0f7b8f3a6a3a9517/libs/milvus/langchain_milvus/vectorstores/milvus.py#L1421

e7217 · 2024-11-25T11:49:54Z

@vkehfdl1 I made an assumption based on the log messages. From the log, it appears that it shows supported: [L2 IP], which led me to conclude that the cosine similarity method is not supported. My assumption could be incorrect, but the issue was resolved when I proceeded this way. To ensure proper functionality, the parameters need to be defined. #1001 Thank you!

https://github.com/langchain-ai/langchain-milvus/blob/288f5197f1e68d1cc98a416e0f7b8f3a6a3a9517/libs/milvus/langchain_milvus/vectorstores/milvus.py#L1421

@vkehfdl1 Sorry, I found that the default metric type is COSINE in the pymilvus repo.
https://github.com/milvus-io/pymilvus/blob/46f12b21c80efdfc718bc2ef86d867b710a06f1c/pymilvus/milvus_client/milvus_client.py#L65
I don’t know why this issue occurred, but other metric types, except for COSINE, were configured successfully.

e7217 · 2024-11-25T12:16:36Z

@vkehfdl1 I made an assumption based on the log messages. From the log, it appears that it shows supported: [L2 IP], which led me to conclude that the cosine similarity method is not supported. My assumption could be incorrect, but the issue was resolved when I proceeded this way. To ensure proper functionality, the parameters need to be defined. #1001 Thank you!
https://github.com/langchain-ai/langchain-milvus/blob/288f5197f1e68d1cc98a416e0f7b8f3a6a3a9517/libs/milvus/langchain_milvus/vectorstores/milvus.py#L1421

@vkehfdl1 Sorry, I found that the default metric type is COSINE in the pymilvus repo. https://github.com/milvus-io/pymilvus/blob/46f12b21c80efdfc718bc2ef86d867b710a06f1c/pymilvus/milvus_client/milvus_client.py#L65 I don’t know why this issue occurred, but other metric types, except for COSINE, were configured successfully.

@vkehfdl1 This issue occurred because the Milvus server version is lower than 2.3.0. For those who are experiencing the same issue, I hope this information helps. Thank you.
milvus-io/pymilvus#1728

elsatch · 2024-12-03T00:09:32Z

This fix might add another error as side effect, regarding the default metric for milvus search.

I am trying to use milvus-lite (that is supported by pymilvus 2.4.2).

I am able to ingest correctly my data using the following config:

vectordb:
  - name: bge_m3_milvus
    db_type: milvus
    embedding_model: huggingface_bge_m3
    collection_name: huggingface_bge_m3
    uri: local_milvus.db
    embedding_batch: 50
    index_type: FLAT
    params:
      metric_type: ""

But when the search module kicks in:

[12/03/24 00:51:04] INFO     [_client.py:1038] >> HTTP Request: GET https://api.gradio.app/gradio-messaging/en "HTTP/1.1    _client.py:1038
                             200 OK"
[12/03/24 00:51:08] INFO     [evaluator.py:228] >> Embedding BM25 corpus...                                                evaluator.py:228
                    INFO     [evaluator.py:248] >> BM25 corpus embedding complete.                                         evaluator.py:248
                    INFO     [SentenceTransformer.py:218] >> Load pretrained SentenceTransformer: BAAI/bge-m3    SentenceTransformer.py:218
[12/03/24 00:51:10] INFO     [SentenceTransformer.py:357] >> 2 prompts are loaded, with the keys: ['query',      SentenceTransformer.py:357
                             'text']
                    INFO     [connections.py:381] >> Pass in the local path local_milvus.db, and run it using            connections.py:381
                             milvus-lite
[12/03/24 00:51:12] INFO     [evaluator.py:205] >> Running node line retrieve_node_line...                                 evaluator.py:205
                    INFO     [node.py:55] >> Running node retrieval...                                                           node.py:55
                    INFO     [run.py:165] >> Running retrieval node - semantic retrieval module...                               run.py:165
                    INFO     [base.py:18] >> Initialize retrieval node - VectorDB                                                base.py:18
                    INFO     [connections.py:381] >> Pass in the local path local_milvus.db, and run it using            connections.py:381
                             milvus-lite
                    INFO     [base.py:31] >> Running retrieval node - VectorDB module...                                         base.py:31
Batches:   0%|                                                                                                       | 0/1 [00:00<?, ?it/s]
Batches: 100%|###############################################################################################| 1/1 [00:00<00:00, 10.55it/s]

Batches:   0%|                                                                                                       | 0/1 [00:00<?, ?it/s]
Batches: 100%|##############################################################################################| 1/1 [00:00<00:00, 137.38it/s]

Batches:   0%|                                                                                                       | 0/1 [00:00<?, ?it/s]
Batches: 100%|##############################################################################################| 1/1 [00:00<00:00, 174.38it/s]

Batches:   0%|                                                                                                       | 0/1 [00:00<?, ?it/s]
Batches: 100%|##############################################################################################| 1/1 [00:00<00:00, 187.40it/s]

Batches:   0%|                                                                                                       | 0/1 [00:00<?, ?it/s]
Batches: 100%|##############################################################################################| 1/1 [00:00<00:00, 182.19it/s]

                    ERROR    [decorators.py:147] >> RPC error: [search], <MilvusException: (code=1100, message=fail to    decorators.py:147
                             search: metric type not match: invalid [expected=][actual=L2]: invalid parameter)>,
                             <Time:{'RPC start': '2024-12-03 00:51:12.248113', 'RPC error': '2024-12-03
                             00:51:12.248668'}>
                    ERROR    [decorators.py:147] >> RPC error: [search], <MilvusException: (code=1100, message=fail to    decorators.py:147
                             search: metric type not match: invalid [expected=][actual=L2]: invalid parameter)>,
                             <Time:{'RPC start': '2024-12-03 00:51:12.251065', 'RPC error': '2024-12-03
                             00:51:12.251423'}>
                    ERROR    [decorators.py:147] >> RPC error: [search], <MilvusException: (code=1100, message=fail to    decorators.py:147
                             search: metric type not match: invalid [expected=][actual=L2]: invalid parameter)>,
                             <Time:{'RPC start': '2024-12-03 00:51:12.253529', 'RPC error': '2024-12-03
                             00:51:12.253798'}>
                    ERROR    [decorators.py:147] >> RPC error: [search], <MilvusException: (code=1100, message=fail to    decorators.py:147
                             search: metric type not match: invalid [expected=][actual=L2]: invalid parameter)>,
                             <Time:{'RPC start': '2024-12-03 00:51:12.255825', 'RPC error': '2024-12-03
                             00:51:12.256095'}>
                    ERROR    [decorators.py:147] >> RPC error: [search], <MilvusException: (code=1100, message=fail to    decorators.py:147
                             search: metric type not match: invalid [expected=][actual=L2]: invalid parameter)>,
                             <Time:{'RPC start': '2024-12-03 00:51:12.258249', 'RPC error': '2024-12-03
                             00:51:12.258537'}>
Ingesting VectorDB... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 1/1 0:00:03
Evaluating...         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0/4 0:00:00

As you can see in the error message, the L2 metric is passed for search, even if metric_type param is set to empty string.

vkehfdl1 · 2024-12-03T03:28:29Z

@elsatch Hi! What will be the distance metric do you want to use? You can configure it through "similarity_metric" key in the config YAML file.

vectordb:
  - name: bge_m3_milvus
    db_type: milvus
    embedding_model: huggingface_bge_m3
    collection_name: huggingface_bge_m3
    uri: local_milvus.db
    embedding_batch: 50
    index_type: FLAT
    similarity_metric: cosine

Like this!
The default metric is cosine in this docs.

If you want to "already ingested" vectordb, simply use below.

vectordb:
  - name: bge_m3_milvus
    db_type: milvus
    embedding_model: huggingface_bge_m3
    collection_name: huggingface_bge_m3
    uri: local_milvus.db
    embedding_batch: 50
    index_type: FLAT
    similarity_metric: l2

e7217 · 2024-12-03T05:14:45Z

@elsatch
Hello,
The vllm module configuration has been updated as follows to accommodate various options:

vectordb:
- name: autorag_2024_xx_xx
  db_type: milvus
  embedding_model: openai
  collection_name: autorag_2024_xx_xx
  uri: http://192.xxx.xxx.xxx:19530
  embedding_batch: 50
  similarity_metric: l2
  index_type: IVF_FLAT
  params: 
    nlist : 16384

Please refer to the following PR for details:
#1001

@vkehfdl1
Since the official documentation has not yet been updated, I will incorporate these changes accordingly. Additionally, in the updated code, the default similarity_metric is set to l2, but I will change it to cosine.

e7217 · 2024-12-03T09:02:28Z

@elsatch Hello, The vllm module configuration has been updated as follows to accommodate various options:
vectordb:
- name: autorag_2024_xx_xx
  db_type: milvus
  embedding_model: openai
  collection_name: autorag_2024_xx_xx
  uri: http://192.xxx.xxx.xxx:19530
  embedding_batch: 50
  similarity_metric: l2
  index_type: IVF_FLAT
  params: 
    nlist : 16384
Please refer to the following PR for details: #1001

@vkehfdl1 Since the official documentation has not yet been updated, I will incorporate these changes accordingly. Additionally, in the updated code, the default similarity_metric is set to l2, but I will change it to cosine.

#1030

fix: change default similarity_metric for milvus

95fbe07

e7217 closed this Nov 25, 2024

e7217 deleted the fix/change-default-similarity-metric-milvus branch November 27, 2024 09:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: change default similarity_metric for milvus#1000

fix: change default similarity_metric for milvus#1000
e7217 wants to merge 1 commit intoMarker-Inc-Korea:mainfrom
e7217:fix/change-default-similarity-metric-milvus

e7217 commented Nov 25, 2024 •

edited

Loading

Uh oh!

vkehfdl1 commented Nov 25, 2024

Uh oh!

vkehfdl1 commented Nov 25, 2024

Uh oh!

e7217 commented Nov 25, 2024 •

edited

Loading

Uh oh!

e7217 commented Nov 25, 2024

Uh oh!

e7217 commented Nov 25, 2024 •

edited

Loading

Uh oh!

elsatch commented Dec 3, 2024

Uh oh!

vkehfdl1 commented Dec 3, 2024

Uh oh!

e7217 commented Dec 3, 2024 •

edited

Loading

Uh oh!

e7217 commented Dec 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

e7217 commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vkehfdl1 commented Nov 25, 2024

Uh oh!

vkehfdl1 commented Nov 25, 2024

Uh oh!

e7217 commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

e7217 commented Nov 25, 2024

Uh oh!

e7217 commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elsatch commented Dec 3, 2024

Uh oh!

vkehfdl1 commented Dec 3, 2024

Uh oh!

e7217 commented Dec 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

e7217 commented Dec 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

e7217 commented Nov 25, 2024 •

edited

Loading

e7217 commented Nov 25, 2024 •

edited

Loading

e7217 commented Nov 25, 2024 •

edited

Loading

e7217 commented Dec 3, 2024 •

edited

Loading