intfloat/multilingual-e5-large-instruct

By intfloat

๐ŸŽฏ Task: Feature Extractionโš–๏ธ mit๐Ÿ“ฆ sentence-transformers

Model Card

Multilingual-E5-large-instruct

Multilingual E5 Text Embeddings: A Technical Report. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, arXiv 2024

This model has 24 layers and the embedding size is 1024.

Usage

Below are examples to encode queries and passages from the MS-MARCO passage ranking dataset.

Transformers

import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel


def average_pool(last_hidden_states: Tensor,
                 attention_mask: Tensor) -> Tensor:
    last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]

def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery: {query}'

# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [
    get_detailed_instruct(task, 'how much protein should a female eat'),
    get_detailed_instruct(task, 'ๅ—็“œ็š„ๅฎถๅธธๅšๆณ•')
]
# No need to add instruction for retrieval documents
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "1.ๆธ…็‚’ๅ—็“œไธ ๅŽŸๆ–™:ๅซฉๅ—็“œๅŠไธช ่ฐƒๆ–™:่‘ฑใ€็›ใ€็™ฝ็ณ–ใ€้ธก็ฒพ ๅšๆณ•: 1ใ€ๅ—็“œ็”จๅˆ€่–„่–„็š„ๅ‰ŠๅŽป่กจ้ขไธ€ๅฑ‚็šฎ,็”จๅ‹บๅญๅˆฎๅŽป็“ค 2ใ€ๆ“ฆๆˆ็ป†ไธ(ๆฒกๆœ‰ๆ“ฆ่œๆฟๅฐฑ็”จๅˆ€ๆ…ขๆ…ขๅˆ‡ๆˆ็ป†ไธ) 3ใ€้”…็ƒง็ƒญๆ”พๆฒน,ๅ…ฅ่‘ฑ่Šฑ็…ธๅ‡บ้ฆ™ๅ‘ณ 4ใ€ๅ…ฅๅ—็“œไธๅฟซ้€Ÿ็ฟป็‚’ไธ€ๅˆ†้’Ÿๅทฆๅณ,ๆ”พ็›ใ€ไธ€็‚น็™ฝ็ณ–ๅ’Œ้ธก็ฒพ่ฐƒๅ‘ณๅ‡บ้”… 2.้ฆ™่‘ฑ็‚’ๅ—็“œ ๅŽŸๆ–™:ๅ—็“œ1ๅช ่ฐƒๆ–™:้ฆ™่‘ฑใ€่’œๆœซใ€ๆฉ„ๆฆ„ๆฒนใ€็› ๅšๆณ•: 1ใ€ๅฐ†ๅ—็“œๅŽป็šฎ,ๅˆ‡ๆˆ็‰‡ 2ใ€ๆฒน้”…8ๆˆ็ƒญๅŽ,ๅฐ†่’œๆœซๆ”พๅ…ฅ็ˆ†้ฆ™ 3ใ€็ˆ†้ฆ™ๅŽ,ๅฐ†ๅ—็“œ็‰‡ๆ”พๅ…ฅ,็ฟป็‚’ 4ใ€ๅœจ็ฟป็‚’็š„ๅŒๆ—ถ,ๅฏไปฅไธๆ—ถๅœฐๅพ€้”…้‡ŒๅŠ ๆฐด,ไฝ†ไธ่ฆๅคชๅคš 5ใ€ๆ”พๅ…ฅ็›,็‚’ๅŒ€ 6ใ€ๅ—็“œๅทฎไธๅคš่ฝฏๅ’Œ็ปตไบ†ไน‹ๅŽ,ๅฐฑๅฏไปฅๅ…ณ็ซ 7ใ€ๆ’’ๅ…ฅ้ฆ™่‘ฑ,ๅณๅฏๅ‡บ้”…"
]
input_texts = queries + documents

tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-large-instruct')
model = AutoModel.from_pretrained('intfloat/multilingual-e5-large-instruct')

# Tokenize the input texts
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt')

outputs = model(**batch_dict)
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

# normalize embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
# => [[91.92852783203125, 67.580322265625], [70.3814468383789, 92.1330795288086]]

Sentence Transformers

from sentence_transformers import SentenceTransformer

def get_detailed_instruct(task_description: str, query: str) -> str:
    return f'Instruct: {task_description}\nQuery: {query}'

# Each query must come with a one-sentence instruction that describes the task
task = 'Given a web search query, retrieve relevant passages that answer the query'
queries = [
    get_detailed_instruct(task, 'how much protein should a female eat'),
    get_detailed_instruct(task, 'ๅ—็“œ็š„ๅฎถๅธธๅšๆณ•')
]
# No need to add instruction for retrieval documents
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "1.ๆธ…็‚’ๅ—็“œไธ ๅŽŸๆ–™:ๅซฉๅ—็“œๅŠไธช ่ฐƒๆ–™:่‘ฑใ€็›ใ€็™ฝ็ณ–ใ€้ธก็ฒพ ๅšๆณ•: 1ใ€ๅ—็“œ็”จๅˆ€่–„่–„็š„ๅ‰ŠๅŽป่กจ้ขไธ€ๅฑ‚็šฎ,็”จๅ‹บๅญๅˆฎๅŽป็“ค 2ใ€ๆ“ฆๆˆ็ป†ไธ(ๆฒกๆœ‰ๆ“ฆ่œๆฟๅฐฑ็”จๅˆ€ๆ…ขๆ…ขๅˆ‡ๆˆ็ป†ไธ) 3ใ€้”…็ƒง็ƒญๆ”พๆฒน,ๅ…ฅ่‘ฑ่Šฑ็…ธๅ‡บ้ฆ™ๅ‘ณ 4ใ€ๅ…ฅๅ—็“œไธๅฟซ้€Ÿ็ฟป็‚’ไธ€ๅˆ†้’Ÿๅทฆๅณ,ๆ”พ็›ใ€ไธ€็‚น็™ฝ็ณ–ๅ’Œ้ธก็ฒพ่ฐƒๅ‘ณๅ‡บ้”… 2.้ฆ™่‘ฑ็‚’ๅ—็“œ ๅŽŸๆ–™:ๅ—็“œ1ๅช ่ฐƒๆ–™:้ฆ™่‘ฑใ€่’œๆœซใ€ๆฉ„ๆฆ„ๆฒนใ€็› ๅšๆณ•: 1ใ€ๅฐ†ๅ—็“œๅŽป็šฎ,ๅˆ‡ๆˆ็‰‡ 2ใ€ๆฒน้”…8ๆˆ็ƒญๅŽ,ๅฐ†่’œๆœซๆ”พๅ…ฅ็ˆ†้ฆ™ 3ใ€็ˆ†้ฆ™ๅŽ,ๅฐ†ๅ—็“œ็‰‡ๆ”พๅ…ฅ,็ฟป็‚’ 4ใ€ๅœจ็ฟป็‚’็š„ๅŒๆ—ถ,ๅฏไปฅไธๆ—ถๅœฐๅพ€้”…้‡ŒๅŠ ๆฐด,ไฝ†ไธ่ฆๅคชๅคš 5ใ€ๆ”พๅ…ฅ็›,็‚’ๅŒ€ 6ใ€ๅ—็“œๅทฎไธๅคš่ฝฏๅ’Œ็ปตไบ†ไน‹ๅŽ,ๅฐฑๅฏไปฅๅ…ณ็ซ 7ใ€ๆ’’ๅ…ฅ้ฆ™่‘ฑ,ๅณๅฏๅ‡บ้”…"
]
input_texts = queries + documents

model = SentenceTransformer('intfloat/multilingual-e5-large-instruct')

embeddings = model.encode(input_texts, convert_to_tensor=True, normalize_embeddings=True)
scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())
# [[91.92853546142578, 67.5802993774414], [70.38143157958984, 92.13307189941406]]

Infinity

Usage with Infinity:

docker run --gpus all -v $PWD/data:/app/.cache -e HF_TOKEN=$HF_TOKEN -p "7997":"7997" \
michaelf34/infinity:0.0.68 \
v2 --model-id intfloat/multilingual-e5-large-instruct --revision "main" --dtype float16 --batch-size 32 --engine torch --port 7997

Supported Languages

This model is initialized from xlm-roberta-large and continually trained on a mixture of multilingual datasets. It supports 100 languages from xlm-roberta, but low-resource languages may see performance degradation.

Training Details

Initialization: xlm-roberta-large

First stage: contrastive pre-training with 1 billion weakly supervised text pairs.

Second stage: fine-tuning on datasets from the E5-mistral paper.

MTEB Benchmark Evaluation

Check out unilm/e5 to reproduce evaluation results on the BEIR and MTEB benchmark.

FAQ

1. Do I need to add instructions to the query?

Yes, this is how the model is trained, otherwise you will see a performance degradation. The task definition should be a one-sentence instruction that describes the task. This is a way to customize text embeddings for different scenarios through natural language instructions.

Please check out unilm/e5/utils.py for instructions we used for evaluation.

On the other hand, there is no need to add instructions to the document side.

2. Why are my reproduced results slightly different from reported in the model card?

Different versions of transformers and pytorch could cause negligible but non-zero performance differences.

3. Why does the cosine similarity scores distribute around 0.7 to 1.0?

This is a known and expected behavior as we use a low temperature 0.01 for InfoNCE contrastive loss.

For text embedding tasks like text retrieval or semantic similarity, what matters is the relative order of the scores instead of the absolute values, so this should not be an issue.

Citation

If you find our paper or models helpful, please consider cite as follows:

@article{wang2024multilingual,
  title={Multilingual E5 Text Embeddings: A Technical Report},
  author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Yang, Linjun and Majumder, Rangan and Wei, Furu},
  journal={arXiv preprint arXiv:2402.05672},
  year={2024}
}

Limitations

Long texts will be truncated to at most 512 tokens.

Architecture & Tags

sentence-transformersonnxsafetensorsxlm-robertafeature-extractionmtebtransformersmultilingualafamarasazbebgbnbrbscacscydadeeleneoeseteufafifrfygagdglguhahehihrhuhyidisitjajvkakkkmknkokukylaloltlvmgmkmlmnmrmsmynenlnoomorpaplpsptrorusasdsiskslsosqsrsusvswtatethtltrugukuruzvixhyizhmodel-indexautotrain_compatibletext-embeddings-inferenceendpoints_compatibleregion:us