Ecosyste.ms: Timeline
Browse the timeline of events for every public repo on GitHub. Data updated hourly from GH Archive.
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
> it's a bit surprising that ModelMeta.similarity_fn_name isn't being utilized. We would love to switch to that one and would encourage a PR for this. It was only recently added to...
sam-hey opened an issue on embeddings-benchmark/mteb
[v2] Getting password prompt for task and tests
Branch: v2.0.0 Steps to Reproduce: ``` import mteb from mteb import MTEB model = mteb.get_model("intfloat/multilingual-e5-small") tasks = mteb.get_tasks(tasks=["NFCorpus"], languages=["en...
sam-hey pushed 1 commit to fix_contriever sam-hey/mteb
- ref: rename EvaluationFunction to ScoringFunction Co-authored-by: Isaac Chung <[email protected]> e4a692f
sam-hey pushed 1 commit to fix_contriever sam-hey/mteb
- ref: rename EvaluationFunction to ScoringFunction Co-authored-by: Isaac Chung <[email protected]> 6d04547
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
> What do you mean by `global similarity functions`? Consolidate all similarity functions into a single **new** file or retain them in `/mteb/evaluation/evaluators/utils.py` for better organizat...
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
I see your point, but I personally find it unintuitive to update the information in multiple places. In my opinion, there should be a single source of truth. Since ModelMeta already has the appropr...
sam-hey opened a pull request on embeddings-benchmark/mteb
add similarity_fn in ModelMeta
## Checklist <!-- Please do not delete this --> - [ ] Run tests locally to make sure nothing is broken using `make test`. - [x] Run the formatter to format the code using `make lint`. cc ...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
<img width="560" alt="image" src="https://github.com/user-attachments/assets/a96935ed-c410-43fa-bdb3-dd5ac2a7c03a" /> Please take a look at the screenshot. The MTEB `ModelMeta` is already availa...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
Just as an idea: I believe the `ModelMeta` object already has this information `similarity_fn_name="cosine"`. Why not add an Enum type to specify the evaluation function? This would make it much mo...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
@Samoed, if I’m not mistaken, the ability to use the `Dot product` seems to have been lost with this PR. Could you please confirm?
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
You are right, I hadn't considered this case. I've added a commit to address it
sam-hey pushed 1 commit to main sam-hey/mteb
- fix: add similarity to SentenceTransformerWrapper d37d526
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
Yes, this is correct, but I changed the code in `DRESModel` to handle this: [https://github.com/sam-hey/mteb/blob/main/mteb/evaluation/evaluators/RetrievalEvaluator.py#L379](https://github.com/s...
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
ColBERT does not implement SentenceTransformerWrapper, so it is not required at this point. The object mentioned above is of type DRESModel, which can handle this functionality.
sam-hey pushed 1 commit to main sam-hey/mteb
- Revert "fix(#1594): fixes implementation of similarity()" This reverts commit 0ebec781b3c000dc36f1dbf9dd59500d6f9ff555. 32a790f
sam-hey pushed 4 commits to main sam-hey/mteb
- fix: Fixed task_type aggregation on leaderboard (#1746) * Fixed task_type aggregation in leaderboard * Fixed an e... 23086af
- 1.28.2 Automatically generated by python-semantic-release 88143c7
- fix: Fixed definition of zero-shot in ModelMeta (#1747) * Corrected zero_shot definition to be based on task names... 0c9b9d8
- 1.28.3 Automatically generated by python-semantic-release dfc76d1
sam-hey pushed 78 commits to main sam-hey/mteb
- Add NanoBEIR Datasets (#1588) * add NanoClimateFeverRetrieval task, still requires some debugging * move task to co... 6731b94
- Update tasks table 9de7f20
- Feat: Evaluate missing languages (#1584) * init * fix tests * update mock retrieval * update tests * use subsets... 48cb97d
- Add IBM Granite Embedding Models (#1613) * add IBM granite embedding models * lint formatting * add adapted_from a... ad05983
- fix: disable co2_tracker for API models (#1614) 7c8e094
- 1.25.2 Automatically generated by python-semantic-release d8c015f
- fix: set `use_instructions` to True in models using prompts (#1616) feat: set `use_instructions` to True in models u... 0c44482
- 1.25.3 Automatically generated by python-semantic-release 2024338
- fix: override existing results (#1617) * fix override existing results * lint * fix tests * add tests with overwr... 272adb1
- 1.25.4 Automatically generated by python-semantic-release bd782d6
- add MSMARCO eval split in MTEB English (classic) benchmark (#1620) * add MSMARCO eval split in MTEB English (classic... e1b74f2
- fix: GermanDPR Dataset Causes Cross-Encoder Failure Due to Unexpected dict (#1621) Fixes #1609 748033e
- fix: properly add mteb_model_meta to model object (#1623) 72a457e
- 1.25.5 Automatically generated by python-semantic-release d8dd96c
- Feat: Add jasper (#1591) * init jasper * init jasper * add to overview * add to overview * remove some p... ef5a068
- fix: Update results_to_dataframe to use BenchmarkResults class (#1628) 02ae4fa
- 1.25.6 Automatically generated by python-semantic-release e8e1a50
- Speed up test_save_predictions (#1631) 1b06601
- fix: Correction of discrepancies for gte-Qweb model (#1637) 2de61b1
- 1.25.7 Automatically generated by python-semantic-release eb643a7
- and 58 more ...
sam-hey closed an issue on embeddings-benchmark/mteb
Discrepancy in Cross-Encoder Scores Between MTEB and SentenceTransformers
I conducted a comparison between the scores generated by MTEB for Cross-Encoders and those produced by the `sentence_transformers` library's `CrossEncoder` for the same model and dataset. Unfortuna...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
Happy to hear that v2 is close😊 ! All files are available in the [repository](https://github.com/sam-hey/mteb-cross-discrepancy/tree/cross-discrepancy). Check out [`eval_mteb_results.py`](htt...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
Sure @orionw, I hope you’re able to pinpoint the cause. Since I needed to modify some code, I forked the project and installed it from source. The last commit hash before my changes is: `ab8805c612...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
Hello @orionw , Thanks for reaching out so quickly! Unfortunately, the problem seems to be a bit more general. I updated the code to use the Cross-Encoder just as a reranker to match the origi...
sam-hey pushed 1 commit to cross-discrepancy sam-hey/mteb-cross-discrepancy
- use bm25 rerank new results d6b201c
sam-hey opened an issue on embeddings-benchmark/mteb
Discrepancy in Cross-Encoder Scores Between MTEB and SentenceTransformers
I conducted a comparison between the scores generated by MTEB for Cross-Encoders and those produced by the `sentence_transformers` library's `CrossEncoder` for the same model and dataset. Unfortuna...