Ecosyste.ms: Timeline
Browse the timeline of events for every public repo on GitHub. Data updated hourly from GH Archive.
sam-hey pushed 4 commits to main sam-hey/mteb
- fix: Fixed task_type aggregation on leaderboard (#1746) * Fixed task_type aggregation in leaderboard * Fixed an e... 23086af
- 1.28.2 Automatically generated by python-semantic-release 88143c7
- fix: Fixed definition of zero-shot in ModelMeta (#1747) * Corrected zero_shot definition to be based on task names... 0c9b9d8
- 1.28.3 Automatically generated by python-semantic-release dfc76d1
sam-hey pushed 78 commits to main sam-hey/mteb
- Add NanoBEIR Datasets (#1588) * add NanoClimateFeverRetrieval task, still requires some debugging * move task to co... 6731b94
- Update tasks table 9de7f20
- Feat: Evaluate missing languages (#1584) * init * fix tests * update mock retrieval * update tests * use subsets... 48cb97d
- Add IBM Granite Embedding Models (#1613) * add IBM granite embedding models * lint formatting * add adapted_from a... ad05983
- fix: disable co2_tracker for API models (#1614) 7c8e094
- 1.25.2 Automatically generated by python-semantic-release d8c015f
- fix: set `use_instructions` to True in models using prompts (#1616) feat: set `use_instructions` to True in models u... 0c44482
- 1.25.3 Automatically generated by python-semantic-release 2024338
- fix: override existing results (#1617) * fix override existing results * lint * fix tests * add tests with overwr... 272adb1
- 1.25.4 Automatically generated by python-semantic-release bd782d6
- add MSMARCO eval split in MTEB English (classic) benchmark (#1620) * add MSMARCO eval split in MTEB English (classic... e1b74f2
- fix: GermanDPR Dataset Causes Cross-Encoder Failure Due to Unexpected dict (#1621) Fixes #1609 748033e
- fix: properly add mteb_model_meta to model object (#1623) 72a457e
- 1.25.5 Automatically generated by python-semantic-release d8dd96c
- Feat: Add jasper (#1591) * init jasper * init jasper * add to overview * add to overview * remove some p... ef5a068
- fix: Update results_to_dataframe to use BenchmarkResults class (#1628) 02ae4fa
- 1.25.6 Automatically generated by python-semantic-release e8e1a50
- Speed up test_save_predictions (#1631) 1b06601
- fix: Correction of discrepancies for gte-Qweb model (#1637) 2de61b1
- 1.25.7 Automatically generated by python-semantic-release eb643a7
- and 58 more ...
sam-hey closed an issue on embeddings-benchmark/mteb
Discrepancy in Cross-Encoder Scores Between MTEB and SentenceTransformers
I conducted a comparison between the scores generated by MTEB for Cross-Encoders and those produced by the `sentence_transformers` library's `CrossEncoder` for the same model and dataset. Unfortuna...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
Happy to hear that v2 is close😊 ! All files are available in the [repository](https://github.com/sam-hey/mteb-cross-discrepancy/tree/cross-discrepancy). Check out [`eval_mteb_results.py`](htt...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
Sure @orionw, I hope you’re able to pinpoint the cause. Since I needed to modify some code, I forked the project and installed it from source. The last commit hash before my changes is: `ab8805c612...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
Hello @orionw , Thanks for reaching out so quickly! Unfortunately, the problem seems to be a bit more general. I updated the code to use the Cross-Encoder just as a reranker to match the origi...
sam-hey pushed 1 commit to cross-discrepancy sam-hey/mteb-cross-discrepancy
- use bm25 rerank new results d6b201c
sam-hey opened an issue on embeddings-benchmark/mteb
Discrepancy in Cross-Encoder Scores Between MTEB and SentenceTransformers
I conducted a comparison between the scores generated by MTEB for Cross-Encoders and those produced by the `sentence_transformers` library's `CrossEncoder` for the same model and dataset. Unfortuna...
sam-hey opened an issue on embeddings-benchmark/mteb
Add support for STS evaluation for Cross-Encoders
Currently, Cross-Encoders are not supported when using STS, and all models are treated as if they generate embeddings. This leads to suboptimal results without raising any errors. Including sup...