Ecosyste.ms: Timeline
Browse the timeline of events for every public repo on GitHub. Data updated hourly from GH Archive.
sam-hey opened a pull request on embeddings-benchmark/mteb
fix: run eval no co2 tracking
Fixes _run_eval() Add test for this case ## Checklist - [ ] Run tests locally to make sure nothing is broken using `make test`. Not all -> https://github.com/embeddings-benchmark/mteb/issues/...
sam-hey created a branch on sam-hey/mteb
fix/run_eval_no_co2 - MTEB: Massive Text Embedding Benchmark
sam-hey pushed 13 commits to main sam-hey/mteb
- fix: Fixed task_type aggregation on leaderboard (#1746) * Fixed task_type aggregation in leaderboard * Fixed an e... 76bb070
- 1.28.2 Automatically generated by python-semantic-release a4975fe
- fix: Fixed definition of zero-shot in ModelMeta (#1747) * Corrected zero_shot definition to be based on task names... 407e205
- 1.28.3 Automatically generated by python-semantic-release edd9d7f
- fix: fixes implementation of similarity() (#1748) * fix(#1594): fixes implementation of similarity() * fix: add sim... 3fe9264
- 1.28.4 Automatically generated by python-semantic-release 75d78c1
- fix: Leaderboard: `K` instead of `M` (#1761) Fixes #1752 972463e
- other: add script for leaderboard compare (#1758) * add script * remove changes * remove changes * add comm... 8bc80aa
- 1.28.5 Automatically generated by python-semantic-release cc27c78
- fix: added annotations for training data (#1742) * fix: Added annotations for arctic embed models * added google ... 3f093c8
- 1.28.6 Automatically generated by python-semantic-release c3b46b7
- fix(#1594): fixes implementation of similarity() 3eef815
- fix: add similarity to SentenceTransformerWrapper 944bd05
sam-hey pushed 3 commits to fix_contriever sam-hey/mteb
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
> it's a bit surprising that ModelMeta.similarity_fn_name isn't being utilized. We would love to switch to that one and would encourage a PR for this. It was only recently added to...
sam-hey opened an issue on embeddings-benchmark/mteb
[v2] Getting password prompt for task and tests
Branch: v2.0.0 Steps to Reproduce: ``` import mteb from mteb import MTEB model = mteb.get_model("intfloat/multilingual-e5-small") tasks = mteb.get_tasks(tasks=["NFCorpus"], languages=["en...
sam-hey pushed 1 commit to fix_contriever sam-hey/mteb
- ref: rename EvaluationFunction to ScoringFunction Co-authored-by: Isaac Chung <[email protected]> e4a692f
sam-hey pushed 1 commit to fix_contriever sam-hey/mteb
- ref: rename EvaluationFunction to ScoringFunction Co-authored-by: Isaac Chung <[email protected]> 6d04547
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
> What do you mean by `global similarity functions`? Consolidate all similarity functions into a single **new** file or retain them in `/mteb/evaluation/evaluators/utils.py` for better organizat...
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
I see your point, but I personally find it unintuitive to update the information in multiple places. In my opinion, there should be a single source of truth. Since ModelMeta already has the appropr...
sam-hey opened a pull request on embeddings-benchmark/mteb
add similarity_fn in ModelMeta
## Checklist <!-- Please do not delete this --> - [ ] Run tests locally to make sure nothing is broken using `make test`. - [x] Run the formatter to format the code using `make lint`. cc ...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
<img width="560" alt="image" src="https://github.com/user-attachments/assets/a96935ed-c410-43fa-bdb3-dd5ac2a7c03a" /> Please take a look at the screenshot. The MTEB `ModelMeta` is already availa...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
Just as an idea: I believe the `ModelMeta` object already has this information `similarity_fn_name="cosine"`. Why not add an Enum type to specify the evaluation function? This would make it much mo...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
@Samoed, if I’m not mistaken, the ability to use the `Dot product` seems to have been lost with this PR. Could you please confirm?
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
You are right, I hadn't considered this case. I've added a commit to address it
sam-hey pushed 1 commit to main sam-hey/mteb
- fix: add similarity to SentenceTransformerWrapper d37d526
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
Yes, this is correct, but I changed the code in `DRESModel` to handle this: [https://github.com/sam-hey/mteb/blob/main/mteb/evaluation/evaluators/RetrievalEvaluator.py#L379](https://github.com/s...
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
ColBERT does not implement SentenceTransformerWrapper, so it is not required at this point. The object mentioned above is of type DRESModel, which can handle this functionality.
sam-hey pushed 1 commit to main sam-hey/mteb
- Revert "fix(#1594): fixes implementation of similarity()" This reverts commit 0ebec781b3c000dc36f1dbf9dd59500d6f9ff555. 32a790f