Ecosyste.ms: Timeline
Browse the timeline of events for every public repo on GitHub. Data updated hourly from GH Archive.
sam-hey opened an issue on lightonai/pylate
Update similarity_fn_name = 'cosine' to max_sim
When loading a model, the `similarity_fn_name` attribute defaults to `cosine` instead of `max_sim`. Since `SentenceTransformer` does not support `max_sim` natively, this functionality must be imple...
sam-hey opened an issue on embeddings-benchmark/mteb
[v2] Refactor to Use Enums Instead of Literals: Removing Magic Literals
## Description: Currently, the codebase relies on literals to describe scoring functions and other constants (magic literals), which can make the code harder to understand and maintain. Using lite...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
> Not sure what you mean here. What is model in this case? When I run: When I run the following code: ``` from sentence_transformers import CrossEncoder, SentenceTransformer from mteb impor...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
@KennethEnevoldsen Currently, this issue affects MAC users with M1 chips or newer CPUs. Reverting to a non-default option could be a good solution if we prefer not to wait for the implementation of...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
> **fix:** Retrieve `name` and `revision` from SentenceTransformers metadata. > **Can't see where this happens? Why do we want this?** Currently, when passing a model, the old way MTEB crea...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
I completely agree with you that this is something worth addressing! However, I’d prefer to keep the scope of this PR as it is for now. At the moment, I don’t have a clear approach in mind for tack...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
By the way, this implementation also resolves the issue where MTEB fails to extract metadata from SentenceTransformers :)
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
What do you mean by duplicate? The code already existed, more or less. I just needed to fix the implementation because it was non-functional and assign the `similarity_fn_name` from st to the `Mode...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
> This currently doesn’t fix the issues with the similarity function, as it focuses only on the retrieval model wrapper. I think the changes should be made in `get_model`. The model name and revisi...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
I made some additional changes, so the PR ended up being a bit larger than originally planned. Please take a look at what I’ve done and share your feedback—I’m looking forward to hearing your thoug...
sam-hey pushed 1 commit to fix_contriever sam-hey/mteb
- fix model meta use new fn for sim operators 700ad58
sam-hey pushed 1 commit to fix_contriever sam-hey/mteb
- add mapping st sim fn name to mteb sim fn name 8ad1e88
sam-hey pushed 19 commits to fix_contriever sam-hey/mteb
- 1.28.2 Automatically generated by python-semantic-release a4975fe
- fix: Fixed definition of zero-shot in ModelMeta (#1747) * Corrected zero_shot definition to be based on task names... 407e205
- 1.28.3 Automatically generated by python-semantic-release edd9d7f
- fix: fixes implementation of similarity() (#1748) * fix(#1594): fixes implementation of similarity() * fix: add sim... 3fe9264
- 1.28.4 Automatically generated by python-semantic-release 75d78c1
- [v2] Remove memory usage (#1751) remove memory usage 4247e22
- fix: Leaderboard: `K` instead of `M` (#1761) Fixes #1752 972463e
- other: add script for leaderboard compare (#1758) * add script * remove changes * remove changes * add comm... 8bc80aa
- 1.28.5 Automatically generated by python-semantic-release cc27c78
- fix: added annotations for training data (#1742) * fix: Added annotations for arctic embed models * added google ... 3f093c8
- 1.28.6 Automatically generated by python-semantic-release c3b46b7
- [v2] fix contriever (add similarity_fn_name to ST wrapper) (#1749) * add dotwrapper * lint * make cleaner * add s... 2b41cb4
- [v2] Refactor evaluators and Abstasks (#1707) * refactor evaluators and tasks * remove slow/fast loading * fix imp... 91871fe
- Merge branch 'refs/heads/main' into v2.0.0 # Conflicts: # mteb/evaluation/evaluators/RetrievalEvaluator.py # mteb/mo... 997a135
- openai remove memory usage f73e7ac
- fix: _run_eval no co tracking 4c89681
- Merge remote-tracking branch 'mteb/v2.0.0' into fix_contriever fae6e31
- fix: bm25s 6298d75
- add enum to models 5a023d6
sam-hey created a comment on an issue on embeddings-benchmark/mteb
@orionw `ModelMeta.similarity_fn_name` is currently ignored and does not affect which similarity function is used. However, [#1749](https://github.com/embeddings-benchmark/mteb/pull/1749) has tempo...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
I believe this issue is critical: https://github.com/mlco2/codecarbon/issues/736. It could become a significant blocker for releasing v2 in the near future, as it affects all Mac users. cc @Kenn...
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
Yes,` _run_eval()` no longer takes in `output_folder`. However, since `co2_tracker=True` is now the default, no tests failed when updating the `_run_eval` function signature. Therefore, this al...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
Disabling `co2_tracker` by setting it to `False` resolves this specific issue for me. However, the tests continue to fail due to the default behavior of `co2_tracker`. I strongly suspect the root c...
sam-hey opened a pull request on embeddings-benchmark/mteb
fix: run eval no co2 tracking
Fixes _run_eval() Add test for this case ## Checklist - [ ] Run tests locally to make sure nothing is broken using `make test`. Not all -> https://github.com/embeddings-benchmark/mteb/issues/...
sam-hey created a branch on sam-hey/mteb
fix/run_eval_no_co2 - MTEB: Massive Text Embedding Benchmark
sam-hey pushed 13 commits to main sam-hey/mteb
- fix: Fixed task_type aggregation on leaderboard (#1746) * Fixed task_type aggregation in leaderboard * Fixed an e... 76bb070
- 1.28.2 Automatically generated by python-semantic-release a4975fe
- fix: Fixed definition of zero-shot in ModelMeta (#1747) * Corrected zero_shot definition to be based on task names... 407e205
- 1.28.3 Automatically generated by python-semantic-release edd9d7f
- fix: fixes implementation of similarity() (#1748) * fix(#1594): fixes implementation of similarity() * fix: add sim... 3fe9264
- 1.28.4 Automatically generated by python-semantic-release 75d78c1
- fix: Leaderboard: `K` instead of `M` (#1761) Fixes #1752 972463e
- other: add script for leaderboard compare (#1758) * add script * remove changes * remove changes * add comm... 8bc80aa
- 1.28.5 Automatically generated by python-semantic-release cc27c78
- fix: added annotations for training data (#1742) * fix: Added annotations for arctic embed models * added google ... 3f093c8
- 1.28.6 Automatically generated by python-semantic-release c3b46b7
- fix(#1594): fixes implementation of similarity() 3eef815
- fix: add similarity to SentenceTransformerWrapper 944bd05
sam-hey pushed 3 commits to fix_contriever sam-hey/mteb