Ecosyste.ms: Timeline
Browse the timeline of events for every public repo on GitHub. Data updated hourly from GH Archive.
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
I actually believe `score_function` can be removed, as it belongs to the old way of passing the score function and was simply not removed. `task_name`: I would keep it minimal at this point. If ...
sam-hey opened an issue on lightonai/pylate
Add Recall Information for Index-Based Evaluation
Hi @noaha, I was wondering if it's possible to include recall information in the evaluation when using an index. I’m not sure if I’ve overlooked an existing implementation or if it’s not currently...
sam-hey opened a pull request on embeddings-benchmark/mteb
feat: Custom search interface
Adding a custom search interface to indicate that the model implements a custom search method. Ref: https://github.com/embeddings-benchmark/mteb/issues/1826 Enables: https://github.com/embeddin...
sam-hey pushed 45 commits to main sam-hey/mteb
- feat: add instruct wrapper (#1768) * add instruct wrapper * use get_task_instruction * add logging messages * app... ee0f15a
- 1.31.0 Automatically generated by python-semantic-release 3598b37
- doc: update pr template (#1871) * doc: update pr template * group testing & add: do not delete --------- Co-auth... 95714d0
- fix: fix jina v1, 2 models (#1872) fix jina models 1d66089
- 1.31.1 Automatically generated by python-semantic-release 7e7571e
- fix: update voyage exp metadata (#1888) * fix: update voyage exp metadata * aded number of parameters e623771
- 1.31.2 Automatically generated by python-semantic-release 21c7390
- fix: External results are preferred when only they have the needed splits (#1893) join_revisions now prefers task_re... 2a41730
- 1.31.3 Automatically generated by python-semantic-release e1be438
- fix: Allow aggregated tasks within benchmarks (#1771) * fix: Allow aggregated tasks within benchmarks Fixes #1231... 8fb59a4
- Update tasks table d939627
- Update tasks table 1ead72f
- Update tasks table cebf5b6
- Update tasks table 8bc101f
- Update tasks table e487eff
- Update tasks table 24d5373
- Update tasks table 2689cb8
- Update tasks table 50f305f
- Update tasks table e8b8ac0
- Update tasks table c77c82c
- and 25 more ...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
> Thanks for raising @sam-hey! > > I can definitely see the benefit! On the other hand, having it standardized makes it so each model class has the same function and is more reliable that way. ...
sam-hey created a comment on a pull request on embeddings-benchmark/leaderboard
Thanks a lot, @orionw! I’ve updated the files as suggested!
sam-hey opened a pull request on huggingface/datasets
fix: dill default for version bigger 0.3.8
Fixes def log for dill version >= 0.3.9
sam-hey opened an issue on lightonai/pylate
Error When Using Training Example on MPS Device
When attempting to train on an MPS device, the following error occurs due to an issue with `torch.compile()`: https://github.com/pytorch/pytorch/issues/96976 To resolve this issue, you can either...
sam-hey created a comment on an issue on lightonai/pylate
I believe this should be resolved in the next release of Pylate. https://github.com/lightonai/pylate/blob/e349918173eb785921dca36e55f91e5ab4ea0828/setup.py#L48
sam-hey opened an issue on lightonai/pylate
Error when creating `ModelCard`
Can be reproduced with: https://github.com/lightonai/pylate?tab=readme-ov-file#knowledge-distillation ``` Error while generating model card: Traceback (most recent call last): File "...venv-pyla...
sam-hey created a comment on an issue on embeddings-benchmark/mteb
As mentioned in https://github.com/embeddings-benchmark/mteb/pull/1869#issuecomment-2614282333, I suggest developing a more general method that is not limited to Cross-Encoders but can also be appl...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
Suggestion: If I understand the intent of this PR correctly, the primary focus is to enable filtering on the leaderboard. Wouldn't it make more sense to add a list of model types as an Enum? This w...
sam-hey created a comment on a pull request on embeddings-benchmark/leaderboard
@orionw, since I know you definitely have the correct rights, I hope you can help out! Thanks a lot!
sam-hey pushed 3 commits to main sam-hey/mteb-results
sam-hey created a comment on an issue on embeddings-benchmark/mteb
It could be a great addition to include a script in the results repository that generates results with the correct settings, such as having CodeCarbon activated, as @Samoed suggested. Additionally...
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
@Samoed, I would definitely encourage measuring this in the pipeline to ensure everyone is aware of any changes not covered by tests. However, we need to keep in mind that coverage can sometimes be...