Ecosyste.ms: Timeline
Browse the timeline of events for every public repo on GitHub. Data updated hourly from GH Archive.
sam-hey pushed 56 commits to custom_search sam-hey/mteb
- fix: Ensure seed is based on RNG State (#1193) e2520df
- Consolidate Retrieval/Reranking/Instruction Variants (#1359) * update * merged retrieval; working * update tas... 9c58518
- fix: Unsure TaskResults can handle runtime and version being unspecified 2a8a370
- Merge branch 'v2.0.0' of https://github.com/embeddings-benchmark/mteb into v2.0.0 dea2b77
- fix: remove NaN handling for retrieval 23d6cb2
- Merge branch 'main' into v2.0.0 8868cd4
- fix: Merge main into v2 (#1454) * fix: Count unique texts, data leaks in calculate metrics (#1438) * add more sta... 5470c88
- feat: enable codecarbon by default (#1428) * enable codecarbon by default * lint * update flag * add allow_... 70a3ff2
- Add decriptive stat almost to all datasets (#1466) * run tasks * remove test script * lint * remove cache ... 0e9b6fd
- fix: Fix test for empty descriptive tasks (#1413) * fix test * skip mock * add message to assert * fix tes... 0a5bedb
- fix: pin datasets version <3.0.0 (#1471) 6da2a1a
- feat: Multilingual retrieval loader (#1473) * multilingual loader * lint a27de33
- fix: add citations to ModelMeta (#1477) * add citations * fix typo 0df0210
- Add descriptive stats to mising tasks and add number of qrels (#1476) * add code for comupting number of qrels * ... 0abe1a0
- 1475 add descriptive stats to all tasks v2 (#1482) * add code for comupting number of qrels * BibleNLPBitextMinin... a7a5214
- fix: Fix `BrightRetrieval` calculate stats (#1484) * fix bright loader * lint * fix comment 99247b2
- Merge main v2 (#1504) * fix: Count unique texts, data leaks in calculate metrics (#1438) * add more stat * ad... 022d355
- Fix: retrieval stats (#1496) * fix bright loader * lint * fix comment * fix stats * fix retrieval stats ... 6383950
- fix: hatespeech filipino (#1522) * fix FilipinoHateSpeechClassification * update tests d54fb75
- feat: Forbid task metadata and add upload functions (#1362) * init * find all wierd repos * move to mteb Wikipedia... dec5d6a
- and 36 more ...
sam-hey created a comment on an issue on lightonai/pylate
Hey, thanks for the response! I should probably rephrase my question a bit. What I mean is the recall for the index—because, as I understand it, with the index approach, many MaxSim operations aren...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
> Thanks @sam-hey for starting! > > Like I said last time, I think it would make more sense to define a new category of search for sparse models. That way we can add more than just BM25S (Pyseri...
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
Yes, I see your point. I'll integrate it into the interface.
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
> Shouldn't this be merged into the v2 branch? It doesn’t break any existing functions, so I wouldn’t say so.
sam-hey created a review comment on a pull request on embeddings-benchmark/mteb
I actually believe `score_function` can be removed, as it belongs to the old way of passing the score function and was simply not removed. `task_name`: I would keep it minimal at this point. If ...
sam-hey opened an issue on lightonai/pylate
Add Recall Information for Index-Based Evaluation
Hi @noaha, I was wondering if it's possible to include recall information in the evaluation when using an index. I’m not sure if I’ve overlooked an existing implementation or if it’s not currently...
sam-hey opened a pull request on embeddings-benchmark/mteb
feat: Custom search interface
Adding a custom search interface to indicate that the model implements a custom search method. Ref: https://github.com/embeddings-benchmark/mteb/issues/1826 Enables: https://github.com/embeddin...
sam-hey pushed 45 commits to main sam-hey/mteb
- feat: add instruct wrapper (#1768) * add instruct wrapper * use get_task_instruction * add logging messages * app... ee0f15a
- 1.31.0 Automatically generated by python-semantic-release 3598b37
- doc: update pr template (#1871) * doc: update pr template * group testing & add: do not delete --------- Co-auth... 95714d0
- fix: fix jina v1, 2 models (#1872) fix jina models 1d66089
- 1.31.1 Automatically generated by python-semantic-release 7e7571e
- fix: update voyage exp metadata (#1888) * fix: update voyage exp metadata * aded number of parameters e623771
- 1.31.2 Automatically generated by python-semantic-release 21c7390
- fix: External results are preferred when only they have the needed splits (#1893) join_revisions now prefers task_re... 2a41730
- 1.31.3 Automatically generated by python-semantic-release e1be438
- fix: Allow aggregated tasks within benchmarks (#1771) * fix: Allow aggregated tasks within benchmarks Fixes #1231... 8fb59a4
- Update tasks table d939627
- Update tasks table 1ead72f
- Update tasks table cebf5b6
- Update tasks table 8bc101f
- Update tasks table e487eff
- Update tasks table 24d5373
- Update tasks table 2689cb8
- Update tasks table 50f305f
- Update tasks table e8b8ac0
- Update tasks table c77c82c
- and 25 more ...
sam-hey created a comment on a pull request on embeddings-benchmark/mteb
> Thanks for raising @sam-hey! > > I can definitely see the benefit! On the other hand, having it standardized makes it so each model class has the same function and is more reliable that way. ...
sam-hey created a comment on a pull request on embeddings-benchmark/leaderboard
Thanks a lot, @orionw! I’ve updated the files as suggested!
sam-hey opened a pull request on huggingface/datasets
fix: dill default for version bigger 0.3.8
Fixes def log for dill version >= 0.3.9
sam-hey opened an issue on lightonai/pylate
Error When Using Training Example on MPS Device
When attempting to train on an MPS device, the following error occurs due to an issue with `torch.compile()`: https://github.com/pytorch/pytorch/issues/96976 To resolve this issue, you can either...
sam-hey created a comment on an issue on lightonai/pylate
I believe this should be resolved in the next release of Pylate. https://github.com/lightonai/pylate/blob/e349918173eb785921dca36e55f91e5ab4ea0828/setup.py#L48
sam-hey opened an issue on lightonai/pylate
Error when creating `ModelCard`
Can be reproduced with: https://github.com/lightonai/pylate?tab=readme-ov-file#knowledge-distillation ``` Error while generating model card: Traceback (most recent call last): File "...venv-pyla...