Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pairwise): allow passing evaluator directly to wait for evaluation to complete #681

Merged
merged 2 commits into from May 13, 2024

Conversation

dqbd
Copy link
Collaborator

@dqbd dqbd commented May 10, 2024

Allows the following syntax

const pairwise = await evaluateComparative(
  [
    evaluate((input) => ({ foo: `first:${input.input}` }), {
      data: TESTING_DATASET_NAME,
    }),
    evaluate((input) => ({ foo: `second:${input.input}` }), {
      data: TESTING_DATASET_NAME,
    }),
  ],
  {
    evaluators: [
      (runs) => ({
        key: "latter_precedence",
        scores: Object.fromEntries(runs.map((run, i) => [run.id, i % 2])),
      }),
    ],
  }
);

@dqbd dqbd requested a review from hinthornw May 10, 2024 15:41
@dqbd dqbd requested a review from hinthornw May 10, 2024 18:12
@dqbd dqbd force-pushed the dqbd/evaluate-comparative-url branch from 7f2debb to 4718dab Compare May 10, 2024 19:09
@dqbd dqbd force-pushed the dqbd/pairwise-pass-evaluator-directly branch from d41bd54 to f4ea490 Compare May 10, 2024 19:09
Base automatically changed from dqbd/evaluate-comparative-url to main May 13, 2024 02:21
@dqbd dqbd merged commit 676f335 into main May 13, 2024
20 checks passed
@dqbd dqbd deleted the dqbd/pairwise-pass-evaluator-directly branch May 13, 2024 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants