jasoncorkill (Jason Corkill)

posted an update about 1 month ago

Post

681

Researchers shouldn’t have to limit experiments because human feedback is too expensive/slow to collect.

So, @RapidataAI is offering up to $50,000 in human annotation credits per research project (can be more for selected topics).

Learn more on our website, and apply via email, we'll do our best to get back to you within 2 days :).

replied to their post about 1 month ago

Thanks for your AI break down. We share the raw results on purpose as we also share the raw question, the table would be inconsistent if we reversed it there. For the ELO updates this was reversed.

If you go to benchmark.ai/svg you can see the per axis aggregation.

posted an update about 1 month ago

Post

2597

Most public benchmarks collapse model performance into one broad preference signal.

That makes it hard to understand which capabilities differentiate between models. It's also almost impossible to inspect the evidence behind it. So @RapidataAI is releasing Benchmark.AI.

We started with an SVG generation benchmark including 42 models, 500 prompts, 1.9M+ human judgements, 300K+ match-ups.

We evaluate models separately on Preference, Alignment and Coherence, while making the prompts, outputs, match-ups and methodology public.

Full dataset: Rapidata/svg-benchmark
Full benchmark: https://www.benchmark.ai/svg

Methodology feedback and benchmark suggestions very welcome!

13 replies

·

replied to their post 2 months ago

https://luma.com/znmf2l7b

posted an update 2 months ago

Post

103

Are you an AI Researcher, Enthusiast, Founder or else and live in or close by to Zürich? Drop in for an Aperol Spritz and Euromaxx on our Balcony on the 14th of August.

1 reply

·

posted an update 11 months ago

Post

4751

Do you remember https://thispersondoesnotexist.com/ ? It was one of the first cases where the future of generative media really hit us. Humans are incredibly good at recognizing and analyzing faces, so they are a very good litmus test for any generative image model.

But none of the current benchmarks measure the ability of models to generate humans independently. So we built our own. We measure the models ability to generate a diverse set of human faces and using over 20'000 human annotations we ranked all of the major models on their ability to generate faces. Find the full ranking here:
https://app.rapidata.ai/mri/benchmarks/68af24ae74482280b62f7596

We have release the full underlying data publicly here on huggingface: Rapidata/Face_Generation_Benchmark

replied to their post about 1 year ago

Perhaps we can provide a couple of thousand human annotations

replied to their post about 1 year ago

Interesting, what kind of data are you collecting?

replied to their post about 1 year ago

Funny, we also noticed that these models will almost always revert to the Question - Answer Style Joke if not prompted otherwise.

reacted to their post with 👀 about 1 year ago

Post

3295

"Why did the bee get married?"

"Because he found his honey!"

This was the "funniest" joke out of 10'000 jokes we generated with LLMs. With 68% of respondents rating it as "funny".

Original jokes are particularly hard for LLMs, as jokes are very nuanced and a lot of context is needed to understand if something is "funny". Something that can only reliably be measured using humans.

LLMs are not equally good at generating jokes in every language. Generated English jokes turned out to be way funnier than the Japanese ones. 46% of English-speaking voters on average found the generated joke funny. The same statistic for other languages:

Vietnamese: 44%
Portuguese: 40%
Arabic: 37%
Japanese: 28%

There is not much variance in generation quality among models for any fixed language. But still Claude Sonnet 4 slightly outperforms others in Vietnamese, Arabic and Japanese and Gemini 2.5 Flash in Portuguese and English

We have release the 1 Million (!) native speaker ratings and the 10'000 jokes as a dataset for anyone to use:
Rapidata/multilingual-llm-jokes-4o-claude-gemini

7 replies

·

posted an update about 1 year ago

Post

3295

"Why did the bee get married?"

"Because he found his honey!"

This was the "funniest" joke out of 10'000 jokes we generated with LLMs. With 68% of respondents rating it as "funny".

Original jokes are particularly hard for LLMs, as jokes are very nuanced and a lot of context is needed to understand if something is "funny". Something that can only reliably be measured using humans.

LLMs are not equally good at generating jokes in every language. Generated English jokes turned out to be way funnier than the Japanese ones. 46% of English-speaking voters on average found the generated joke funny. The same statistic for other languages:

Vietnamese: 44%
Portuguese: 40%
Arabic: 37%
Japanese: 28%

There is not much variance in generation quality among models for any fixed language. But still Claude Sonnet 4 slightly outperforms others in Vietnamese, Arabic and Japanese and Gemini 2.5 Flash in Portuguese and English

We have release the 1 Million (!) native speaker ratings and the 10'000 jokes as a dataset for anyone to use:
Rapidata/multilingual-llm-jokes-4o-claude-gemini

7 replies

·

reacted to their post with 🧠❤️🚀 over 1 year ago

Post

2449

Imagine you could have an Image Arena score equivalent at each checkpoint during training. We released the first version of just that:
Crowd-Eval

Add one line of code to your training loop and you will have a new real human loss curve in your W&B dashboard.

Thousands of real humans from around the world rating your model in real time at the cost of a few dollars per checkpoint is a game changer.

Check it out here: https://github.com/RapidataAI/crowd-eval

First 5 people to put it in their loop get 100'000 human responses for free! (ping me)

posted an update over 1 year ago

Post

2449

Imagine you could have an Image Arena score equivalent at each checkpoint during training. We released the first version of just that:
Crowd-Eval

Add one line of code to your training loop and you will have a new real human loss curve in your W&B dashboard.

Thousands of real humans from around the world rating your model in real time at the cost of a few dollars per checkpoint is a game changer.

Check it out here: https://github.com/RapidataAI/crowd-eval

First 5 people to put it in their loop get 100'000 human responses for free! (ping me)

replied to their post over 1 year ago

Good catch :) yes, we uploaded them shortly after!

replied to their post over 1 year ago

Hey Jackson, can you please elaborate?

reacted to their post with 👍👀🔥 over 1 year ago

Post

4032

Benchmark Update: @google Veo3 (Text-to-Video)

Two months ago, we benchmarked @google ’s Veo2 model. It fell short, struggling with style consistency and temporal coherence, trailing behind Runway, Pika, @tencent , and even @alibaba-pai .

That’s changed.

We just wrapped up benchmarking Veo3, and the improvements are substantial. It outperformed every other model by a wide margin across all key metrics. Not just better, dominating across style, coherence, and prompt adherence. It's rare to see such a clear lead in today’s hyper-competitive T2V landscape.

Dataset coming soon. Stay tuned.

5 replies

·

Jason Corkill

AI & ML interests

Recent Activity

Organizations

Jason Corkill

AI & ML interests

Recent Activity

Organizations

jasoncorkill's activity