Has OpenGVLab Lumina Outperformed OpenAI’s Model?

We’ve just released the results from a large-scale human evaluation (400k annotations) of OpenGVLab’s newest text-to-image model, Lumina. Surprisingly, Lumina outperforms OpenAI’s DALL-E 3 in terms of alignment, although it ranks #6 in our overall human preference benchmark.

To support further development in text-to-image models, we’re making our entire human-annotated dataset publicly available. If you’re working on model improvements and need high-quality data, feel free to explore.

We welcome your feedback and look forward to any insights you might share!

Rapidata/OpenGVLab_Lumina_t2i_human_preference

liked a dataset 20 days ago

Rapidata/OpenGVLab_Lumina_t2i_human_preference

Viewer • Updated 20 days ago • 13k • 1.16k • 13

reacted to jasoncorkill's post with 🚀🚀 26 days ago

Post

2508

This dataset was collected in roughly 4 hours using the Rapidata Python API, showcasing how quickly large-scale annotations can be performed with the right tooling!

All that at less than the cost of a single hour of a typical ML engineer in Zurich!

The new dataset of ~22,000 human annotations evaluating AI-generated videos based on different dimensions, such as Prompt-Video Alignment, Word for Word Prompt Alignment, Style, Speed of Time flow and Quality of Physics.

Rapidata/text-2-video-Rich-Human-Feedback

reacted to jasoncorkill's post with 👀 26 days ago

Post

2848

Integrating human feedback is vital for evolving AI models. Boost quality, scalability, and cost-effectiveness with our crowdsourcing tool!

..Or run A/B tests and gather thousands of responses in minutes. Upload two images, ask a question, and watch the insights roll in!

Check it out here and let us know your feedback: https://app.rapidata.ai/compare

reacted to jasoncorkill's post with ❤️ about 1 month ago

Post

4555

Runway Gen-3 Alpha: The Style and Coherence Champion

Runway's latest video generation model, Gen-3 Alpha, is something special. It ranks #3 overall on our text-to-video human preference benchmark, but in terms of style and coherence, it outperforms even OpenAI Sora.

However, it struggles with alignment, making it less predictable for controlled outputs.

We've released a new dataset with human evaluations of Runway Gen-3 Alpha: Rapidata's text-2-video human preferences dataset. If you're working on video generation and want to see how your model compares to the biggest players, we can benchmark it for you.

🚀 DM us if you’re interested!

Dataset: Rapidata/text-2-video-human-preferences-runway-alpha

1 reply

liked 3 datasets about 1 month ago

upvoted a collection about 1 month ago

Sora Rich annotation

Collection

Contains a rich set of different types of annotations for a set of 198 videos • 6 items • Updated Feb 4 • 10

liked 5 datasets about 1 month ago

Rapidata/sora-video-generation-time-flow

Viewer • Updated Feb 4 • 198 • 68 • 13

Rapidata/sora-video-generation-aligned-words

Viewer • Updated Feb 4 • 48 • 400 • 15

Rapidata/sora-video-generation-alignment-likert-scoring

Viewer • Updated Feb 4 • 198 • 415 • 13

Rapidata/Runway_Frames_t2i_human_preferences

Viewer • Updated Feb 3 • 25.5k • 242 • 13

Rapidata/sora-video-generation-physics-likert-scoring

Viewer • Updated Feb 4 • 198 • 307 • 18

reacted to jasoncorkill's post with 🚀 about 2 months ago

Post

2685

We benchmarked @xai-org 's Aurora model, as far as we know the first public evaluation of the model at scale.

We collected 401k human annotations in over the past ~2 days for this, we have uploaded all of the annotation data here on huggingface with a fully permissive license
Rapidata/xAI_Aurora_t2i_human_preferences

liked a dataset about 2 months ago

Rapidata/xAI_Aurora_t2i_human_preferences

Viewer • Updated Feb 3 • 26.2k • 196 • 15