Sieve

Simulation and human videos are an exciting data well for robot world models due to the particular difficulty of collecting real-world data, but Sergey Levine (UC Berkeley professor + co-founder of Physical Intelligence) just published an incredibly sobering take on the danger of venturing too deeply in this direction. Some researchers have gotten so excited in other data sources to the extent that they're being treated as a replacement to the real thing. But just like how LLMs use lots of text data and VLMs use text-image pairs, VLAs (vision-language-action) models in robotics need a lot of data of robots performing real-world tasks. Instead of treating simulation or human video (i.e. FPV videos posted online) as a complete replacement, we should treat it the same way we treat internet data in LLM and VLM pre-training - something less relevant to the ultimate goals of the model but still relevant enough to provide useful world knowledge. At Sieve, we're excited to be contributing to this problem area through our early work with robotics labs making use human videos for VLA pre-training. If you're interested in learning more about Sergey's take or our human video offering, check out the links in comments.

We benchmarked 14 of the top AI dubbing tools at Sieve, and the results shocked us. Even some of the best-known names failed at preserving speaker identity or handling multi-speaker videos. For context, the tests were conducted by third-party native speakers across 8 languages and benchmarked each tool on translation accuracy, grammar, accent preservation, voice identity, timing sync, audio clarity, speaker consistency, and more. Here are the 5 tools that performed best overall on a scale of 5: 1. Sieve & VEED.IO - best overall in accent preservation, speaker identity, and contextual translation 2/ Panjaya - strong emotional nuance and timing 3/ HeyGen - solid voice quality, but struggled with multiple speakers 4/ Vozo AI - crisp sync and clarity 5/ Dubly.AI - great naturalness, ranks 1st for japanese & portuguese One thing is clear: Voice cloning ≠ good dubbing. True quality comes from systems that understand who’s speaking, what they mean, and how it should sound in another language. Explore our methodology and the full results below (link in the comments). Note: VEED has the same score as Sieve because their dubbing pipeline is powered by Sieve under the hood.

Multi-modal LLMs like GPT-4o and Gemini are starting to outperform traditional models on core video and audio understanding tasks. One example is with diarization and it's use in AI dubbing for handling multi-speaker videos, where existing solutions fall apart. To get it right, you need to: - Detect speaker turns in real-time - Sync translated audio without collapsing natural pauses - Maintain distinct voices across languages - Avoid awkward overlaps or robotic silences It’s one of the hardest parts of dubbing. You also need to know who is speaking and how they should sound. In Arabic, for example, the phrase “you went” is “roht” when addressing a man and “rohti” when addressing a woman; mixing them up is grammatically wrong. We’re rolling out a major upgrade soon that makes our dubbing pipeline much better at this. More details (and behind-the-scenes R&D) dropping soon. P.S: here's a before and after output comparing the natively multi-modal speaker handling vs the more traditional version.

Stoked to welcome Ahi to the team at Sieve. Ahi's background is unique - from building the world's smallest batteries to training high quality TTS and diffusion models in the Computational Image Group at Rice. If you're interested in joining a fast-growing team working on internet-scale computer vision problems, DM me :)

Transparency in how AI capabilities work and the way in which they're evaluated help developers build trust in using them. At Sieve, we're constantly developing internal evaluation systems that enable to us to ship the highest quality AI video capabilities in the world. Tomorrow, we'll be sharing a behind-the-scenes look at how we evaluate one of our most nuanced solutions and the specific optimizations we made to score highly.

We're in London at VEED.IO HQ this weekend with 150 hackers building Gen AI video tools. They got access to caffeine, pizza, a bunch of Sieve credits, and a few unreleased APIs 👀 Let's see what they cook up in the next 24 hours!