Pegasus 1.6 by TwelveLabs - Transforms egocentric video into robot training data
by•
Pegasus 1.6 is the TwelveLabs model that understands video from a first-person point of view, built for the egocentric and teleoperated footage robotics and physical AI teams already collect. It recognizes entities more accurately, and produces persistent metadata across segment types. Point it at raw teleoperation or wearable footage and get back labeled, timestamped, robot-ready data. No manual labeling pass required.
Replies
Hi everyone, I'm Aiden, co-founder and CTO at TwelveLabs. Excited to share Pegasus 1.6 with you today.
Every robotics and physical AI team we talk to is sitting on the same problem: hours of first-person footage that's the raw material for action labeling and next-action prediction, and no foundation model actually built to understand it. General-purpose video models are trained on broadcast and third-person footage. Point them at a wearable or a robot's own camera and they miss what's actually happening.
Pegasus 1.6 is our answer. It's the first TwelveLabs model built for egocentric (first-person) video understanding. Pegasus 1.6 expands on Pegasus 1.5's Time-Based Metadata (TBM), which extracts timestamped, structured metadata from custom schemas, by adding support for egocentric video and task narration. We've also sharpened detailed recognition of hand, tool, and object interactions , while speeding up processing for high-volume video workloads.
As an update to 1.5’s capabilities, we also have improved performance on in-depth metadata extraction and on grounded entity recognition, the ability to name a character based on context clues in the video.
Concretely, for robotics and data teams that means: point Pegasus at raw egocentric clips and get back structured action labels, compliance ratings, and timestamps you can use directly in a training pipeline instead of a manual labeling pass that costs $4–$22/hour of footage.
Physical AI is a new domain for us, so if you're working with egocentric (human or teleoperated) data and hit edge cases, I want to hear about them. Thanks!
S/O for this new launch! keep up the great work
had a blast collaborating with @c3lim and the @TwelveLabs team on this new launch.
Pegasus 1.5 launched 6 months ago, with autonomous and reliable segmentation, long-form video support, and SOTA performances. Pegasus 1.6 pushes the model forward by adding support for first-person video, and faster processes, enabling new opportunities for robotics and physical AI teams.
The community loved 1.5 (Top Product of the Day, featured in the daily newsletter), hope you'll enjoy 1.6!
Follow @TwelveLabs for future launches (spoiler alert: soon)
Do you think egocentric video could become one of the main data sources for training robots?
Congrats @aiden_lee7 & team!
Thanks for hunting @fmerian :)
thanks for the continuous support!
@aiden_lee7 @fmerian @hamza_afzal_butt
Yes, definitely! That's the most traction we are seeing in this space and our models are purpose built to lead that effort in terms of quality and scale!
Go pegasus1.6!
let's gogogog!
@tessak22 you're the best, Tessa! thanks for the continuous support
Heard about TwelveLabs because of the ElevenLabs acronym, and love seeing it grow into such a mature, cool product!
Really neat. How dependent is it on existing commercial models or is it enough of its own thing?
This is seriously impressive. What's next? What's on your roadmap for 1.7? Looking forward to your future launches
S/O to ?makers for the great work here! make sure to follow @TwelveLabs to get notified about the future launches
what's next: Jockey, the first AI that understands your entire media library across photos and video. It's launching today on Product Hunt. See launch: producthunt.com/products/twelvelabs
Any plans to host this on bedrock ?
Turning raw teleoperation and wearable footage into labeled, timestamped robot training data solves a bottleneck that physical AI teams feel constantly, since collecting the video is easy and annotating it is not. Understanding first person perspective specifically is the right specialization. Do the persistent metadata labels stay consistent across long recording sessions?
Spot on. IMHO this could be a game changer for robotics and physical AI teams.
@karimbenkeroum
Hi Kareem, great question! Pegasus supports Time Based Metadata mode which allows for fine atomic action segmentation on long videos. Each segment is then labeled for fields such as action description, hands used, action per hand, tool used, and so on. So you can see a high quality dense captioning output that is perfect for downstream physical AI tasks!
oh yes, and try it for yourself on the playground at twelvelabs.io