AWS and FOX join forces for AI-fueled highlights

Amazon’s open source Media Replay Engine (MRE), the key to success: the engine is the main tool used for building groundbreaking recap feature and Catch Up With Highlights

New tech advances make viewers more demanding; sports fans have come to expect a dynamic viewing experience due to digitization and continued advancement of broadcast delivery technology. And this comes from all kind of users; whether streaming a game live or recording to view on-demand, audiences are increasingly looking for standout elements in the coverage of their favorite games, from compelling graphics and data insights to timely replays and highlights.

Besides, when the games are some of the most anticipated in sports, the stakes are even higher. Rising to the occasion, FOX built and deployed its new AI-powered Catch Up With Highlights feature ahead of its US broadcast coverage of the FIFA World Cup Qatar 2022™.

Leveraging Amazon’s open source Media Replay Engine (MRE) framework for automating video clipping and replay generation, alongside Amazon Web Services (AWS) infrastructure and solutions, Catch Up With Highlights is an end-to-end solution that can ingest broadcast streams, identify valuable match moments through AI-trained computer vision, and then encode delivery streams with metadata across user devices. With AWS already the foundation of Fox Corporation media workflows, the development team was able to stand up the complex new feature in about eight weeks.

FOX’s Vice President of Data Products and Machine Learning (ML) Strategy, Phil Martin, who led the Catch Up With Highlights development and first deployment, shared, “Over the last five years our team has had tremendous success with unlocking the power of data through the building of advanced and intelligent platforms and products. In addition, we’ve really rallied around the idea of revolutionizing fandom. We believe that sports should be enjoyable and accessible for everyone, which is why we’ve designed our platform to make it easy to engage with and enjoy. Our Catch Up With Highlights feature is a great example of this philosophy. We understand that the level of play at events like the World Cup is exceptional, and we wanted our feature to reflect that standard. To achieve this, we developed a cutting-edge AI technology to create a highlights engine that can dynamically retell the game’s story in real-time, highlighting key moments and events as they unfold. This ensures that our users can stay up-to-date with the action and experience the excitement of the game, no matter where they are. Prior to and through this journey, AWS has been a trusted partner of ours and together we were able to take another step forward in revolutionizing fandom with data all while on the world’s biggest sporting stage.”

Upon launch, the feature ran seamlessly with very little human input. Martin and his team built Catch Up With Highlights to continually optimize the highlight recap video as the game progressed, retelling the game story upwards of 20 times per match with event-driven updates. This approach ensured that the viewers would be able to access a comprehensive recap of the game to date, available via the FOX Sports app on connected TVs and mobile devices. Incremental improvements were rolled out based on performance, such as adding a thumbs-up or down; and event enhancements, including shots saved and penalty kicks. Ultimately, 731 recap videos were published autonomously across 61 matches.

The Catch Up With Highlights workflow includes AWS Lambda event-driven serverless compute, and computer vision with Amazon Rekognition. Sports data is housed in an Amazon DynamoDB NoSQL database, then piped into MRE. Live video is quickly delivered to viewers via AWS Elemental MediaConnect, with videos stored in Amazon Simple Storage Service (Amazon S3), and Amazon ElastiCache for caching live clips.