Artificial intelligence (AI) is increasingly used to tackle Ultra HD (UHD) video production challenges, offering solutions for improving resolution, colour accuracy, dynamic range, and frame rates. Pixop, a company specializing in AI-driven video enhancement, has been at the forefront of this transformation.
Pixop was founded in 2017 with the goal of democratizing video enhancement. Led by CEO Morten Kolle and Chief Technology Officer Jon Frydensbjerg, the company applies cutting-edge deep learning techniques to tackle some of the most pressing challenges in Ultra HD workflows. Early in its development, Pixop experimented with techniques inspired by groundbreaking research by Google in image super-resolution. While initial trials revealed the unique complexities of video workflows, these experiences refined their approach. They launched their first cloud-based platform in 2019, enabling users to upload, enhance, and download video content. Frydensbjerg brings over two decades of experience in video technology and innovation, contributing to Pixop’s mission of enabling cost-effective, high-quality video production through deep learning.
While Pixop’s current focus is on video enhancement, audio remains outside their scope for now.
To better understand these advancements, the Ultra HD Forum interviewed Frydensbjerg to discuss the role of AI in enhancing Ultra HD content. Pixop joined the Ultra HD Forum in late 2024.
Jon Frydensbjerg
The Role of AI in Ultra HD Workflows
Pixop specializes in using deep learning tailored to video-specific challenges. Unlike general-purpose AI systems, its neural networks are trained specifically for video enhancement tasks like upscaling, noise reduction, and dynamic range optimization.
Jon Frydensbjerg explains, “The type of AI we use is based on neural networks, or what people call deep learning. It’s task-specific and optimized for things like resolution enhancement or noise reduction, unlike general-purpose models like GPT or tools to optimize encoding parameters”.
This deep learning approach involves training neural networks on video-specific tasks, such as upscaling resolution, improving dynamic range, and reducing noise. Training consists of starting with pristine video content, degrading it intentionally, and then teaching the AI to “hallucinate” the missing parts to reconstruct the original quality. Unlike AI systems that optimize encoding settings, Pixop’s AI focuses on content quality by reconstructing and enhancing visual details for UHD content to meet the highest standards of clarity and vibrancy while ensuring that the artistic intent of the original content remains intact. The technology is designed to enhance without over-processing, preserving the filmmaker’s creative vision, colour grading, and overall aesthetic choices so that UHD content meets the highest standards of clarity and vibrancy.
AI’s Impact on UHD Quality
AI is being applied to address several key challenges in Ultra HD workflows, bringing measurable improvements:
- Resolution Enhancement: Neural networks predict and generate missing details to upscale lower-resolution content into UHD. “Our system doesn’t just stretch pixels. It uses machine learning to predict and generate the missing details, so the final output looks sharp and natural,” says Frydensbjerg.
- Dynamic Range Optimization and Colour Accuracy: Ultra HD involves High Dynamic Range (HDR) content, which must often be adapted for Standard Dynamic Range (SDR) displays. AI enables advanced inverse-tone mapping that preserves detail and contrast. Additionally, AI dynamically adjusts and enriches colours, ensuring accuracy and vibrancy across frames.
Did You Know?
AI technologies enhance resolution and dynamically optimize colours and inverse-tone-mapping, reducing inconsistencies and elevating visual quality across frames.
- Frame Rate Interpolation: Smooth motion is critical for sports and action-heavy content. AI generates intermediate frames using motion interpolation techniques, ensuring fluid playback even at higher frame rates.
- Addressing Compression Artifacts: Compression artefacts from encoding are a significant concern for UHD content. AI reduces these issues, improving visual clarity while maintaining efficient data usage. This step is critical for delivering premium-quality streaming experiences. For instance, AI can effectively smooth out macroblocking or banding artefacts caused by aggressive compression, particularly in high-motion scenes like sports or action sequences.
Real-World Performance: 20% MOS Improvement for Live Content
AI’s capabilities in live production environments are particularly noteworthy. Internal testing at Pixop shows that the AI delivers an average of a 20% improvement in Mean Opinion Score (MOS) for lossy compressed live content, with a processing latency of 600 milliseconds, including HEVC live mezzanine encoding. Frydensbjerg explains, “This enhancement makes a noticeable difference, elevating content quality significantly while maintaining live broadcast requirements”.
This performance is critical for broadcasters to balance high-quality delivery with real-time constraints.
Integration into Production Workflows
Pixop’s AI integrates into existing workflows, providing live and pre-recorded content flexibility. “You can either integrate our live pipeline or use our container mode, where we decode, enhance, and re-encode content in formats like H.264 or HEVC,” Frydensbjerg explains. The technology is specifically designed for broadcasters with existing HD production pipelines. This goal is to accelerate production workflows and ensure efficient upscaling, tone mapping, and artefact reduction while maintaining compatibility with existing broadcast setups.
Challenges, Iterative Learning, and Lessons Learned
Like with any cutting-edge technology, Pixop’s journey in AI video enhancement has not been without challenges. This AI approach works best for content with reasonable starting quality, where AI can elevate it to exceptional levels. Frydensbjerg notes its limitations: “Our AI is currently less effective when working with extremely low-quality input”.
Another hurdle involved early experiments with inverse tone mapping, which proved more complex than initially anticipated. “We learned that certain approaches simply didn’t yield the expected results,” Jon Frydensbjerg explained, emphasizing the importance of iterative refinement. A key breakthrough was recognising that over-exposure compensation is essential to achieving natural and visually consistent results, ensuring that bright areas retain detail without appearing washed out.
These experiences shaped Pixop’s current approach, focusing on techniques that align with established standards and deliver consistent, practical improvements. “We’ve had to iterate and refine many of our approaches to ensure they align with industry standards and deliver practical results,” Frydensbjerg shares.
Future Applications and Directions
AI’s role in UHD workflows continues to evolve, with exciting possibilities on the horizon:
Real-Time Enhancements: AI’s ability to dynamically improve video quality during live broadcasts is set to expand. “There’s definitely room to improve workflows with 4K and beyond. AI has so much potential to help, especially as the quality of input sources improves,” Frydensbjerg notes. For example, AI could upscale and optimize a low-resolution SDR in real-time during a live sports event, delivering a consistent UHD experience for viewers.
Emerging Use Cases: Beyond resolution and dynamic range, AI could play a role in optimizations that improve encoding efficiency, intelligent downscaling, metadata generation, and enhancing user-specific viewing experiences.
As AI technology matures, its potential to enhance Ultra HD workflows—both in real-time and for on-demand content experiences—seems limitless.
Conclusion
AI is redefining Ultra HD production by enhancing resolution, dynamic range, and colour accuracy while streamlining workflows. As innovation continues, AI will set new benchmarks for video quality and make premium UHD content accessible to a broader audience. With AI at the forefront, the future of Ultra HD production and distribution seem brighter, more innovative, and more accessible than ever.
Experience the Latest AI Advancements for Ultra HD at NAB 2025
Pixop will showcase the latest experimental version of their live upscaling technology at the Ultra HD Forum’s booth in the Futures Park, West Hall, during NAB 2025. Witness firsthand how AI can transform live UHD content with minimal latency, delivering unparalleled quality improvements. Alongside four other Ultra HD demos, attendees can watch live demonstrations of AI-driven resolution enhancement, artefact reduction and dynamic range up-conversion on real-time UHD content streams captured from an HD camera.
Interview conducted by Ben Schwarz
