Google Rolls Out Real-Time AI Video Features in Gemini Live

Google has begun the gradual rollout of real-time video understanding features in Gemini Live, expanding the assistant's capabilities beyond traditional voice and text input. These new tools allow Gemini to analyze your phone screen or camera feed in real-time and provide relevant, contextual responses—marking a major leap forward in Google's push toward intelligent multimodal AI.

Live Screen and Camera Understanding

As part of the Gemini Advanced tier under the Google One AI Premium plan, two core features are now becoming available to some users:

1. Screen Sharing with Gemini Live – This feature allows users to share their smartphone screen with Gemini and ask questions about what's being displayed. Whether it's code in an IDE, a webpage, or a document, Gemini can interpret and respond accordingly. It's a natural extension of Google's goal to make Gemini a true visual assistant—one that doesn't just talk back, but sees and understands the digital environment you're working in.

2. Live Camera Feed Interpretation – Using your phone's camera, Gemini can also analyze the real world in real time. A demo shared by Google shows someone asking Gemini for advice on paint colors while holding up a freshly glazed piece of pottery. The assistant responds quickly and accurately, identifying objects and offering helpful insights.

These capabilities are powered by the same foundational work demonstrated in Project Astra, a research initiative aimed at building a real-time, multimodal AI assistant. Astra blends vision, voice, and contextual understanding, making it possible for AI to see what you see and assist in ways that feel seamless and intuitive.

The Reddit thread where we uncovered this had a demo - 

The Tech Behind It

These new features are driven by Google’s Gemini 1.5 Pro model, which supports a large context window—up to 2 million tokens. This allows the assistant to keep track of extended interactions, analyze complex visual or textual inputs, and respond more fluidly.

Speed and latency were significant technical challenges in enabling these real-time interactions. Google spent months optimizing both the model and infrastructure to make it possible for Gemini to interpret screen or camera inputs and return useful responses with minimal delay.

Designed for the Future of AI Assistants

Gemini Live is designed to feel natural—conversations can be interrupted and resumed, with the assistant remembering previous points and adapting as the interaction evolves. The assistant doesn’t just respond; it collaborates. With multimodal interaction—voice, touch, text, video, and visual input—Google is building toward a vision of AI that acts more like a helpful co-pilot than a simple chatbot.

Beyond real-time video understanding, other developments across the Gemini ecosystem continue to push boundaries. For instance, Gemini 1.5 Flash, a lighter model variant, focuses on speed for tasks like summarization and captioning. Another model, Veo, can generate video content from text prompts, while Gemini Nano, designed for on-device tasks, continues to improve in performance and responsiveness.

A Shift Toward Agent-Based AI

Google's AI vision increasingly centers around agents—intelligent tools that can take meaningful actions on your behalf, not just respond to prompts. With Astra and Gemini Live, these agents are becoming more aware of your digital and physical context, making them capable of proactive help rather than reactive answers.

As these features roll out to more users over the coming weeks, the line between AI assistant and intelligent collaborator continues to blur. With the ability to see, understand, and act across modalities, Gemini Live marks a major milestone in AI's evolution—and sets the pace in the growing race for next-gen assistants.