Forward-looking: Sony has filed a patent for an AI system that predicts a game's next frame before the graphics hardware finishes rendering it, aiming to reduce stuttering without introducing the additional latency associated with some frame-generation methods. The system uses the most recently completed frame and, optionally, recent controller input to generate a predicted frame that can be displayed while the hardware renders the next one.
The approach targets a drawback of some frame-generation techniques: they require two completed frames before generating an image to insert between them. Waiting for the second frame adds latency. Sony's system would instead use an already available frame to predict the next one, rather than waiting to interpolate between two completed frames.
That difference could help games maintain a steadier frame delivery rate when rendering slows down. However, the patent filing includes no performance measurements, leaving its effects on latency and visual quality unproven.
Sony's design uses a jointly trained encoder and decoder. The encoder converts a rendered frame into an embedding – a compact numerical representation of the image. The decoder then uses that representation to generate a predicted frame.
During gameplay, the system runs alongside the graphics pipeline. While the hardware renders the next frame, the AI processes the previous one and sends its prediction to the display buffer. The generated image can therefore appear before the next rendered frame is ready.
The model can also use motion vectors, which describe how parts of an image move, and recent controller input. These additional inputs provide more information than the previous frame alone: motion vectors indicate where objects are heading, while button presses reveal the player's latest actions.
Training uses consecutive video frames. The network starts with randomly initialized weights, takes one frame as input, and attempts to predict the next. It then compares its output with the actual subsequent frame, producing a loss value that measures the prediction error.
The patent's claims describe checking this loss against a threshold during training. Because the video itself provides reference images, the process does not require people to label the training material. The decoder also outputs more color channels than the encoder receives, allowing it to generate multiple predicted frames at once.
Sony also describes a non-AI alternative that shifts pixel blocks according to their motion. This method would generate synthetic frames without using the trained encoder-decoder model. The filing also describes maintaining a fixed frame delivery rate without reducing visual quality, although whether the generated frames can actually preserve that quality remains an open question.
The practical test will be whether the AI can generate accurate frames quickly enough to fill gaps when rendering slows down. The patent outlines a method for doing so, but it does not establish how well the system works in practice.
Sony patents AI that predicts a game's next frame before the GPU finishes rendering it
