Researchers develop a way to view livestreamed video games from any angle

The bleeding edge: Researchers at the University of Waterloo are developing a new way to watch video game streams on services like Twitch and YouTube. The technique takes the 2D live video and converts it into a 3D interactive experience where viewers can watch the action from any angle.

Many multiplayer shooters have a spectator mode that allows users to watch what is going on from different perspectives. It's an excellent way to observe what strategies other players use or to enjoy watching an outstanding player destroy the competition. However, to do this, you must be playing the game.

Jeremy Hartmann and Daniel Vogel from the University of Waterloo's Cheriton School of Computer Science published their research in late April. The paper titled "Enhanced Videogame Livestreaming by Reconstructing an Interactive 3D Game View for Spectators, explains a way to achieve a similar effect as spectator mode, but without installing the game.

The technique takes the 2D video stream and adds in the game's depth buffer, camera pose, and projection matrix to create a 3D interactive environment that viewers can explore as the streamer plays the game. Spectators can move their cameras around freely in both 3D and VR.

At first, it looks like just a strange angle of the screen's action superimposed on a grey background. However, as the player moves and looks around, simple untextured models will begin to form as the system builds the environment in real-time.

It is not the same as an in-game spectator mode. As mentioned, everything but the player's perspective is untextured. Also, anything outside of the stream's view is unanimated. For example, in the video above, you can see how the player's model is fully rendered but slides around the environment as if it were on wheels.

The lack of textures and animation does somewhat detract from the viewing experience, but keep in mind that this method of rendering video-game live streams in 3D is in its infancy. Advancements in the technique Hartmann and Vogel describe and machine learning may lead to better and more enjoyable interactive live streaming experiences that put the viewer right into the game.