You can now generate AI videos with just 6GB of VRAM

[zohaibahd]

In brief: AI video generation may soon no longer be limited to expensive subscriptions or high-powered servers. Thanks to a recent breakthrough, even a gaming laptop could generate full-length AI videos.

The breakthrough comes from Lvmin Zhang of GitHub and Maneesh Agrawala of Stanford University. The duo developed FramePack, a neural network architecture that enables high-quality video diffusion with as little as 6GB of VRAM. This is a significant achievement, especially given the model's size – 13 billion parameters – which allows it to generate full 60-second clips at 30 FPS using only a mid-range GPU.

The key lies in how FramePack operates. Traditional video diffusion models rely on previously generated frames to predict the next one. As the video length increases, so does the "temporal context" – the number of past frames the model must consider – resulting in higher memory demands. This is why most models require 12GB of VRAM or more to run efficiently.

FramePack flips that on its head. Instead of letting memory usage balloon with longer clips, it compresses input frames based on importance into a fixed-length context, keeping the memory footprint compact and consistent regardless of video duration.

This innovation allows the model to process thousands of frames, even with large architectures, on laptop-grade GPUs. It also enables training with batch sizes comparable to those used in image diffusion models.

But FramePack doesn't just reduce memory demands, it also addresses drifting – a common issue where video quality degrades over time. By using intelligent compression patterns and scheduling techniques, FramePack helps maintain visual consistency from beginning to end.

To top it off, the model includes a user-friendly GUI. Users can upload images, enter text prompts, and view a live preview as frames are generated. On an RTX 4090, optimized generation speeds reach up to 0.6 frames per second. Naturally, performance is lower on less powerful GPUs, but even an RTX 3060 can handle it.

Currently, FramePack supports Nvidia's RTX 30, 40, and the new 50 series GPUs, provided they support FP16 or BF16 data formats. There's no confirmed support yet for AMD or Intel GPUs, but the model works across multiple operating systems, including Linux.

You can find full model details and source code on GitHub.

Permalink to story:

You can now generate AI videos with just 6GB of VRAM

 

[DanteSmith]

https://github.com/lllyasviel/FramePack

It looks good enough in simple movements, but also scary when distorting anatomy totally (example of the girl dancing, but neck twisting to back). The updated version of Amuse also features compact video generation models that run on mid-end GPUs.

 

[psycros]

Yes, let's make it even easier to create deepfakes while putting all creative professionals out of work. Sounds like a real utopia.

 

[AnilD]

The idea of compressing temporal context into a fixed size is such a smart shift—feels like a foundational change, not just an optimization. Could open the door to a lot more experimentation by indie creators and researchers without deep pockets.

 

[yRaz]

Don't tell nVidias

 

[Axeia]

while putting all creative professionals out of work. Sounds like a real utopia.

always found that to be a poor argument. How many farmhands are put
out of jobs by horses working the fields, how many horses are put out of a job by tractors.
If there is a way to do things cheaper it'll be done cheaper. Can't just employ people for the sake of employing people

 

[WhiteLeaff]

always found that to be a poor argument. How many farmhands are put
out of jobs by horses working the fields, how many horses are put out of a job by tractors.
If there is a way to do things cheaper it'll be done cheaper. Can't just employ people for the sake of employing people

Your argument is also poor. Food is an essential commodity that everyone needs, and the population has skyrocketed many times over since the mechanization of production.

The same can't be said for other sectors, and no, jobs don't go elsewhere. They will disappear.

 

[Axeia]

Your argument is also poor. Food is an essential commodity that everyone needs, and the population has skyrocketed many times over since the mechanization of production.

The same can't be said for other sectors, and no, jobs don't go elsewhere. They will disappear.

Fine take the typical phone operators example then. Used to be a massive industry of (almost exclusively women) switching some plugs around to get the phone connected to the right place. Automated away.
Oh wait let me guess only artists have some special right to make a living the way they want to? Painting pottery good enough? Practically automated away. In some rare cases you might get one hand painted for a fat premium with the painting done in India or some other low wage place.

Yes jobs will disappear, I always assumed that'd be the end goal. Everyone does what they choose to do whilst machines take care of everything. And sure I don't think humanity is ready for that, or will ever be as a few people will always take up the majority of resources to have it better than everyone else. But I don't see why artists are any different than anyone else. Nothing stopping them from still doing it as a hobby if they really think it makes a difference in the world of simply enjoy doing it, just can't make a living out of it. Just because I enjoy coding doesn't mean I want it to be a requirement that all code is hand written.

 

[WhiteLeaff]

Fine take the typical phone operators example then. Used to be a massive industry of (almost exclusively women) switching some plugs around to get the phone connected to the right place. Automated away.
Oh wait let me guess only artists have some special right to make a living the way they want to? Painting pottery good enough? Practically automated away. In some rare cases you might get one hand painted for a fat premium with the painting done in India or some other low wage place.

Yes jobs will disappear, I always assumed that'd be the end goal. Everyone does what they choose to do whilst machines take care of everything. And sure I don't think humanity is ready for that, or will ever be as a few people will always take up the majority of resources to have it better than everyone else. But I don't see why artists are any different than anyone else. Nothing stopping them from still doing it as a hobby if they really think it makes a difference in the world of simply enjoy doing it, just can't make a living out of it. Just because I enjoy coding doesn't mean I want it to be a requirement that all code is hand written.

Yes, humanity's goal is to destroy itself and annul any trace of culture, goals, etc.

We're going to automate everything, making 90-95% of people zombies earning the minimum to live on universal credit and be happy just existing, consuming generic content of even worse quality than today. The brains begin to rot inside the heads and generation after generation the IQs reduce, leading to involution. Yay.

 

[Zahu]

always found that to be a poor argument. How many farmhands are put
out of jobs by horses working the fields, how many horses are put out of a job by tractors.
If there is a way to do things cheaper it'll be done cheaper. Can't just employ people for the sake of employing people

Thats what our government (USA) has been doing for decades. Employing people for the sake of employing people.

 

[Axeia]

Yes, humanity's goal is to destroy itself and annul any trace of culture, goals, etc.

We're going to automate everything, making 90-95% of people zombies earning the minimum to live on universal credit and be happy just existing, consuming generic content of even worse quality than today. The brains begin to rot inside the heads and generation after generation the IQs reduce, leading to involution. Yay.

Alternatively with all the time freed up we have an utopia and can focus our efforts on how to exterminate all suffering and explore space.
Nah just kidding. The future will definitely be closer to Cyberpunk than it is to Startrek.
We can't solve the housing shortage because it's more profitable not to. We can't solve world hunger whilst throwing away more food than is needed to fix it... Because why help those so poor they can't even afford food, there's no profit in it.