The current Unix USB Installers (e.g.
Universal USB Installer from PenDriveLinux.com) create a persistent file on the boot partition together with the other files needed for booting. This boot partition is formatted with FAT32 and because of limitations in FAT32, the maximum size of this file is 4GB. To have more persistent storage, this file can be replaced by a partition. The size of such a partition is only limited by the size of the USB drive.
I used the GParted partition manager to resize the FAT32 partition and make space for a big EXT4 partition for the persistent storage. The persistent file can simply be deleted and at the next boot, the Ubuntu system will automatically use the bigger persistent partition. There is one complication, though: you need another Ubuntu system to do this because Ubuntu cannot modify its own active boot partition.