Packets that still have to encode and adhere to strict VGA timings (Hsync/Vsync/blanking), and you arent allowed to send anything useful between the packets unless its a blanking period.