For some reason I thought fine tuning is not possible without specialized hardware (A100 / H100). Where can I learn more about hardware requirements for fine tuning on consumer GPUs?
There is not a lot of great content out there making this clear, but basically all that matters for basic fine tuning is how much VRAM you have -- since the 3090 / 4090 have 24GB VRAM they're both pretty decent fine tuning chips. I think you could probably fine-tune a model up to ~13B parameters on one of them with PEFT (https://github.com/huggingface/peft)
Definitely possible on even older off-the-shelf hardware. I use 24GB 4090s for 13b-sized models and have even used 12GB Titans for 7b models, admittedly at much slower rates.
I have a 3080Ti with 12Gb VRAM and would like to try fine tuning the same Mistral 7B model (which I found incredibly potent). Any tips on how to get started?
