Absolutely. This is already what I've been doing myself. Forked zed so I could use my local rig (4x 3090) to do FIM completions using a finetuned qwen coder 32B.
Only barrier for some folks will be if they're not familiar with rust or local LLMs in general, but it really wasn't that difficult looking back on it. Amounted to about an afternoon's worth of work.
At present it's not possible just via configuration, but you can configure a custom endpoint for both the assistant and inline assistant in settings.json.
To get custom tab completions working you need to mimic one of the completion provider apis (like copilot) [1] and direct the requests to your own endpoint. In my case, I am running llama.cpp and and MITM-proxying to its newer /infill endpoint [2]
That's why I mention the rust piece may be a blocker for some, you do have to hack apart the src a bit to get things working. At the time I started this, the only two completion providers available were supermaven [3] and copilot. You could mimic either.
I actually had a similar idea - and have a PR up on Zed right now to add in support for specifying a HTTP/HTTPS proxy to the copilot completion APIs: https://github.com/zed-industries/zed/pull/24364
Branch is a bit out of date vs main now but it should work if you build off it.
When I tried a few weeks back I was only able to get the latter to actually process requests, but haven't had a chance to look into it much further yet... first one's probably your best bet
