I think Google has some big advantages in cost with TPUs and their crazy datacenter infra (stuff like optical circuit switches) but I'd guess long context is still going to be expensive initially.

They used ML to get a ~40% cooling efficiency on their datacenters. It's mentioned in a talk by Cassie Kozyrkov.

Yeah I'm specifically interested in this because I'm in a lot of local telegram groups which I have no patience to catch up on every day. I'd love to have ChatGPT summarise it for me based on a list of topics I care about.

Sadly the cost of GPT-4 (even turbo) tends to balloon for this usecase. And GPT-3.5-turbo while much cheaper and more than accurate enough, has a context window that's too shallow.

I wonder if Telegram will add this kind of feature also for premium users (which I also subscribe to) but I imagine it won't work at the current pricing levels. But it would be nice not having to build it myself.

GPT3.5 and GPT4 are not the only options though, right? I don't follow that closely but there must be other models with longer context length that are roughly GPT3.5 quality by now, and they even probably use the same API.

I don't really know. The benefit of ChatGPT is that it's so big, there are so many nice APIs for it :)

I'm not so deep into it all.

Mistral 8x7b has can handle context of ~32,000 pretty comfortably and it benchmarks at or above GPT3.5

Is that the sliding context window size? Because I didn't have good results with sliding context windows in the regular Mistral models.

Yeah, I think they fine-tune without a specific window size target to achieve and then keep expanding context until it starts falling over.

I imagine it will cost peanuts within a year