> Been running local LLMs on my 7900 XTX for months and the ROCm experience has been... rough.
Just out of curiosity... how so?
I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of. Actually, I've barely had any problems at all, other than the card being limited to 24GB of VRAM. :-(
I'm halfway tempted to splurge on a Radeon Pro board to get more VRAM, but ... haven't bitten the bullet yet.
Did you have complete hardware lockups when VRAM is exceeded? I had quite a few on my 7900XTX with llama.cpp (Arch Linux, various driver versions). Once I dial in the quant and context size that never exceed VRAM, it is stable; before that I swear a lot and keep pressing the hardware reset button.
Yes, it completely crashes the machine. I didn't even think it was unexpected until I read your comment. I guess this is what I come to expect when using anything except firefox or neovim
Nope. I've exceeded available VRAM a few times, and never had to do anything other than maybe restart Ollama. To be fair though, that's "exceed available VRAM" in terms of the initial model load (eg, using a model that would never load in 24GB). I don't know that I've ever started working with a successfully loaded model and then pushed past available VRAM by pushing stuff into the context.
I've had a few of those "model psychosis" incidents where the context gets so big that the model just loses all coherence and starts spewing gibberish though. Those are always fun.
I remember reading a review of Neuromancer once, where people were making light of the thing with Linda Lee stealing Case's black-market RAM, on the basis that RAM is now a dirt-cheap commodity and not something that would hold great value.
There is a considerable overlap between people who believe that a reality tv personality has real power and those who believe that reality tv is unscripted.
But according to the Birmingham modifications of 1973, subsection 12.b, stroke 7a, a player so conceding is not deemed to have actually conceded unless they be within a finite number of hops from Mornington Crescent station at the time of the concession.
Capitalism - in principle - doesn't even require that investors as such exist, much less that they own everything. That our current system has evolved into a pseudo-capitalist system which introduces all sorts of State created constructs (like corporations), and borrows elements from fascism, socialism, etc., doesn't mean that it must be this way.
That's a ridiculously extreme example, IME. Having been working with Java for 25+ years, I've never seen anything like that in code written by a shop I worked at, FWIW. And as a corollary I'm about 100% certain that you could find some equivalent extreme case to use as an example to dunk on any language/platform.
this is what we mean by "JavaCulture" - java | JVM really wonderful things but the culture is what sets these things back
Maybe there are still some Java shops out there living like this. But to me, this is a 20 year old outdated notion. shrug
I also have 20+ years of experience and haven't seen that type of code either.
I am however struggling to understand an over engineered Rust "micro" service at work. There's no language or framework that is free from this.
Maybe it's because I have not worked in typical enterprise companies?
Spring Boot can be a really quick way to get things done. My current job is not in enterprise, but we use Spring Boot for most of our services, and rarely have issues related to Spring itself. We have micro services handling many transactions per minute and starts up within 10 seconds. The CPU usage is low and decent memory usage, even on the small pod sizes that we use. There are ways to reduce the startup, but we do not have the need. The code is easy to reason about and new hires are productive within a short time.
Let's not go too far! I've seen some crap code, granted. But for the most part, the crap code I've seen is, I believe, stuff written by people who would have written crap no matter what language they were working in.
That said, I have in fact also had some outstanding and talented colleagues over the years as well. A point that's probably worth reflecting on, so thanks for the reminder.
It wasn't me, but had I seen any of that stuff, I certainly would have flagged it as well. There are other places to have those conversations. Keep this crap off of HN.
At the very least, that should certainly be an option that users can select. And when the user selects a feed algo, it should stay fucking set until that same user actively chooses to change it.
Soul of a New Machine was one of the first books that got me interested in the tech industry, computers, etc. Reading it as a teen probably contributed substantially to the direction of my career up to the present day.
I would second the black bar for Kidder -- The Soul of a New Machine constitutes the literary foundation of our craft: it is our Odyssey. Speaking personally, I have spoken and written about Soul many times ([0][1][2]) -- and I know that its impact from me is far from unique.
RIP Tracy Kidder -- and thank you for giving us all permission to feel passion for the machine.
[1]: https://en.wikipedia.org/wiki/Technocracy_(EP)
reply