Speaking as a huge fan of Ruby, I think it's absolutely the wrong language to write games in, because games are enormously sensitive to runtime fluctuations, and as you found out, Ruby has them. In particular, a stop-the-world GC is just never going to give you satisfactory performance, unless every GC pass happens in <1ms and never happens more than once per frame.
That said, you might consider Lua as a primary scripting engine for your game. It's got a lot of your standard scripting language perks, but it's also obnoxiously fast and has an incremental garbage collector. Furthermore, there's no GIL, and it's designed to be embeddable, making it a very powerful choice for game development.
Usually the way you solve GC issues is by either not allocating after some initial massive allocation, or by running the GC constantly and having it clean up bits and peices every frame. I don't know enough about ruby to know if this is possible but this is how you handle GC in Java and C#.
As for Lua, Garry's mod runs almost entirely in Lua (outside of the source engine).
You can run GC manually in Ruby, but at the end of the day, it's a stop-the-world GC; your program doesn't get to do anything until it decides that it's finished. If that takes too long, you drop that frame. Additionally, Ruby doesn't have the concept of primitives (mostly), and all objects are stored on the heap! So, every time you use a float (which you do a lot of in games), you end up with a new struct on the heap that has to be garbage collected eventually. Ouch.
Languages with incremental GCs will yield between phases of the GC pass, letting program execution continue even when a GC pass is in progress, making sure that you don't end up with dropped frames, which vastly improves the player experience.
Mike Pall's LuaJIT[0] is excellent from a game dev perspective--incremental GC (so you can spend a fixed amount of your frame budget in it) and JIT with a huge number of optimisation make it fairly compelling to write the top 50% of your game in (and not a terrible choice for the bottom 50%).
I think you were absolutely right to switch from Ruby, I can't see the problems that you had getting any better unless Ruby lets you manage object allocation (in which case you could allocate to a per-frame memory buffer and just nuke it at the end of every frame).
The cause of its popularity in the games community comes from the fact that it is, by design, meant to be embedded in larger programs. So, for example, it will compile on any platform with a conforming ANSI C compiler.
And for x86/x64 and ARM, you can try LuaJIT to get even more zing (though you probably won't need to).
Just because the Lua community will ream you out for the mistake (though it's the only thing they're mean about), it's "Lua" (Portuguese for "moon"), not an acronym. :)
I think it's absolutely the wrong language to write games in, because games are enormously sensitive to runtime fluctuations
No-one is nailing down the definition of "game" in this entire discussion. For a FPS or game filled with live special effects, sure. For a chess game, a puzzler, or even a graphic adventure? Ruby's GC wouldn't be a problem at all (although Ruby has other issues that could well crop up).
Seems to me that no matter what the language du jour may be, games are still often written in c++. When you're not getting the performance out of the tools you're using and you've spent some time trying to work around the problems, I think it's the right call to switch tools.
Thanks: after three days I think it was the right call. I just found myself burning too much brain energy on things I didn't want to have to care about.
You might want to explore Lua as an alternative. Lots of games are written in Lua/C/C++ combo. Low level primitive routines in C/C++ and high level game stuff in Lua.
Writing everything in C++ is not my idea of fun. It's definitely possible to write almost all of the game code in a scripting language. It would be reasonably easy to abstract all the native APIs you want, but you could also just let someone else do it: http://love2d.org/
Of course you did, and I say that as a self-described Ruby programmer. What I really want to know is what gave you the idea that it might be a good idea to use Ruby for a project so obviously not suitable for Ruby?
Yeah, thought about it (Gosu was great for getting started), but decided I wanted to get back to fundamentals and switch to a slightly lower level framework to aid my understanding of what's really going on under the hood.
ruby excels where time to market and RAD are more important than performance, and where the standard and other libraries work well. Ruby saves lots of time doing things like CRUD operations where you're basically mapping on set of fields to another in a predictable way. When performance and memory are an issue and you can't trivially scale your code (eg. More processes or more machines) C++ is a big win.
