I think maybe the original Boltzmann brain may have been about pre-existing matter particles having the ability to just "jump" to a new location (similar to how they can tunnel thru a barrier), and so you can have them all just "jump" into the shape of a perfect square, perfect crystal, or even perfectly formed cat.

So it's not necessarily about the "from pure empty space" version of it. This is all just a thought experiment and not intended to be taken literally.