XUL was only part of the problem. Camino's goal was to make a browser that was designed for the Mac. Firefox's goal (once running on the Mac was added after the fact) was to make a cross platform browser that ran on the Mac. That's not just about UI toolkits, it drives basic decisions.
Firefox doesn't use Keychain because cross-platform trumped integration. Its localization (a different download for every language, instead of one multi-language binary like every Mac app) is "wrong" for the same reason. It doesn't integrate with the Me card in Address Book for form fill data for the same reason. It makes you set your accept-language list manually instead of using the identical OS-level list for the same reason. The list, some small and some large, goes on, and it all adds up.
It affects UI behaviors too. I don't know if they eventually fixed it, but for a while closing the last window would quit the app, which is against the platform guidelines for a multi-window app, and was widely hated by Mac users. The justification was that it was consistent with Windows Firefox.
Those of us who chose to work on Camino in our free time did so because it was the product we wanted to build, and Mac Firefox was clearly never going to be no matter how much effort we might have put into it. You can call it a waste if you like, but I think you'd be hard pressed to find someone who worked on Camino who regrets the time they spent on it, or considers it a waste.
As for it being an "obvious dead end", until Gecko embedding was deprecated there was no reason Camino couldn't have continued indefinitely. How is developing software that many people still love and use a dead end? Not everyone in the world views dominating the market as the definition of success.
