This one's the other way around. It was spelled "color" in early English, French, and Latin. At some point the French officially added a "u" in order to make the spelling more phonetic, and the English aristocracy imitated that even though it makes no sense given English pronunciation.
I'm French. It's funny, because now that you mention it, the verbs "colorer" and "colorier", both related to colors, are still spelled this way, while the word for color itself is "couleur"... and I never really noticed.
Could you cite that? Sounds interesting, if somewhat contrary to expectation since Chaucer manuscripts already show -our for some words later spelled with -or.
While the part about the English aristocracy is somewhat glib (at the point in time when the "u" was added, arguably the aristocracy were still French, or at least Norman), but the fact that it was "color" in old French and Latin is indisputable, and that for a while both spellings were acceptable in England even as "color" become enshrined in America.
In U.S., via Noah Webster, -or is nearly universal (but not in glamour), while in Britain -our is used in most cases (but with many exceptions: author, error, tenor, senator, ancestor, horror etc.). The -our form predominated after c. 1300, but Mencken reports that the first three folios of Shakespeare's plays used both spellings indiscriminately and with equal frequency; only in the Fourth Folio of 1685 does -our become consistent.
A partial revival of -or on the Latin model took place from 16c. (governour began to lose its -u- 16c. and it was gone by 19c.), and also among phonetic spellers in both England and America (John Wesley wrote that -or was "a fashionable impropriety" in England in 1791).
[...]
Fowler notes the British drop the -u- when forming adjectives ending in -orous (humorous) and derivatives in -ation and -ize, in which cases the Latin origin is respected (such as vaporize). When the Americans began to consistently spell it one way, however, the British reflexively hardened their insistence on the other. "The American abolition of -our in such words as honour and favour has probably retarded rather than quickened English progress in the same direction." [Fowler]
The sibling comment is great, but I'd also like to add that it's important to keep in mind that English spelling was not particularly standardized until relatively recently. People spelled words and even their own names however it seemed best to them at that particular moment in time. Variants like "color" and "colour" existed alongside each other peacefully for centuries and even in a single writer's mind.
"Color" did not exit the British isles until quite late! For example, perusing through Google Books, you can find it in the 1829 edition of the London "Encyclopaedia", and in the 1844 edition of "The London Journal of Arts, Sciences, and Manufacture and Repertory of Patent Inventions."
