Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
GPT-4o's Chinese token-training data is polluted by spam and porn websites (technologyreview.com)
16 points by maxutility on May 21, 2024 | hide | past | favorite | 12 comments


> The new tokenizer has 200,000 tokens in total, and about 25% are in non-English languages.

You have to hand it to the British: of all the different peoples and nations who tried their hand at world domination, the tiny little island of Britain won.


Well, it's because of the Americans not the British.

The internet has primarily been built out in the US.


Yeah but follow the effects back to their causes: the colonies that became America were British colonies, not French or Spanish or Portuguese or Dutch colonies.

In other words: America was founded by British people. Settlers/colonizers from Britain.


> the colonies that became America were British colonies, not French or Spanish or Portuguese or Dutch colonies.

Some of them were British from the outset, but not even all of the original 13 Colonies were originally British:

- Parts of New York, New Jersey, Delaware, and Connecticut were Dutch before they were British ("even old New York was once New Amsterdam / why they changed it, I can't say; people just liked it better that way")

- Parts of Delaware, Maryland, New Jersey, and Pennsylvania were Swedish before they were Dutch before they were British

And that's to say nothing about the various Spanish (and later Mexican), French, and Russian colonies annexed throughout the US' post-revolutionary history. If we're gonna assign credit for the Internet to old European imperial powers, Spain in particular probably has just as much (if not more) to do with it than Britain, given that California was originally a Spanish colony (and then a Mexican colony, and then briefly its own country before the US annexed it).


To be precise the two colonies that were not always British|English from the outset had both become British|English colonies for a full century prior to the American Revolutionary War.

https://en.wikipedia.org/wiki/Province_of_New_Jersey (and its division into East & West Jersey until Royal union under British Queen Anne in 1704)

https://en.wikipedia.org/wiki/Delaware_Colony


> America was founded by British people. Settlers/colonizers from Britain.

Not exactly - as pointed out not all the colonies were established by England (prior to the Union) and while under British authority by the time of the Revolutionary War the people within the colonies were not all from the United Kingdom as most were born in the Americas and a good number were descended from non-United Kingdom parts of Europe.


They blessed the world with the largest number of independance days


Don't hand anything to them. Analyze, Deconstruct and Sabotage the western white supremacist empire until it collapses and looses its death grip on humanity.


Britain seems to have made a Carthage.


pure curiosity, haha, just run it (https://gist.github.com/ctlllll/4451e94f3b2ca415515f3ee369c8...) through translate to english:

here are the first tokens till ctrl+c ---

"It's a bit outrageous..."

Japanese Adult Videos Free to Watch

China Welfare Lottery Daily

Long-lasting Free Hot Online Selections

Uncensored High Definition Free Videos

DaFa Fast Three - Big/Small, Single/Double

Leave Something for the Master

QQ's Daily Lottery

Japanese Full-Length Adult Film

DaFa Fast Three Lottery Results

Invitation Code for CaiShen Competition

Free Videos Online Watch

Uncensored High Definition Free Videos

Uncensored Videos - First, Second, Third Zones

DaFa Timely Lottery Plan

DaFa Timely Lottery Results

DaFa Timely Lottery How-to

DaFa Fast Three by the State?


Aww. The data they stole isn’t ideal? Such a shame.


well, that's a given, bro. you guys only steal yet you complain when what you steal makes your result "poor".




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: