Hacker News new | past | comments | ask | show | jobs | submit login

Good point, assuming some extent of collapse is crucial, and the question is if different perplexities due to tokenization can happen in principle. You are right that in "Alice" vs. "A|lice" example we get the same perplexity after re-normalization, I can't come up with an example where it would be different right now.



Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: