Tokenizes chinese text into "words" for learning purposes and then renders the text in a GUI where you can click on a word to get the definition. It's not perfect, but a LLM fine tuned for it will eventually result in much better "tokenization".
Tokenizes chinese text into "words" for learning purposes and then renders the text in a GUI where you can click on a word to get the definition. It's not perfect, but a LLM fine tuned for it will eventually result in much better "tokenization".