Just curious whether it's possible to push any of your performance improvements ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		pamelafox 78 days ago \| parent \| context \| favorite \| on: Show HN: TokenDagger – A tokenizer faster than Ope... Just curious whether it's possible to push any of your performance improvements to tiktoken itself?

matthewolfe 78 days ago [–]

I probably will. Was hesitant initially, because adding PCRE2 as a dependency might cause issues to existing projects. I believe this was discussed briefly in a closed PR with other performance improvements.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact