perf: regexp2 pretokenizer, pooled buffers, offset-free fast encode path - #80

perf: regexp2 pretokenizer, pooled buffers, offset-free fast encode path#80

aeryncaen wants to merge 2 commits into

sugarme:mastersugarme/tokenizer:masterfrom

aeryncaen:perf/qwen-pretokenizer-regexp2aeryncaen/tokenizer:perf/qwen-pretokenizer-regexp2Copy head branch name to clipboard