SkillPeek tokenizer assets / 分词词表来源

Only tokenizer data is included; no model weights. / 仅包含分词数据，不包含模型权重。
GPT tokenizer: https://github.com/niieani/gpt-tokenizer (MIT)
Tokenizer runtime: https://github.com/huggingface/tokenizers.js (Apache-2.0)

qwen3
https://huggingface.co/Qwen/Qwen3-4B/tree/1cfa9a7208912126459214e8b04321603b3df60c
/tokenizers/qwen3/LICENSE

deepseek-v3
https://huggingface.co/deepseek-ai/DeepSeek-V3/tree/e815299b0bcbac849fa540c768ef21845365c9eb
/tokenizers/deepseek-v3/LICENSE-CODE
/tokenizers/deepseek-v3/LICENSE-MODEL

llama3
https://huggingface.co/onnx-community/Llama-3.2-1B-Instruct-ONNX/tree/82ac69368916173bad498ad390290faab92d4fb0
/tokenizers/llama3/LICENSE.txt
/tokenizers/llama3/USE_POLICY.md

Built with Llama
Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.