WeLM v4.5 分词器深度分析

9. 修复方案（按推荐度排序） 行动项

9.1 方案 B（强烈推荐）：改 tokenizer_class 为 PreTrainedTokenizerFast

操作：修改三个目录的 tokenizer_config.json：

// 修改前
{
  ...
  "tokenizer_class": "Qwen2Tokenizer"
}

// 修改后
{
  ...
  "tokenizer_class": "PreTrainedTokenizerFast"
}

需要修改的目录：

• /apdcephfs/sg-m2/group_airesearch_1/models/WeLM-v4.5-80B-A3B-Instruct-0608/
• /apdcephfs/sg-m2/group_airesearch_1/models/WeLM-v4.5-80B-A3B-Instruct-0608-stacked-cache/（顺手统一）
• /apdcephfs/sg-m2/group_airesearch_1/models/WeLM-v4.5-80B-A3B-Instruct-0608-VLM-OE-stacked-cache/（顺手统一）

9.2 方案 A（仅 5.x 用户）：改为 TokenizersBackend

9.3 方案 C（长期）：下一代 WeLM 切换主流派

9.4 调试时的诊断代码

建议在 check_tokenizer.py 里加上以下打印，帮助快速定位走的是哪条路径：

import transformers
from transformers import AutoTokenizer

print(f"[INFO] transformers 版本: {transformers.__version__}")
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

print(f"[INFO] tokenizer 类型: {type(tokenizer).__name__}")
print(f"[INFO] is_fast: {tokenizer.is_fast}")
if hasattr(tokenizer, "backend_tokenizer"):
    print(f"[INFO] backend pre_tokenizer: {tokenizer.backend_tokenizer.pre_tokenizer}")

ids = tokenizer.encode("1234567890123", add_special_tokens=False)
print(f"[INFO] encode('1234567890123'): {ids}")
print(f"[INFO] decoded tokens: {[tokenizer.decode([i]) for i in ids]}")

10. 附录：相关代码片段与验证脚本

10.1 5.x Qwen2Tokenizer 关键源码

路径：transformers/models/qwen2/tokenization_qwen2.py（transformers 5.8.1）

PRETOKENIZE_REGEX = (
    r"(?i:'s|'t|'re|'ve|'m|'ll|'d)"
    r"|[^\r\n\p{L}\p{N}]?\p{L}+"
    r"|\p{N}"
    r"| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+"
)

class Qwen2Tokenizer(TokenizersBackend):
    def __init__(self, vocab_file, merges_file, ..., **kwargs):
        ...
        self._tokenizer = Tokenizer(BPE(vocab=vocab, merges=merges, ...))
        self._tokenizer.normalizer = normalizers.NFC()
        self._tokenizer.pre_tokenizer = pre_tokenizers.Sequence([
            pre_tokenizers.Split(Regex(PRETOKENIZE_REGEX),
                                 behavior="isolated", invert=False),
            pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False),
        ])

10.2 AutoTokenizer 决策逻辑

路径：transformers/models/auto/tokenization_auto.py

elif tokenizer_config_class is not None:
    tokenizer_class_candidate = tokenizer_config_class
    tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)

    # 老的 slow tokenizer 类名 → 重定向到新后端
    if tokenizer_class is not None and tokenizer_class.__name__ == "PythonBackend":
        tokenizer_class = TokenizersBackend

    # 找不到类名 → 兜底走 TokenizersBackend，原样加载 tokenizer.json
    if tokenizer_class is None:
        tokenizer_class = TokenizersBackend

    return tokenizer_class.from_pretrained(
        pretrained_model_name_or_path, *inputs, **kwargs
    )

10.3 业界横向对比扫描脚本（核心片段）

import json, os
from tokenizers import Tokenizer

ROOT = "/apdcephfs/sg-m2/group_airesearch_1/models"

def collect_split_regex(pre, out):
    if isinstance(pre, dict):
        if pre.get("type") == "Split":
            pat = pre.get("pattern", {})
            if isinstance(pat, dict) and "Regex" in pat:
                out.append(pat["Regex"])
        for v in pre.values():
            collect_split_regex(v, out)
    elif isinstance(pre, list):
        for v in pre:
            collect_split_regex(v, out)

def digit_strategy(regs):
    joined = " || ".join(regs)
    if r"(?=(\d{3})+(?!\d))" in joined:
        return "3-digit-group (派别 C / Qwen2 老式)"
    if r"\p{N}{1,3}" in joined or r"[0-9]{1,3}" in joined:
        return "1~3-digit-chunk (派别 B / 主流新派)"
    if "\\p{N}" in joined or "[0-9]" in joined:
        return "single-digit (派别 A)"
    return "no-explicit-digit-rule"

for name in os.listdir(ROOT):
    tj = os.path.join(ROOT, name, "tokenizer.json")
    if not os.path.exists(tj):
        continue
    with open(tj) as f:
        d = json.load(f)
    regs = []
    collect_split_regex(d.get("pre_tokenizer"), regs)
    print(f"{name:60s} -> {digit_strategy(regs)}")

    # 实测
    tk = Tokenizer.from_file(tj)
    enc = tk.encode("The number is 1234567890123 today.")
    digit_toks = [t for t in enc.tokens if any(c.isdigit() for c in t)]
    print(f"  digit_tokens={digit_toks}")

10.4 关键参考路径