Token Encoder

class TokenEncoder

TokenEncoder class for different tokenization encoding algorithms.

Public Types

enum Type

Token encoder algorithm type.

Values:

enumerator BPE

Byte Pair Encoding.

enumerator SENTENCEPIECE

SentencePiece encoding (unimplemented)

enumerator WORDPIECE

WordPiece encoding (unimplemented)

Public Functions

TokenEncoder(Type type = BPE) noexcept

Constructor for token encoder.

Parameters:

type – Encoder algorithm type (default: BPE)

~TokenEncoder() noexcept = default

bool initialize(

TokenToRanks const &vocab,

TokenToRanks const &specialTokens = {}

)

Initialize with vocabulary.

Parameters:

• vocab – Main vocabulary mapping

• specialTokens – Special tokens mapping

Returns:

true if vocab is non-empty and initialization completes; false if vocab is empty

bool encode(

std::string const &piece,

std::vector<Rank> &output

) const noexcept

Encode a piece of text using the algorithm.

Parameters:

• piece – Input text piece (already pretokenized)

• output – Vector to store token IDs

Returns:

true if piece is within size limits and encoding completes successfully; false if piece exceeds 1MB limit or encoding algorithm fails

bool decode(

std::vector<Rank> const &tokens,

std::string &output,

bool skipSpecialTokens = false

) const noexcept

Decode token IDs back to text.

Parameters:

• tokens – Vector of token IDs

• output – String to store result

• skipSpecialTokens – Whether to skip special tokens

Returns:

true if all tokens are found in vocabulary or skipped successfully; false if unknown tokens are encountered and skipSpecialTokens is false

inline Type getType() const noexcept

Get encoder type.

Returns:

Encoder algorithm type

inline size_t getVocabSize() const noexcept

Get vocabulary size.

Returns:

Total number of tokens in vocabulary

bool hasToken(std::string const &token) const noexcept

Check if token exists in vocabulary.

Parameters:

token – Token string to check

Returns:

true if token exists, false otherwise

Rank getTokenRank(std::string const &token) const noexcept

Get token rank from token string.

Parameters:

token – Token string

Returns:

Token rank/ID

std::string getRankToken(Rank rank) const

Get token string from rank.

Parameters:

rank – Token rank/ID

Returns:

Token string

inline void setByteFallback(bool enable) noexcept

Enable or disable byte-level fallback for unknown tokens.

Parameters:

enable – Whether to enable byte fallback

inline void setMergePriorities(

std::unordered_map<std::string, Rank> mergePriorities

) noexcept

Set explicit BPE merge priorities from tokenizer.json merges list.

When set, the BPE algorithm uses these priorities instead of vocabulary rank to determine merge order. This is required for SentencePiece-style tokenizers (e.g. Gemma) where merge order != vocabulary rank order.

Parameters:

mergePriorities – Map from null-byte-separated pair key (“a\0b”) to merge priority (0 = highest)