Fertility
A metric measuring how many tokens a tokenizer produces per word or semantic unit, with higher fertility indicating less efficient encoding and higher costs.
Fertility quantifies tokenization efficiency by measuring the ratio of tokens produced to words or characters in the input text. Languages underrepresented in tokenizer training data exhibit higher fertility, requiring more tokens to encode the same meaning. Because transformer attention scales quadratically with sequence length, doubled fertility quadruples computational cost. Research shows fertility differences of up to 15x between languages, creating a structural cost and quality disadvantage for non-English text.
Also known as
token fertility, tokenization fertility, multilingual tax, token tax