Results for “tokenisation”

Subword tokenisation degrades morphological generalisation in agglutinative low-resource languages

Byte-pair encoding and its variants underpin nearly all modern language models, but their behaviour on agglutinative morphology has been evaluated almost exclu…