jw dZy)c|yt|}tttdfd}|D]}t|dkDs||S)a This function is to create a new dictionary used for improving tokenization model for multi-syllable words languages such as vi, zh or th. This function takes the lexicon as input and output a dictionary that contains three set: words, prefixes and suffixes where prefixes set should contains all the prefixes in the lexicon and similar for suffixes. The point of having prefixes/suffixes sets in the dictionary is just to make it easier to check during data preparation. :param shorthand - language and dataset, eg: vi_vlsp, zh_gsdsimp :param lexicon - set of words used to create dictionary :return a dictionary object that contains words and their prefixes and suffixes. N)wordsprefixessuffixesc|dvr}dj|d}d}tdt|dz D]I}|||z}|t||z dz |z}dj|dj|Kyy)Nrrr)addrangelen)wordprefixsuffixi dictionarys d/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/minisbd/modules/dictionary.pyadd_wordz#create_dictionary..add_words z'* * w  # #D )FF1SYq[) 3$q')c$i!ma/069:&**62:&**62  3 +r )setr )lexiconrr rs @rcreate_dictionaryrsX'lG%CEceDJ 3 t9Q; TN rN)rrrrs r