_jf* ddlmZddlmZddlmZmZmZmZm Z ddl Z ddl m Z ddl mZddlmZdd lmZmZmZmZeZdd ed eed ed dfdZGddeZGddZy)) OrderedDict)Path)AnyDictListOptionalUnionN) BloomFilter)Errors) get_string_id)SimpleFrozenDict ensure_pathload_language_dataregistrylangtablesstrictreturnLookupsct}|tjvrK|rGt|dkDr9t t j jdj|||Stjj|}|D]U}||vr/|r*t t j j||i}nt||}|j||W|S)aLoad the data from the spacy-lookups-data package for a given language, if available. Returns an empty `Lookups` container if there's no data or if the package is not installed. lang (str): The language code (corresponds to entry point exposed by the spacy-lookups-data package). tables (List[str]): Name of tables to load, e.g. ["lemma_lookup", "lemma_exc"] strict (bool): Whether to raise an error if a table doesn't exist. RETURNS (Lookups): The lookups container containing the loaded tables. rz, )tabler) rrlookupslen ValueErrorr E955formatjoingetr add_table)rrrrdatar language_datas W/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/spacy/lookups.py load_lookupsr$siG 8### c&kAoV[[//dii6Gd/ST T      %D0   !3!3%d!3!KLLM.tE{;M%/0 Nc eZdZdZeddedeeddfdZddeedeeddfdZ de ee fd e ddfd Z de ee fd e ddfd Zde ee fde fd Zdde ee fd ee de fdZde ee fdefdZdefdZdeddfdZy)TablezA table in the lookups. Subclass of builtin dict that implements a slightly more consistent and unified API. Includes a Bloom filter to speed up missed lookups. Nr!namerc:||}|j||S)zInitialize a new table from a dict. data (dict): The dictionary. name (str): Optional table name for reference. DOCS: https://spacy.io/api/lookups#table.from_dict r()update)clsr!r(selfs r# from_dictzTable.from_dict3s~ D r%ctj|||_d|_|t t |dn |j}t j||_|r|j|yy)zInitialize a new table. name (str): Optional table name for reference. data (dict): Initial data, used to hint Bloom Filter. DOCS: https://spacy.io/api/lookups#table.init g.ANr ) r__init__r( default_sizemaxrr from_error_ratebloomr+)r-r(r!sizes r#r0zTable.__init__@sd T" $($4s3t9a $:K:K 006  KK  r%keyvaluec~t|}tj||||jj |y)zSet new key/value pair. String keys will be hashed. key (str / int): The key to set. value: The value to set. N)r r __setitem__r4addr-r6r7s r#r9zTable.__setitem__Qs0 C c51 sr%c|||<y)zSet new key/value pair. String keys will be hashed. Same as table[key] = value. key (str / int): The key to set. value: The value to set. Nr;s r#setz Table.set[s S r%cDt|}tj||S)zGet the value for a given key. String keys will be hashed. key (str / int): The key to get. RETURNS: The value. )r r __getitem__r-r6s r#r@zTable.__getitem__ds! C &&tS11r%defaultcFt|}tj|||S)zGet the value for a given key. String keys will be hashed. key (str / int): The key to get. default: The default value to return. RETURNS: The value. )r rr)r-r6rBs r#rz Table.getms!C tS'22r%cbt|}||jvrytj||S)zCheck whether a key is in the table. String keys will be hashed. key (str / int): The key to check. RETURNS (bool): Whether the key is in the table. F)r r4r __contains__rAs r#rEzTable.__contains__ws0 C  djj ''c22r%c|jt|j|jj d}t j |S)zSerialize table to a bytestring. RETURNS (bytes): The serialized table. DOCS: https://spacy.io/api/lookups#table.to_bytes )r(dictr4)r(rGitemsr4to_bytessrsly msgpack_dumps)r-r!s r#rIzTable.to_bytessCII&ZZ((*  ""4((r% bytes_datactj|}|jdi}|d|_t j |d|_|j|j||S)zLoad a table from a bytestring. bytes_data (bytes): The data to load. RETURNS (Table): The loaded table. DOCS: https://spacy.io/api/lookups#table.from_bytes rGr(r4) rJ msgpack_loadsrr(r from_bytesr4clearr+)r-rLloadedr!s r#rOzTable.from_bytessa$$Z0zz&"%6N  ]--fWo>   D r%)N)NN)__name__ __module__ __qualname____doc__ classmethodrGrstrr.r0r intrr9r>r@rboolrEbytesrIrOr=r%r#r'r',s  T # '  Xc]$SW"uS#XstuS#Xst2uS#X2323uS#X3#3#3 3c3h 3D 3 )% )Uwr%r'ceZdZdZddZdedefdZdefdZ e de efdZ e fded edefd Zded eddfd Zefded edefdZdedefdZdedefdZdefdZdeddfdZ ddeeefdeddfdZ ddeeefdeddfdZy)raSContainer for large lookup tables and dictionaries, e.g. lemmatization data or tokenizer exception lists. Lookups are available via vocab.lookups, so they can be accessed before the pipeline components are applied (e.g. in the tokenizer and lemmatizer), as well as within the pipeline components via doc.vocab.lookups. rNci|_y)zXInitialize the Lookups object. DOCS: https://spacy.io/api/lookups#init N_tablesr-s r#r0zLookups.__init__s *, r%r(c$|j|S)zCheck if the lookups contain a table of a given name. Delegates to Lookups.has_table. name (str): Name of the table. RETURNS (bool): Whether a table of that name is in the lookups. ) has_tabler-r(s r#rEzLookups.__contains__s~~d##r%c,t|jS)z3RETURNS (int): The number of tables in the lookups.)rr^r_s r#__len__zLookups.__len__s4<<  r%cHt|jjS)z8RETURNS (List[str]): Names of all tables in the lookups.)listr^keysr_s r#rzLookups.tablessDLL%%'((r%r!c||jvr)ttjj |t ||}||j |<|S)aAdd a new table to the lookups. Raises an error if the table exists. name (str): Unique name of table. data (dict): Optional data to add to the table. RETURNS (Table): The newly added table. DOCS: https://spacy.io/api/lookups#add_table r*)r(r!)rrr E158rr'r^)r-r(r!rs r#r zLookups.add_tablesL 4;; V[[//T/:; ;4d+" T r%rc"||j|<y)zSet a table. name (str): Name of the table to set. table (Table): The Table to set. DOCS: https://spacy.io/api/lookups#set_table Nr])r-r(rs r# set_tablezLookups.set_tables# Tr%rBc||jvr?|tk(r4ttjj ||j |S|j|S)a<Get a table. Raises an error if the table doesn't exist and no default value is provided. name (str): Name of the table. default (Any): Optional default value to return if table doesn't exist. RETURNS (Table): The table. DOCS: https://spacy.io/api/lookups#get_table r(r)r^UNSETKeyErrorr E159rr)r-r(rBs r# get_tablezLookups.get_tablesO t|| #%v{{11tDKK1PQQN||D!!r%c||jvr4ttjj ||j |jj |S)zRemove a table. Raises an error if the table doesn't exist. name (str): Name of the table to remove. RETURNS (Table): The removed table. DOCS: https://spacy.io/api/lookups#remove_table rm)r^ror rprrpoprbs r# remove_tablezLookups.remove_tablesH t|| #6;;--4 -LM M||%%r%c||jvS)zCheck if the lookups contain a table of a given name. name (str): Name of the table. RETURNS (bool): Whether a table of that name exists. DOCS: https://spacy.io/api/lookups#has_table r]rbs r#razLookups.has_tablest||##r%c @tj|jS)zSerialize the lookups to a bytestring. RETURNS (bytes): The serialized Lookups. DOCS: https://spacy.io/api/lookups#to_bytes )rJrKr^)r-kwargss r#rIzLookups.to_bytess""4<<00r%rLc i|_tj|jD]\}}t |||j|< |S)zLoad the lookups from a bytestring. bytes_data (bytes): The data to load. RETURNS (Lookups): The loaded Lookups. DOCS: https://spacy.io/api/lookups#from_bytes )r^rJrNrHr')r-rLrwr6r7s r#rOzLookups.from_bytes sL --j9??A 2JC %c5 1DLL  2 r%pathfilenamec t|}|js|j||z }|jd5}|j |j dddy#1swYyxYw)zSave the lookups to a directory as lookups.bin. Expects a path to a directory, which will be created if it doesn't exist. path (str / Path): The file path. DOCS: https://spacy.io/api/lookups#to_disk wbN)rexistsmkdiropenwriterI)r-ryrzrwfilepathfile_s r#to_diskzLookups.to_disks^4 {{} JJL(? ]]4  )E KK  ( ) ) )s  A++A4c t|}||z }|jr;|jd5}|j}ddd|j S|S#1swYxYw)a Load lookups from a directory containing a lookups.bin. Will skip loading if the file doesn't exist. path (str / Path): The directory path. RETURNS (Lookups): The loaded lookups. DOCS: https://spacy.io/api/lookups#from_disk rbN)rr}rreadrO)r-ryrzrwrrr!s r# from_diskzLookups.from_disk)sc4 (? ?? t$ $zz| $??4( (  $ $s AA')rN)z lookups.bin)rRrSrTrUr0rWrYrErXrdpropertyrrrrGr'r rkrnrrqrtrarZrIrOr rrrr=r%r#rrsC,$$$!!)S ))1A0B c  u #c#%#D#38"c"C"E" & & &$c$d$1E1 U  7D)#t)$)03) )$7D#t)$03 r%)T) collectionsrpathlibrtypingrrrrr rJ preshed.bloomr errorsr stringsr utilrrrrobjectrnrWrYr$r'rr=r%r#rsj#33 %"MMsDIty:sKslXXr%