]ju$dZddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZddlmZmZGddeZGd d Zy) zI Utility functions for the loading and conversion of CoNLL-format files. N)ZipFile)Document) IDTEXTLEMMAUPOSXPOSFEATSHEADDEPRELDEPSMISCNER START_CHAREND_CHAR) FIELD_TO_IDX FIELD_NUMc eZdZy) CoNLLErrorN)__name__ __module__ __qualname__\/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/stanza/utils/conll.pyrr srrceZdZed dZedZedZedZed dZed dZ ed dZ ed Z ed d Z y)CoNLLc fgg}}gg}}t|D]\}}|jjd}t|dk(r6t|dkDsB|j |g}|j |g}i|j dr|j ||j d}|rd|dvrt|tk7r*td|dzdtd t|d |d | ||gz }t|dkDr"|j ||j |||fS) a Load the file or string into the CoNLL-U format data. Input: file or string reader, where the data is in CoNLL-U format. Output: a tuple whose first element is a list of list of list for each token in each sentence in the data, where the innermost list represents all fields of a token; and whose second element is a list of lists for each comment in each sentence in the data. z r# .zCannot parse CoNLL line z : expecting z fields, z found at line z ) enumeratelstriprstriplenappend startswithsplitrr) fignore_gappingdocsent doc_comments sent_commentsline_idxlinearrays r load_conllzCoNLL.load_conllsbT&("m 'l NHd;;='' 2D4yA~t9q=JJt$D '' 6$&M??3'!((. 4(!cU1Xou:*$'? |z&CoNLL.convert_conll..E*aa3q6*ar"maxsplitc32K|]}t|ywr6r7r9s rr<z&CoNLL.convert_conll..Hr=r>-) r#rconvert_conll_token ValueErrorrstrrtupler)r') doc_conlldoc_dict doc_emptysent_idx sent_conll sent_dict sent_empty token_idx token_conll token_dictes r convert_conllzCoNLL.convert_conll4s?  $-i$8 ) HjIJ*3J*? 1& ;z!&!:!:;!GJ*R.(%**a:b>;O;OPS^_;O;`*a%aJrN%%j1%**a:b>;O;OPS^_;O;`*a%aJrN$$Z0 1 OOI &   Z ( ) """z$%QU]_hjmnojpTq%qrxyyzsC;; D$DD$c t|}dj|}|jd}|Dcgc]0}|jdDcgc]}|jdc}2}}}|Scc}wcc}}w)a Convert the dictionary format input data to the CoNLL-U format output data. This is the reverse function of `convert_conll`, but does not include sentence level annotations or comments. Can call this on a Document using `CoNLL.convert_dict(doc.to_dict())` Input: dictionary format data, which is a list of list of dictionaries for each token in each sentence in the data. Output: CoNLL-U format data as a list of list of list for each token in each sentence in the data. z{:c}z  r )rformatr))rHr,text sentencessentencer;rGs r convert_dictzCoNLL.convert_dictNshx }}S!JJv& S\]xX^^D-ABaggdmB] ]C]sA2 A- #A2-A2ci}tD]r}|t|}|dk7r|tk(rt|||<n|||<|ttdk(sE|tt|t<|tt|t<t|S)z Convert the CoNLL-U format input token to the dictionary format output token. Input: a list of all CoNLL-U fields for the token. Output: a dictionary that maps from field name to value. _)rr r8rr)rOrPfieldvalues rrCzCoNLL.convert_conll_token_s  ! EE U 34E|D=(+E Ju%(-Ju%<-.#5#.|D/A#B 4 $/ U0C$D 5! ErNcZt||gr t||grJd|r |sJd|r/tj|}tj ||\}}n|r]t |5}|j|5}tj tj|d|\}}ddddddn/t|d5}tj ||\}}dddtj\} } | | fS#1swY^xYw#1swY2xYw#1swY>xYw)zV Load the CoNLL-U format data from file or string into lists of dictionaries. z%either use input file or input stringz*must provide input_file if zip_file is setutf-8encodingN) anyallioStringIOrr3ropen TextIOWrapperrR) input_file input_strr+zip_fileinfilerGr.zinfinrHrIs r conll2dictzCoNLL.conll2dictss=J *+CY8O4PyRyyP JT(TTJ [[+F&+&6&6v~&N #I| " xcXXj)xS.3.>.>r?O?OPS^e?fhv.w+I|x x xj73 Ps*/*:*:3*O' < P$11)<)y00xx x x P Ps0+D=/D ,D D! D DD!D*cZtj||||\}}}t|d||S)NrjrVcommentsempty_sentences)rrnr)rhrir+rjrHr.rIs r conll2doczCoNLL.conll2docs6,1,<,F;'1,)/>1"*;TL\n{"|)T1%,%6%6B+9+?+?+AB G,)/ #&%K(0z$%*GM# ,&""3' ''1$$U+- ,. { a {@PbopG% **6A-335AH6 KK #N rcFt|}tj||y)zq Convert the dictionary format input data to the CoNLL-U format output data and write to a file. N)rrwrite_doc2conll)rHfilenamer,s r dict2conllzCoNLL.dict2conlls x  c8,rct|dr!|jdj|yt|||5}|jdj|dddy#1swYyxYw)a Writes the doc as a conll file to the given file. If passed a string, that filename will be opened. Otherwise, filename.write() will be called. Note that the output needs an extra at the end to be a legal output file writez{:C} r`N)hasattrrrUrf)r,rmoderaoutfiles rrzCoNLL.write_doc2conlls_ 8W % NN:,,S1 2hx8 6G j//45 6 6 6s !A&&A/)T)NNTN)wr_) rrr staticmethodr3rRrYrCrnrtrrrrrrrrs ! !D##2 &11(__''R-- 6 6rr)__doc__osrdzipfilerstanza.models.common.docrrrrrr r r r r rrrrrrrDrrrrrrs? -||||<  ~6~6r