jH(ddlZddlZejdZejdZejdZdZGddZd dZ d Z Gd d Z y) Nz\n\s*\nz^[\d]+([,\.]+[\d]+)*[,\.]*$z\scg}t|D]4\}\}}|dkDr|dk(r||dz ddk(r"|j||f6|S)Nr ) enumerateappend)parafilteredicharlabels ^/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/minisbd/modules/data.pyfilter_consecutive_whitespacesr s^H%dO'=D% q5s{tAaCy|s2u & ' Oc8eZdZdfd ZdZdZdZdZxZS)TokenizationDatasetct||i|||_||_||_t j |}|Dcgc]}|j}}|Dcgc]}|s| }}|D cgc]}|D cgc]} dc} } }} |jjdd} t|| D cgc]A\}} t|| D cgc]$\} }| r| dk(stjd| |f&c}} Cc}} } }|_ |jDcgc] }t|c}|_ ycc}wcc}wcc} wcc} }wcc}} wcc}} } }wcc}w)Nr skip_newlineF r)super__init__args dictionaryvocabNEWLINE_WHITESPACE_REsplitrstripgetzip WHITESPACE_REsubdatar)selfconfigtextrrrkwargs text_chunkspt_labelsrpcr r x __class__s r rzTokenizationDataset.__init__s= $)&) $ +11$7 -89rryy{9 9$/6b2r6 6,78bb/1/88yy}}^U; $'{F#;==R*-R]%$\dVZl%((d3U;]= AE J13A6J :6!8]= KsGD0D5$D5/ D?8 D: D?6E )E6E E:D?E c |jDcgc]'}tjtd|D)c}Scc}w)z Returns a list of the labels for all of the sentences in this DataLoader Used at eval time to compare to the results, for example c3&K|] }|d ywrN.0r+s r z-TokenizationDataset.labels..1s1qad1s)r!nparraylist)r"sents r r)zTokenizationDataset.labels+s2 @DyyIt1D112IIIs,>ct|}t|jdDcgc]}d}}t|jdDcgc]}d}}||d}||d}d} d} td|jddzD]} || z|dz krN| rL|||| zdjz }||jdvrdnd} | || dz <||jdvrd} || z dk\rN| rL||| z dj|z}||jdvrdnd} | || dz <||jdvrd} | r| r||zS||zScc}wcc}w) zT This function is to extract dictionary features for each character num_dict_featrTrwordsprefixesFsuffixes)lenrangerlowerr) r"ridxlengthr dict_forward_featsdict_backward_feats forward_word backward_wordprefixsuffixwindowfeats r extract_dict_featz%TokenizationDataset.extract_dict_feat3sT).tyy/I)JKAaKK*/ /0J*KLQqLLCy| S ! Adii8:; Ff )fSZ 0 3 9 9 ;; (DOOG,DDq!/3"6!8,tz'BB"Ff "v $SZ 0 3 9 9 ;m K )T__W-EEq104#F1H-  (CC"F&!$777/ ."$777;LLs E  Ecg}gjdD]S}|dk(s|dk(r|dk(rd}n,|dk(rd}n#|dk(rd }ntd j|j|Ufd }fd }djdv}djdv}jd } g} g} g} t |D]\} \}}||}|r&| t |dz k(rdnd}|j||r| dk(rdnd}|j|| rj || }||z}| j|| j|| j|t | dkDr|j|| | | |S)z7 Convert a paragraph to a list of processed sentences. feat_funcs end_of_para start_of_para space_beforec,|jdrdSdS)Nrrr) startswithr+s r z7TokenizationDataset.para_to_sentences..asall3&7Qr capitalizedc0|djrdSdS)Nrr)isupperrRs r rSz7TokenizationDataset.para_to_sentences..csadlln!rnumericc4tj|dSdS)Nrr) NUMERIC_REmatchrRs r rSz7TokenizationDataset.para_to_sentences..esz'7'7':'FQrz#Feature function "{}" is undefined.c:Dcgc] }|| c}Scc}wNr0)r+ffuncss r rSz7TokenizationDataset.para_to_sentences..ls%#8QAaD#8#8sctj|Dcgc]}jj|c}tj|tj|t |fScc}wr\)r4r5runit2idr6) sent_units sent_labels sent_featsyr"s r process_sentencez?TokenizationDataset.para_to_sentences..process_sentencensVHHZHdjj003HIHH[)HHZ($& &Hs"A1use_dictionaryrr)r ValueErrorformatrrr=rJ)r"rres feat_funcfunccomposite_funcreuse_end_of_parause_start_of_pararf current_unitscurrent_labels current_featsr unitr featsr] dict_featsr^s` @r para_to_sentencesz%TokenizationDataset.para_to_sentencesXs<0 IM)Y/-IN*>m+;i'N !F!M!Mi!XYY LL   9 & (499\+BB+tyy/FF#34  )$ ( A}e"4(Ec$ik)Aq Q aAQ Q!33D!<  *   &  ! !% (   '# (& }  ! JJ' ~}U V rc|jjd}|jjd}|\}}}}|jd} ||k7jdj } t dt || D} tjt|| f|tj} tjt|| fdtj} tjt|| | ftj}g}tt|D]}t||| |||<||||| |f| |d| |||z f<||||| |f| |d| |||z f<||||| |f||d| |||z f<|j!||||| |dg| | |z ||zzz| | ||fS)a Advance to a new position in a batch where we have partially processed the batch If we have previously built a batch of data and made predictions on them, then when we are trying to make prediction on later characters in those paragraphs, we can avoid rebuilding the converted data from scratch and just (essentially) advance the indices/offsets from where we read converted data in this old batch. In this case, eval_offsets index within the old_batch to advance the strings to process. zrc3,K|] \}}||z ywr\r0)r2r ls r r3z8TokenizationDataset.advance_old_batch..s>daac>sdtypeN)rr`shapesumtolistmaxrr4fullr=int64int32zerosfloat32r>minr)r" eval_offsets old_batchunkidpadidounitsolabels ofeaturesoraw feat_sizelenspad_lenunitsr)features raw_unitsr s r advance_old_batchz%TokenizationDataset.advance_old_batchs ""7+ ""7++4(DOOB' %$$Q'..0>c,&=>>Vg.RXXF#f+w/288D88S['9=RZZP s6{# sA!,q/47;LO5;A|AtTUw?VX6YF12Q,q/122 38A!\RS_UYZ[U\E\B\8]HQ4$q'LO344 5   T!W\!_T!W= WW[\]W^M^amnoapMp@qq r  sfh 11r)NNN) __name__ __module__ __qualname__rr)rJrur __classcell__)r,s@r rrs K&J#8J8t2rrc|sggfSrtt|fd|}ntt|d|}tt|}|d|dfS)z Sort data and return both the data and the original indices. One useful application is to sort by length, which can be done with key=len Returns the data as a sorted list, then the indices of the original list. c|dSNrr0)r+keys r rSz#sort_with_indices..sAaD r)rreversec |dSrr0rRs r rSz#sort_with_indices..s !rrr)sortedrtupler)r!rrorderedresults ` r sort_with_indicesrsZ 2v  4.A7S4ngN 3= !F !9fQi rct|t|k(sJdt|dk(rgSttt||Dcgc] }t|c}\}}|Scc}w)z: Unsort a sorted list, based on the original idx. z9Number of list elements must match with original indices.r)r=rrr6) sorted_listoidxtr(unsorteds r unsortrsg { s4y (e*ee ( ;1 $'D+0F)G$HIq47IKAx OJs A&c.eZdZdZdZdZdZdZdZy) SortedDatasetal Holds a TokenizationDataset for use in a torch DataLoader The torch DataLoader is different from the DataLoader defined here and allows for cpu & gpu parallelism. Updating output_predictions to use this class as a wrapper to a TokenizationDataset means the calculation of features can happen in parallel, saving quite a bit of time. cv||_t|jjt\|_|_y)N)r)datasetrr!r=indices)r"rs r rzSortedDataset.__init__s( "3DLL4E4E3"O 4??rc.t||jSr\)rr)r"arrs r rzSortedDataset.unsortsc4<<((rc  td|Dr td|dddjd}|jjj d}t d|Ddz}tjt||f|tj }tjt||fdtj }tjt|||ftj }g}t|D]l\} } | d\} } } }| || dt| f<| || dt| f<| || dt| ddf<|j|dg|t|z zzn||||fS) Nc38K|]}t|dkDywr/r=r1s r r3z(SortedDataset.collate..s+as1vz+sz:Expected all paragraphs to have no preset sentence splits!rrxrwc3>K|]}t|ddyw)rNrr1s r r3z(SortedDataset.collate..s4qc!A$q'l4srr{)anyrgr}rrr`rr4rr=rrrrrr)r"samplesrrrrr)rrr sampleu_l_f_r_s r collatezSortedDataset.collateso +7+ +YZ ZAJqM!$**2.  ""**734G44q8Ww/bhhG#g,0"BHHE88S\7I>bjjQ "7+ CIAv#AYNBB!#E!Xc"gX+ "$F1hs2wh; ')HQR!^ $   R7)wR/@"AA B  Cfh 11rN) rrr__doc__rrrrrr0rr rrs"P@)2rr)NF) renumpyr4compilerrYrrrrrrr0rr rsg # :.RZZ6 7  5!  X2X2x "+2+2r