aja ddlZddlmZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZddlZddlmZmZmZmZmZmZddlmZmZmZmZddlmZmZdd lm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&m'Z'm(Z(dd l)m*Z*m+Z+ddl,m-Z-ddl.m/Z/dZ0dZ1dZ2ejge0dZ4ejge1dZ5eGddeZ6ddde e&de e7de edefdZ8ddde e&de9de edefdZ:de e7de6fdZ;d e7d!e7de6fd"Zd&Z?eGd'd(Z@Gd)d*e/ZAd+ZBy),N) dataclass)partial) AnyCallableDictIterableListOptionalTupleUnioncast)ConfigModelOps Optimizerget_current_opsset_dropout_rate)Floats2dInts1dInts2dRagged)Protocolruntime_checkable)Errors)Language)Scorer)DocSpan SpanGroup)Examplevalidate_examples)Vocab) TrainablePipea4 [model] @architectures = "spacy.SpanCategorizer.v1" scorer = {"@layers": "spacy.LinearLogistic.v1"} [model.reducer] @layers = spacy.mean_max_reducer.v1 hidden_size = 128 [model.tok2vec] @architectures = "spacy.Tok2Vec.v2" [model.tok2vec.embed] @architectures = "spacy.MultiHashEmbed.v2" width = 96 rows = [5000, 1000, 2500, 1000] attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"] include_static_vectors = false [model.tok2vec.encode] @architectures = "spacy.MaxoutWindowEncoder.v2" width = ${model.tok2vec.embed.width} window_size = 1 maxout_pieces = 3 depth = 4 a& [model] @architectures = "spacy.SpanCategorizer.v1" scorer = {"@layers": "Softmax.v2"} [model.reducer] @layers = spacy.mean_max_reducer.v1 hidden_size = 128 [model.tok2vec] @architectures = "spacy.Tok2Vec.v2" [model.tok2vec.embed] @architectures = "spacy.MultiHashEmbed.v1" width = 96 rows = [5000, 1000, 2500, 1000] attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"] include_static_vectors = false [model.tok2vec.encode] @architectures = "spacy.MaxoutWindowEncoder.v2" width = ${model.tok2vec.embed.width} window_size = 1 maxout_pieces = 3 depth = 4 scmodelc2eZdZdddeedeedefdZy) SuggesterNopsdocsr+returncyN)selfr,r+s `/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/spacy/pipeline/spancat.py__call__zSuggester.__call__Qs) __name__ __module__ __qualname__rrr rrr3r0r4r2r)r)OsDHXXc]XHSMXVXr4r)r*r,sizesr+r-c | t}g}g}|D]}|jjt|d}|j d}d}|D]}|t|krX|dt||dz z } |j |jj | | |zf||djdz }|sl|djdk(rJ|dj|j ||j|} t|dkDr&t|jj|| } n't|jjdd| } | jjdk(sJ| S) Nidtype)r$rr$r=rrr)rxparangelenreshapeappendhstackshapendim asarray1irvstackzerosdataXd) r,r8r+spanslengthsdocstartslengthsize starts_size lengths_arrayoutputs r2ngram_suggesterrTTsh { EG s3xs3( ) rrKrCstartendr rasarrayrArr?rI) r,rUr+rKrLrMrOspanrRrSs r2preset_spans_suggesterr[qs { EG 99Y  ),  djj$((34!   vWC!@AM 5zA~ E 5}E M V3 7G Mr4c$tt|S)zSuggest all spans of the given lengths. Spans are returned as a ragged array of integers. The array has two columns, indicating the start and end position.r8)rrTr]s r2build_ngram_suggesterr^s ?% 00r4min_sizemax_sizecHtt||dz}t|S)zSuggest all spans of the given lengths between a given min and max value - both inclusive. Spans are returned as a ragged array of integers. The array has two columns, indicating the start and end position.r$)listranger^)r_r`r8s r2build_ngram_range_suggesterrds$ xA. /E  ''r4c$tt|S)zSuggest all spans that are already stored in doc.spans[spans_key]. This is useful when an upstream component is used to set the spans on the Doc such as a SpanRuler or SpanFinder.rU)rr[rfs r2build_preset_spans_suggesterrgs )Y ??r4examplesc t|}d|d|jd|jdd|jdfd|jdfd tj|fi|S) Nspans_rUattr allow_overlapTgettercT|jj|tdgSr/)rKgetrA)rMkey attr_prefixs r2zspancat_score..s#399==S5E5G1H"#Mr4has_annotationc |jvSr/)rK)rMrps r2rrzspancat_score..sC3994Dr4)dict setdefaultr score_spans)rhkwargsrqrps @@r2 spancat_scorerys &\FK  C f cU34 ot, M &(DE   h 1& 11r4ctSr/)ryr0r4r2make_spancat_scorerr{s r4c"eZdZdZdZdZdZy) _Intervalsz: Helper class to avoid storing overlapping spans. c"t|_yr/)setrangesr1s r2__init__z_Intervals.__init__s e r4c\t||D]}|jj|yr/)rcradd)r1r:jes r2rz_Intervals.adds'q! A KKOOA  r4cP|\}}t||D]}||jvsyy)NTF)rcr)r1rangr:rrs r2 __contains__z_Intervals.__contains__s41q! ADKK r4N)r5r6r7__doc__rrrr0r4r2r}r}sr4r}ceZdZdZ dd7e d8e+d9e dede,f d;Z.y)?SpanCategorizerz_Pipeline component to label spans of text. DOCS: https://spacy.io/api/spancategorizer FrK?TNg?)add_negative_labelrUnegative_weightrl max_positive thresholdscorervocabr' suggesternamerrUrrlrrrr-cg|| | ||d|_||_||_||_||_| |_||_|s2| /| dkDr)ttjj| yyy)a}Initialize the multi-label or multi-class span categorizer. vocab (Vocab): The shared vocabulary. model (thinc.api.Model): The Thinc Model powering the pipeline component. For multi-class classification (single label per span) we recommend using a Softmax classifier as a the final layer, while for multi-label classification (multiple possible labels per span) we recommend Logistic. suggester (Callable[[Iterable[Doc], Optional[Ops]], Ragged]): A function that suggests spans. Spans are returned as a ragged array with two integer columns, for the start and end positions. name (str): The component instance name, used to add entries to the losses during training. spans_key (str): Key of the Doc.spans dict to save the spans under. During initialization and training, the component will look for spans on the reference document under the same key. Defaults to `"spans"`. add_negative_label (bool): Learn to predict a special 'negative_label' when a Span is not annotated. threshold (Optional[float]): Minimum probability to consider a prediction positive. Defaults to 0.5. Spans with a positive prediction will be saved on the Doc. max_positive (Optional[int]): Maximum number of labels to consider positive per span. Defaults to None, indicating no limit. negative_weight (float): Multiplier for the loss terms. Can be used to downweight the negative samples if there are too many when add_negative_label is True. Otherwise its unused. allow_overlap (bool): If True the data is assumed to contain overlapping spans. Otherwise it produces non-overlapping spans greedily prioritizing higher assigned label scores. Only used when max_positive is 1. scorer (Optional[Callable]): The scoring method. Defaults to Scorer.score_spans for the Doc.spans[spans_key] with overlapping spans allowed. DOCS: https://spacy.io/api/spancategorizer#init )labelsrUrrrrlNr$)r) cfgrrr'rrr ValueErrorrE1051format) r1rr'rrrrUrrlrrrs r2rzSpanCategorizer.__init__sf""(.* $  "   "4!9lQ>NV\\00l0KL L?O!9}r4c2t|jdS)zKey of the doc.spans dict to save the spans under. During initialization and training, the component will look for spans on the reference document under the same key. rU)strrrs r2rpzSpanCategorizer.keys 488K())r4c,d}|jjdr|jjd}no|jjdrT|jj djdr*|jj djd}|j||j k(rZ|j sMttjj|j|jjdyyy)zbDNN2$$ KK&&DII$**:L:LT:R&S%3>r4labelct|tsttj||j vry|j |jdj||jjj|y)zAdd a new label to the pipe. label (str): The label to add. RETURNS (int): 0 if label is already present, otherwise 1. DOCS: https://spacy.io/api/spancategorizer#add_label rrr$) isinstancerrrE187rrrrCrstringsr)r1rs r2 add_labelzSpanCategorizer.add_label%sj%%V[[) ) DKK  ! !!%( u%r4c2t|jdS)zRETURNS (Tuple[str]): The labels currently added to the component. DOCS: https://spacy.io/api/spancategorizer#labels r)tuplerrs r2rzSpanCategorizer.labels6s TXXh'((r4c,t|jS)zRETURNS (List[str]): Information about the component's labels. DOCS: https://spacy.io/api/spancategorizer#label_data )rbrrs r2 label_datazSpanCategorizer.label_data>s DKK  r4c`t|jDcic]\}}|| c}}Scc}}w)z(RETURNS (Dict[str, int]): The label map.) enumerater)r1r:rs r2 _label_mapzSpanCategorizer._label_mapFs)*34;;)?@XQq@@@s *ct|jrt|jdzSt|jS)z RETURNS (int): Number of labels.r$)rrArrs r2rzSpanCategorizer._n_labelsKs0  " "t{{#a' 't{{# #r4cF|jrt|jSy)z8RETURNS (Union[int, None]): Index of the negative label.N)rrArrs r2_negative_label_iz!SpanCategorizer._negative_label_iSs  " "t' 'r4r,c |j||jj}|jj dk(r*|jjj dd}||fS|jj ||f}||fS)zApply the pipeline's model to a batch of docs, without modifying them. docs (Iterable[Doc]): The documents to predict. RETURNS: The models prediction for each document. DOCS: https://spacy.io/api/spancategorizer#predict r*r)rr'r+rLsumalloc2fpredict)r1r,indicesscoress r2rzSpanCategorizer.predict[s}..4::>>.: ??   A %ZZ^^++Aq1FZZ''w8Fr4 candidates)candidates_keyrc|j||jj}t||D]L\}}g|j|<|j D])}|j|j ||d|d+Ny)aoUse the spancat suggester to add a list of span candidates to a list of docs. This method is intended to be used for debugging purposes. docs (Iterable[Doc]): The documents to modify. candidates_key (str): Key of the Doc.spans dict to save the candidate spans under. DOCS: https://spacy.io/api/spancategorizer#set_candidates r*rr$N)rr'r+ziprKrJrC)r1r,rsuggester_outputrrMindexs r2set_candidateszSpanCategorizer.set_candidatesjs >>$DJJNN>C"#3T: KOJ(*CIIn %#** K .)00U1Xa1IJ K Kr4c |\}}d}t|D]\}}||j}tt|jd} |jddk(r?|j ||||||j |z| |j|j<n=|j||||||j |z|j|j<||j |z }y)aModify a batch of Doc objects, using pre-computed scores. docs (Iterable[Doc]): The documents to modify. scores: The scores to set, produced by SpanCategorizer.predict. DOCS: https://spacy.io/api/spancategorizer#set_annotations rrlrr$N) rrJr boolr_make_span_group_singlelabelrLrKrp_make_span_group_multilabel) r1r,indices_scoresrroffsetr:rM indices_irls r2set_annotationszSpanCategorizer.set_annotations|s)o )FAs ))I txx'@AMxx'1,&*&G&G6FW__Q-?$?@! ' $((#'+&F&F6FW__Q-?$?@' $((# gooa( (F! )r4)dropsgdlossesrhrrrcn|i}|j|jdt|d|j|t d|Ds|S|Dcgc]}|j }}|j ||jj}|jjdk(r|St|j||jj||f\}} |j|||f\} } | | ||j|||jxx| z cc<|Scc}w)a1Learn from a batch of documents and gold-standard information, updating the pipe's model. Delegates to predict and get_loss. examples (Iterable[Example]): A batch of Example objects. drop (float): The dropout rate. sgd (thinc.api.Optimizer): The optimizer. losses (Dict[str, float]): Optional record of the loss during training. Updated using the component name as the key. RETURNS (Dict[str, float]): The updated losses dictionary. DOCS: https://spacy.io/api/spancategorizer#update rzSpanCategorizer.updatec3bK|]'}|jrt|jnd)yw)rN) predictedrA).0egs r2 z)SpanCategorizer.update..s$O 3r||$!;Os-/r*r)rvrr"_validate_categoriesanyrrr'r+rLrr begin_updateget_loss finish_update) r1rhrrrrr,rKrbackprop_scoreslossd_scoress r2updatezSpanCategorizer.updates( >F$))S)($<= !!(+OhOOM'/0 00t8 ==   ! #MT*"&**"9"94-"Hx%Ah! ?   s #tyyT! 1sD2 spans_scoresc|\}}t|jjj|j|jjj|j }t j|j|j}|jr"t j|jd}d}|j}t|D]\} } i} || j} t|j | D],} t!| | df}t!| | df}|| z| ||f<.|j#| D]L}|j$|j&f}|| vs | |}||j(}d|||f<|jsHd|<N||j | z }|jjj+|d}|jr)t j,d}d|||j.f<||z }|jr/t1t2|j4d}|dk7r |xx|zcc<t3|dzj7}||fS) akFind the loss and gradient of loss for the batch of documents and their predicted scores. examples (Iterable[Examples]): The batch of examples. spans_scores: Scores representing the model's predictions. RETURNS (Tuple[float, float]): The loss and the gradient. DOCS: https://spacy.io/api/spancategorizer#get_loss r;rr$rrfrr)rr'r+to_numpydatarLnumpyrIrEr<ronesrrrJrcint_get_aligned_spansrWrXlabel_rYnonzerorr floatrr)r1rhrrKrtargetnegative_spansr label_mapr:r spans_indexspans_irrWrX gold_spanrprowknegative_samplesr neg_weightrs r2rzSpanCategorizer.get_losssK% v JJNN # #EJJ /1H1H1W V\\>  " ""ZZa:NOO x( 'EArKAhooG5==+, 7GAqDM*'!Q$-(,2QJ UCL) 7"44R8 2   6+%%c*C!)"2"23A%(F36N...1s+ 2 emmA& &F+ ',''c':  " "$}}^vx?O9P< ue| <D#'&* $$r8G#445$$h  $$ c# $$  $$L Xg->  W ' '' '  '\# 5 55 5  5  5r4rc|dk(r!tjd}|jS|dk(r!tjd}|jSt dt d|)N make_spancatzspacy.pipeline.factoriesmake_spancat_singlelabelzmodule z has no attribute ) importlib import_moduler r!AttributeErrorr5)rmodules r2 __getattr__r&sf ~(()CD""" + +(()CD... 78*,>tfE FFr4)Cr" dataclassesr functoolsrtypingrrrrr r r r r r thinc.apirrrrrr thinc.typesrrrrcompatrrerrorsrlanguagerrrtokensrrr trainingr!r"rr#trainable_piper%spancat_default_config"spancat_singlelabel_default_configDEFAULT_SPANS_KEYfrom_strDEFAULT_SPANCAT_MODEL!DEFAULT_SPANCAT_SINGLELABEL_MODELr)rrTrr[r^rdrgryr{r}rr&r0r4r2r8s!TTT VV880))1)6&"4))*@A'J$*H$5$5&% % ! YYY DH 3- $S 3;C= <BF 3-$'19# .1c1y1(#(((@C@I@ 2HW- 2DcN 2  (GmGVGr4