aj|68ddlZddlmZddlmZmZmZmZmZm Z m Z ddl Z ddl m Z mZmZmZddlmZddlmZddlmZdd lmZdd lmZmZmZdd lmZd d lm Z dZ!e jEe!dZ#dZ$dZ%deedee&effdZ'dZ(Gdde Z)dZ*y)N)islice)AnyCallableDictIterableListOptionalTuple)ConfigModel Optimizerset_dropout_rate)Errors)Language)Scorer)Doc)Examplevalidate_examplesvalidate_get_examples)Vocab) TrainablePipeaW [model] @architectures = "spacy.TextCatEnsemble.v2" [model.tok2vec] @architectures = "spacy.Tok2Vec.v2" [model.tok2vec.embed] @architectures = "spacy.MultiHashEmbed.v2" width = 64 rows = [2000, 2000, 500, 1000, 500] attrs = ["NORM", "LOWER", "PREFIX", "SUFFIX", "SHAPE"] include_static_vectors = false [model.tok2vec.encode] @architectures = "spacy.MaxoutWindowEncoder.v2" width = ${model.tok2vec.embed.width} window_size = 1 maxout_pieces = 3 depth = 2 [model.linear_model] @architectures = "spacy.TextCatBOW.v3" exclusive_classes = true length = 262144 ngram_size = 1 no_output_layer = false modelz [model] @architectures = "spacy.TextCatBOW.v3" exclusive_classes = true length = 262144 ngram_size = 1 no_output_layer = false a` [model] @architectures = "spacy.TextCatReduce.v1" exclusive_classes = true use_reduce_first = false use_reduce_last = false use_reduce_max = false use_reduce_mean = true [model.tok2vec] @architectures = "spacy.HashEmbedCNN.v2" pretrained_vectors = null width = 96 depth = 4 embed_size = 2000 window_size = 1 maxout_pieces = 3 subword_features = true examplesreturnc 4tj|dfddi|S)Ncats multi_labelF)r score_cats)rkwargss `/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/spacy/pipeline/textcat.py textcat_scorer#Ls/        ctSN)r#r$r"make_textcat_scorerr(Us r$c eZdZdZ d$eddedededede e dd f d Z e d Z e deefd Ze deefd ZdeefdZdeedd fdZdd d ddeedede ede eeefdeeeff dZdd d ddeedede ede eeefdeeeff dZdeedeej8ej8ffdZdeedeeeffdZdedefdZ d d d dde geefde e!d e eed!e edd f d"Z"deefd#Z#y )%TextCategorizerzmPipeline component for single-label text classification. DOCS: https://spacy.io/api/textcategorizer )scorervocabrname thresholdr+rNcv||_||_||_d|_g|dd}t ||_||_y)aaInitialize a text categorizer for single-label classification. vocab (Vocab): The shared vocabulary. model (thinc.api.Model): The Thinc Model powering the pipeline component. name (str): The component instance name, used to add entries to the losses during training. threshold (float): Unused, not needed for single-label (exclusive classes) classification. scorer (Optional[Callable]): The scoring method. Defaults to Scorer.score_cats for the attribute "cats". DOCS: https://spacy.io/api/textcategorizer#init N)labelsr.positive_label)r,rr-_rehearsal_modeldictcfgr+)selfr,rr-r.r+r4s r"__init__zTextCategorizer.__init___sE,   $""  9 r$cy)NFr'r5s r"support_missing_valuesz&TextCategorizer.support_missing_valuess r$c2t|jdS)zRETURNS (Tuple[str]): The labels currently added to the component. DOCS: https://spacy.io/api/textcategorizer#labels r0)tupler4r8s r"r0zTextCategorizer.labelss TXXh'((r$c|jS)zRETURNS (List[str]): Information about the component's labels. DOCS: https://spacy.io/api/textcategorizer#label_data )r0r8s r" label_datazTextCategorizer.label_datas {{r$docsctd|Dss|Dcgc]}|j}}|jjj}|j t t|t |jf}|S|jj|}|jjj|}|Scc}w)zApply the pipeline's model to a batch of docs, without modifying them. docs (Iterable[Doc]): The documents to predict. RETURNS: The models prediction for each document. DOCS: https://spacy.io/api/textcategorizer#predict c32K|]}t|ywr&len.0docs r" z*TextCategorizer.predict..,3s8,) anytensorropsxpzerosrBlistr0predictasarray)r5r>rEtensorsrLscoress r"rOzTextCategorizer.predicts,t,,-12cszz2G2""BXXs4:DKK0@ABFM##D)''/ 3sCct|D]?\}}t|jD]"\}}t|||f|j|<$Ay)aModify a batch of Doc objects, using pre-computed scores. docs (Iterable[Doc]): The documents to modify. scores: The scores to set, produced by TextCategorizer.predict. DOCS: https://spacy.io/api/textcategorizer#set_annotations N) enumerater0floatr)r5r>rRirEjlabels r"set_annotationszTextCategorizer.set_annotationssQ o 6FAs%dkk2 65"'q!t "5 6 6r$)dropsgdlossesrr[r\r]c|i}|j|jdt|d|j|t d|Ds|St |j ||j j|Dcgc]}|jc}\}}|j||\}} || ||j|||jxx|z cc<|Scc}w)a1Learn from a batch of documents and gold-standard information, updating the pipe's model. Delegates to predict and get_loss. examples (Iterable[Example]): A batch of Example objects. drop (float): The dropout rate. sgd (thinc.api.Optimizer): The optimizer. losses (Dict[str, float]): Optional record of the loss during training. Updated using the component name as the key. RETURNS (Dict[str, float]): The updated losses dictionary. DOCS: https://spacy.io/api/textcategorizer#update rZzTextCategorizer.updatec3bK|]'}|jrt|jnd)yw)rN) predictedrB)rDegs r"rFz)TextCategorizer.update..s$O 3r||$!;Os-/) setdefaultr-r_validate_categoriesrIrr begin_updater`get_loss finish_update) r5rr[r\r]rarR bp_scoreslossd_scoress r"updatezTextCategorizer.updates( >F$))S)($<= !!(+OhOOMT* JJ33H4UbR\\4UV x8h( ?   s #tyyT! 5VsC'c4|i}|j|jd|j|St|d|j ||Dcgc]}|j }}t d|Ds|St|j||jj|\}}|jj|\} } || z } || ||j|||jxx| dzjz cc<|Scc}w)aPerform a "rehearsal" update from a batch of data. Rehearsal updates teach the current model to make predictions similar to an initial model, to try to address the "catastrophic forgetting" problem. This feature is experimental. examples (Iterable[Example]): A batch of Example objects. drop (float): The dropout rate. sgd (thinc.api.Optimizer): The optimizer. losses (Dict[str, float]): Optional record of the loss during training. Updated using the component name as the key. RETURNS (Dict[str, float]): The updated losses dictionary. DOCS: https://spacy.io/api/textcategorizer#rehearse rZzTextCategorizer.rehearsec32K|]}t|ywr&rArCs r"rFz+TextCategorizer.rehearse..rGrHr) rbr-r2rrcr`rIrrrdrfsum) r5rr[r\r]rar>rRrgtarget_gradients r"rehearsezTextCategorizer.rehearses, >F$))S)  (M($>? !!(+'/0 00,t,,MT* JJ33D9 ))66t< F?( ?   s #tyyhk..00 1sDc"tt|}tj|t|jfd}tj |t|jfd}t |D]m\}}t |jD]P\}}||jjvr|jj||||f<=|jsJd|||f<Ro|jjj|}||fS)Nf)dtyperZ) rBrNnumpyrMr0onesrT referencerr9rrKrP) r5r nr_examplestruths not_missingrVrarWrXs r"_examples_to_truthz"TextCategorizer._examples_to_truths$x.) k3t{{+;? !!(+"55h? jjnn,,[9 F?K%q[..0&'11r$rXct|tsttj||j vry|j |jdj||jrZd|jjvrB|jjd|jt|j |_ |jjj|y)zAdd a new label to the pipe. label (str): The label to add. RETURNS (int): 0 if label is already present, otherwise 1. DOCS: https://spacy.io/api/textcategorizer#add_label rr0 resize_outputr) isinstancestr ValueErrorrE187r0_allow_extra_labelr4appendrattrsrBr,stringsadd)r5rXs r" add_labelzTextCategorizer.add_label(s%%V[[) ) DKK  ! !!%( ::/TZZ-=-==:))/:4::s4;;GWXDJ u%r$)nlpr0r1 get_examplesrr0r1c|t|d|j||9|D].}|jjD]}|j |0n|D]}|j |t |j dkrttj|||j vr6tjj||j }t|t |j dk7r6tjj||j }t|||jd<tt|d} | D cgc]} | j } } |j#| \} } |j%t | dkDs/Jtj&j|j(t | dkDs/Jtj&j|j(|j*j-| | ycc} w) aInitialize the pipe for training, using a representative set of data examples. get_examples (Callable[[], Iterable[Example]]): Function that returns a representative sample of gold-standard Example objects. nlp (Language): The current nlp object the component is part of. labels (Optional[Iterable[str]]): The labels to add to the component, typically generated by the `init labels` command. If no labels are provided, the get_examples callback is used to extract the labels from the data. positive_label (Optional[str]): The positive label for a binary task with exclusive classes, `None` otherwise and by default. DOCS: https://spacy.io/api/textcategorizer#initialize zTextCategorizer.initializeNr) pos_labelr0r1 r)r-)XY)rrcyrrrBr0rrE867E920formatE919r4rNrrwr{_require_labelsE923r-r initialize)r5rrr0r1examplecatrXerrsubbatchra doc_sample label_sampleros r"rzTextCategorizer.initialize;s, l,HI !!,.1 >'> ("99>>(CNN3'( (  &u% & t{{ a V[[) )  %T[[0kk((>$++(V o%4;;1$kk((>$++(V o%%3!"|~r23-56rbll6 611(; a :"FFKK$6$6DII$6$FF"< 1$Hfkk&8&8dii&8&HH$  l; 7s(H9c|D]}t|jjj}|j ddkDr=t t jj|jj|D]6}|dk(r |dk(rt t jj|y)zKCheck whether the provided examples all have single-label cats annotations.g?r)valuerZ)valN) rNrwrvaluescountrrE895rE851)r5rexvalsrs r"rcz$TextCategorizer._validate_categoriesls BB ))0023Dzz#" !3!3",,:K:K!3!LMM Bs cSj$V[[%7%7C%7%@AA B  Br$)textcat)$__name__ __module__ __qualname____doc__r#rr rrUr rr6propertyr9r r0rr=rrrOrYrr rrjrqrundarrayr{reintrrrrcr'r$r"r*r*Ysm &3      "   D )c ))DIHSM$ 6HSM 6d 6 #'-1 #7## # i # c5j)* # c5j #R#'-1 )7#) ) i ) c5j)* ) c5j )V # ) # u}}emm+ , #2'!22uUE\?R2&ss.#'*.(, /<r8G#445/<h  /< #' /< ! /< /Br$r*cx|dk(r!tjd}|jStdtd|)N make_textcatzspacy.pipeline.factorieszmodule z has no attribute ) importlib import_modulerAttributeErrorr)r-modules r" __getattr__rxs@ ~(()CD""" 78*,>tfE FFr$)+r itertoolsrtypingrrrrrr r ru thinc.apir r r rerrorsrlanguagerr+rtokensrtrainingrrrr,rtrainable_pipersingle_label_default_configfrom_strDEFAULT_SINGLE_TEXTCAT_MODELsingle_label_bow_configsingle_label_cnn_configrr#r(r*rr'r$r"rsGGG @@HH)8 &x001LMgV*HW-DcN[Bm[B~Gr$