`jX^|ddlZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZmZmZdd lmZmZdd lmZdd lm Z m!Z!d d l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(e'jRde%dddde%dddde&dddde&ddddddde&ddd d!e&dd"d#d$e&dd%d&d'e&d(d)d*d+e&dd,d-d.d/e&dd0d1d1d2f d3ed4ed5e*d6e ed7e+d8e e*d9e e*d:e*d;e+de+d?e+dd]jwZ?gd^Z@y)`N) defaultdict)Path)AnyDictListOptionalTupleUnioncast) RegistryError)Config)MarkdownRendererPrinter get_raw_input)aboututil)importlib_metadata)ModelMetaSchemavalidate) SDIST_SUFFIX WHEEL_SUFFIXArgOptappstring_to_listpackage.zDirectory with pipeline dataTF)helpexists file_okayzOutput parent directoryz--codez-cznComma-separated paths to Python file with additional code (registered functions) to be included in the package)rz --meta-pathz--meta-mzPath to meta.json)rr dir_okayz --create-metaz-Cz$Create meta.json, even if one existsz--namez-nzPackage name to override metaz --versionz-vz Package version to override metasdistz--buildz-bz>Comma-separated formats to build: sdist and/or wheel, or none.z--forcez-fz-Fz3Force overwriting existing data in output directoryz$--require-parent/--no-require-parentz-Rz;Include the parent package (e.g. spacy) in the requirements input_dir output_dir code_paths meta_path create_metanameversionbuildforcerequire_parentc tt|\} } t|D cgc]} t| j}} t |||||||| | || d ycc} w)a% Generate an installable Python package for a pipeline. Includes binary data, meta and required installation files. A new directory will be created in the specified output directory, and the data will be copied over. If --create-meta is set and a meta.json already exists in the output directory, the existing values will be used as the defaults in the command-line prompt. After packaging, "python -m build --sdist" is run in the package directory, which will create a .tar.gz archive that can be installed via "pip install". If additional code files are provided (e.g. Python files containing custom registered functions like pipeline components), they are copied into the package and imported in the __init__.py. DOCS: https://spacy.io/api/cli#package F) r)r(r+r,r* create_sdist create_wheelr.r/silentN)get_build_formatsrrstripr) r&r'r(r)r*r+r,r-r.r/r1r2ps [/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/spacy/cli/package.py package_clir8smL"3>%3H!IL,+9*+EFa$qwwy/FJF  !!% Gs Ar1r2r3returnc t| | } tj|} tj|}tj|}|r*ts t sd}| j |ddt s| j d| r| js| j d| d|r|js| j d|d|s|r6|rd nd |rd nd g}| jd d jd|D|D]F}|js| j d|dtj|j|H|r| jdt|d|r$|js| j d|d|xs|dz }|jr|js| j d|dtj |}t#||| }|dr4| jdt|ddd j|d|P|j%s| j d|dddt'|s| j d|ddd||d<|||d<|s| jd|n t)|| }t+t,|}|r@| j d t/d!j|t1j2d|d}|j5|d"d#zs |d"d#|}|d$z|dz}||z }||z }|jr5| rt7j8t;|n| j d%d&dt=j>|d'(t7j@t;|t;||z tBD]C}||z |z }|jst7jDt;|t;|E|d)z }|js;tG|}tI||tI||z d)z || jd*n| jd+g}|D]E}|jK|jt7jDt;|t;|GtI|dz tjL|d,-tI|d.z tNtI|d/z tPtRjUd!jd0|D1}tI|d2z || jd3|d4||rtjV|5tYjZt0j\d5d6d7d8gt^j`jE9}|jbd:k7r9| j d;tjdt0j\d.d gd<=ddd|d>z |tfz }| jd?||rtjV|5tYjZt0j\d5d6d7d@gt^j`jE9}|jbd:k7r9| j dAtjdt0j\d.dBgd<=dddtijjdCd#|} |d>z | tlz }!| jdD|!dE|vr| j d|dFyy#1swY.xYw#1swYjxYw)GN)no_printprettyzJGenerating wheels requires 'build' or 'wheel' (deprecated) to be installedzpip install buildrexitszGenerating packages without the 'build' package is deprecated and will not be supported in the future. To install 'build': pip install buildzCan't locate pipeline datazOutput directory not foundr%r"wheelzBuilding package artifacts: , c3&K|] }|s| yw)N).0opts r7 zpackage..s9U#QT#9UszCan't find code filez Including z" Python module(s) with custom codezCan't find pipeline meta.jsonz meta.jsonzCan't load pipeline meta.json)r/ requirementsz, package requirement(s) from meta and configz Model name ('zR') is not a valid module name. This is required so it can be imported as a module.zWe recommend names that use ASCII A-Z, a-z, _ (underscore), and 0-9. For specific details see: https://docs.python.org/3/reference/lexical_analysis.html#identifiersza') is not a permitted package name. This is required to correctly load the model with spacy.load.zWe recommend names that use ASCII A-Z, a-z, _ (underscore), and 0-9. For specific details see: https://www.python.org/dev/peps/pep-0426/#namer+r,zLoaded meta.json from filezInvalid pipeline meta.json lang_-z Package directory already existszgPlease delete the directory and try again, or use the `--force` flag to overwrite existing directories.T)parents README.mdz"Generated README.md from meta.jsonz0Using existing README.md from pipeline directoryr)indentzsetup.pyz MANIFEST.inc3&K|] }d| yw)zfrom . import NrB)rCms r7rEzpackage..s@1N1#.@s)importsz __init__.pyz(Successfully created package directory ''r#r-.z--sdist)envrzgCreating sdist with 'python -m build' failed. Falling back to deprecated use of 'python setup.py sdist'F)capturedistz*Successfully created zipped Python packagez--wheelzzCreating wheel with 'python -m build' failed. Falling back to deprecated use of 'wheel' with 'python setup.py bdist_wheel' bdist_wheelz_+z!Successfully created binary wheel__ze') contains a run of underscores. Runs of underscores are not significant in installed package names.)7rr ensure_path has_wheel has_buildfailwarnr infojoin import_filestemgoodlenis_filesrsly read_jsonget_meta isidentifier_is_permitted_package_name generate_metarrprintsysexit startswithshutilrmtreestrrmkdircopytreeFILENAMES_DOCScopygenerate_readme create_fileappend json_dumpsTEMPLATE_SETUPTEMPLATE_MANIFEST TEMPLATE_INITformat working_dir subprocessrun executableosenviron returncode run_commandrresubr)"r&r'r)r(r+r,r*r1r2r/r.r3msg input_path output_patherropts code_pathmetaerrors model_name model_name_v main_path package_path file_name file_path readme_pathreadmerPinit_pyretzip_filewheel_name_squashedr?s" r7rrms 6f* 5C!!),J"":.K  +IIK X  )3 ;   Z..0 -zC k002 -{!D|'RLbQ / 9U9U0U/VWX4 ! HH+YaH @ 3 4  :c*o..PQR))+ 0)1E4Y4I    Y%6%6%8 0)1E ??9 %D ItN CD N T.1234  IId>* +    " HHv&FFr  *$/ HHv&PP[  V !Y  -y9T3' ot ,F  -. dii   fJ  f!3 4V ~Qzl3 #d9o5L\)Iz)L  MM#l+ , HH2D    JJ|T* OOC NC |(C$DE#8  </);     KKII 78k)K     &K(L</+=vF 56 CDG7 y~~& C NC $567 K')9)9$q)IJ J&7 M)+<="" @@@#G },g6HH7 ~Q GS   i ( W..wY?JJOO%C~~"H  #..*g!FPUV Wv%<.(GG =I   i ( ..wY?JJOO%C~~"4   ^^Z?  !ffT3 =F"(;'<\N%KK 4u= z J<(R R G W W   s>B\9B]9]]c, ddl}y#t$rYywxYw)NrTF)r? ImportError)r?s r7rYrYs s  cb tjdy#tj$rYywxYw)Nr-TF)rr,PackageNotFoundErrorrBr7rZrZ$s0""7+  2 2s ..configexcludec4d}tj}tt}dD]Q}tj||D]4\}}|dj ds||dddj |6S|jdijD]}d|vs|d j |d t} |d d } |jD]m\} } | D]c} tjj| | d z| z}|jd }|s>| j |d jd deog}| D]|}|tjk(r||vs|j|}|s0|d}||vs||vr>tj |}tj"|}|j%||~|S#t$r: tjj| | }n#t$r}|dd}~wwxYwYwxYw)aIf the config includes references to registered functions that are provided by third-party packages (spacy-transformers, other libraries), we want to include them in meta["requirements"] so that the package specifies them as dependencies and the user won't have to do it manually. We do this by: - traversing the config to check for registered function (@ keys) - looking up the functions and getting their module - looking up the module version and generating an appropriate version range config (Config): The pipeline config. exclude (list): List of packages to exclude (e.g. that already exist in meta). RETURNS (list): The versioned requirements. )spacyz spacy-legacyz spacy-nightlythincrd)nlp components@rNrfactory factoriesrrHrRmoduler)rpackages_distributionsrset walk_dictrmaddgetvaluesitemsregistryfindr splitr __title__get_package_versionget_minor_version_rangerw)rr own_packages distributionsfuncssectionpathvalue componentmodulesrHreg_name func_names func_name func_inforegerr module_name dependenciesrUpkgr, version_ranges r7get_third_party_dependenciesr/s?"PL//1M  E(/>>&/: /KD%Bx""3'd2hqrl#''. //ZZ b188:9  ! +  " "9Y#7 89eG %= D % ?*# ?I + MM..xi9OP $--1K Ih/55c:1=> ? ?L = %// )  - ' $$[1D1g,&#.2237 $ < H? H H H HHformatscgd}|D];}||vst}d|ddj|}|j|d=|rd|vryd |vd |vfS) N)r%r?nonezUnknown build format: z . Supported: r@rr=r)FFr%r?)rr^r[)r supportedformrrs r7r4r4ksu*I# y )C*4& dii >R=STC HHSH " # f' w 7 2 33rrcontentsch|j|jddj|y)Nwzutf-8)encoding)touchopenwrite)rrs r7rvrvws' OO NN3N)//9r model_path existing_metac ddddddddd}tjt|}|j|jtj t j|d<|j||jjt|jj|jjj|jjjd|d <t jd k7rt j|d <|jd g|d Dcgc]}tj |d }}t#|j$|}|d j'||r=t j|d vr(|d j)t j|dz|Scc}w)Nenpipeline0.0.0r"MIT)rHr+r, descriptionauthoremailurllicense spacy_version)widthvectorskeysr+rrparent_packagerFr)r)rload_model_from_pathrupdaterrr __version__vocabvectors_lengthrbrn_keysr+r setdefaultsplit_requirementrrextendrw)rrr/rrreq existing_reqsreqss r7rfrf|sv  D  # #D$4 5CKK 889J9JKDKK ))syy(() !!(( !!&& DO  '!!& OONB'?CN?STT++C03TMT ' M JD%%//n1EE ^##EOOd?6K$KL K Us2F?rc |xsi}dd|jddfdd|jddfdd|jdd fd d |jd dfd d |jd dfdd|jddfdd|jddfdd|jddfg}|jd|jd|D] \}}}t||}|dk(r|r|n|||<"|S)NrHzPipeline languagerr+z Pipeline namerr,zPackage versionrrzPackage descriptionrAuthorrz Author emailrzAuthor websiterLicenserzGenerating meta.jsonzEnter the package settings for your pipeline. The following information will be read from your pipeline data: pipeline, vectors.r")rdividertextr)rrrsettingssettingdescdefaultresponses r7riris   BD $dhhvt&<= $((6:">? %txx 7'CD -txx t/LM 8TXXh56 .$((7D"9:  $((5$"78 Itxx 59: HKK&'HH C#+Lw w/#+r>g8W L Krrct}|d}|d|d}|d}dj|jdgDcgc]}|j|c}}dj|jdgDcgc]}|j|c}}|jdi}|jd d d |jdd d |jd d d} |jdxsd} |jdd} |jd} t |jd} |jd}t t tttf|jd}tt tttf|jd}|jd|j|f|jd|j|f|jd|j|df|jd|f|jd|f|jd| f|jd| xsdf|jd | r|j| ndf|jd!d"|vr|j| |d"n| fg }|r|j||j|j|d#d$g|r2|j|jd%d&|j||r2|j|jd%d'|j|| r|j| |j Scc}wcc}w)(z Generate a Markdown-formatted README text from a model meta.json. Used within the GitHub release notes and as content for README.md file added to model packages. rHrIr+r,r@rrrrrz keys, z unique vectors (rz dimensions)rn/anotesr"rsourcesrlabels performanceNameVersionspaCyrzDefault Pipeline ComponentsVectorsSourcesrrrFeature Descriptionz Label SchemeAccuracy)rr^rcode_format_sources_format_label_schemer rrpr_format_accuracyboldlinkrtabletitler)rmdrHr+r,r6rrvecsrrr license_namerr label_schemeaccuracy table_datas r7rurus  B *+ | rxx:&' x  u 77NEHKs M6<M;datacj|rt|tsyg}|D]}t|tsd|i}|jd}|s+|jd}|jd}|s|ndj ||}|r|dj |z }|j |dj |S)Nrr+rrz[{}]({})z ({})z
) isinstancelistdictrr|rwr^)rrsourcer+rrresults r7rrs z$-G &$'f%Fzz&! jjH% j&7&7c&B  gnnV, ,Fv  ;;w rspeedc |syt}|jDcgc] \}}t|ttfs||f"}}}|Dcgc]/\}}||vr&|j |j |dzdf1}}}|j|j|ddg|jScc}}wcc}}w)Nr"dz.2fTypeScore) rrrintfloatrupperrrr) rrrkvscalarsaccscorescoress r7rrs   B"&**,N$!Q*Qe 2M1vNGN" C g   %#+c!24F FF288FVW- ./ 77NOsB4B44B:c f|syt}d}d}g}|jD]\}}|s |j|j|}dj |Dcgc],}|jt |j dd.c}} |j|| f|t|z }|dz }|syd|d|d } |jd |jd | d |j|j|d dg|jd|jScc}w)Nr"rr@|z\|rzView label scheme (z labels for z components)z
z z ComponentLabelsz
) rrrrr^rpreplacerwrbrrr) rrn_labelsn_pipes label_datapipercol1labelcol2 label_infos r7rrs$   BHGJ  f wwrwwt}%yyvVe"''#e*"4"4S%"@AVW4,'CK1  &xj WI\RJFF;FFYzl* -.FF288Jh 7 89FF< 77NWs1D. package_namecTtjd|tj}|duS)Nz)^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$)rsearch IGNORECASE)r>permitted_matchs r7rhrh!s(ii4lBMMO $ &&ra #!/usr/bin/env python import io import json from os import path, walk from shutil import copy from setuptools import setup def load_meta(fp): with io.open(fp, encoding='utf8') as f: return json.load(f) def load_readme(fp): if path.exists(fp): with io.open(fp, encoding='utf8') as f: return f.read() return "" def list_files(data_dir): output = [] for root, _, filenames in walk(data_dir): for filename in filenames: if not filename.startswith('.'): output.append(path.join(root, filename)) output = [path.relpath(p, path.dirname(data_dir)) for p in output] output.append('meta.json') return output def list_requirements(meta): # Up to version 3.7, we included the parent package # in requirements by default. This behaviour is removed # in 3.8, with a setting to include the parent package in # the requirements list in the meta if desired. requirements = [] if 'setup_requires' in meta: requirements += meta['setup_requires'] if 'requirements' in meta: requirements += meta['requirements'] return requirements def setup_package(): root = path.abspath(path.dirname(__file__)) meta_path = path.join(root, 'meta.json') meta = load_meta(meta_path) readme_path = path.join(root, 'README.md') readme = load_readme(readme_path) model_name = str(meta['lang'] + '_' + meta['name']) model_dir = path.join(model_name, model_name + '-' + meta['version']) copy(meta_path, path.join(model_name)) copy(meta_path, model_dir) setup( name=model_name, description=meta.get('description'), long_description=readme, author=meta.get('author'), author_email=meta.get('email'), url=meta.get('url'), version=meta['version'], license=meta.get('license'), packages=[model_name], package_data={model_name: list_files(model_dir)}, install_requires=list_requirements(meta), zip_safe=False, entry_points={'spacy_models': ['{m} = {m}'.format(m=model_name)]} ) if __name__ == '__main__': setup_package() zN include meta.json include LICENSE include LICENSES_SOURCES include README.md z from pathlib import Path from spacy.util import load_model_from_init_py, get_model_meta {imports} __version__ = get_model_meta(Path(__file__).parent)['version'] def load(**overrides): return load_model_from_init_py(__file__, **overrides) )LICENSELICENSES_SOURCESrL)F)Arrrnr~rk collectionsrpathlibrtypingrrrrr r r rd cataloguer thinc.apir wasabirrrr"rrcompatrschemasrr_utilrrrrrrcommandrpboolr8rrYrZSimpleFrozenListrr4rvrfrirurrrrhlstripryr5rzr{rsrBrr7rRsO  #@@@ #;;'/LL Y 0 +DE  }  !$  ! +Q h#B! k4&H M     B  .  J [TT  TT~T./T4 3-5T:c];T@ ATL MTZ[TTt!%! k k k ~k T k 3- k c] k k k k k  k  k  k \44*?)>)>)@9 9!#Y9 #Y9x 4tCy 4U4:-> 4:4:3:4:!"c4i "S>"" #s(^ "Jc3hg$sCx..,$sCx.,S,^ # # &BI 4S> DI c tCH~#2'S'T'LX FHY^  EG   FH>r