j TUddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZddlZddlmZmZmZmZmZddlmZddl m!Z!dd l"m#Z#dd l$m%Z%m&Z&m'Z'm(Z(m)Z)ed Z*ed Z+ed dZ,ediZ-ee.e/d<d$dZ0Gdde e*e+fZ1ejdde de de3de1de4f dZ5ejdde de de3de1de f dZ6ede1Z7de7de e3e e3e ffde7fdZ8dgfde7de9de7fd Z:de1d!e e1ge7fde7fd"Z;gd#Zee<ddfd?Z>d;e;de!e_refsr<r=r?N)r9dimsparamslayersshimsattrsrefsr6forwardrArBrCrDrErFcz||_|tt|}t|d|t|d|| | n t |_t |_t||_ t||_ t| |_ t||_ t||_tj 5txj"dz c_tj"|_dddi|_|j)D])\}} d|j&|<| |j+|| +y#1swYMxYw)zInitialize a new model.Nr8r9r)r5rr0setattrrr6rr:dictr;r>r@listr<r=r$r4r3r7r?items set_param) selfr5rGr9rArBrCrDrErFr6values r/__init__zModel.__init__Us <:t,Dgw'fd#/3/@"} $Z 5k $Z F| 5k  ! ! & OOq OooDG &!<<> ,KD%%)D  T " tU+ ,  & &s 6/D11D:r)c|jS)zmA list of child layers of the model. You can append to it to add layers but not reassign it. )r<rNs r/rCz Model.layers|s ||r1c|jSr+)r=rRs r/rDz Model.shimss {{r1c|jS)zfA dict of the model's attrs. You can write to it to update attrs but not reassign it. )r>rRs r/rEz Model.attrss {{r1.cHt|jjS)z8Get the names of registered parameter (including unset).)tupler?keysrRs r/ param_nameszModel.param_namessT%%**,--r1cvt|jDcgc]}|j|s|c}Scc}w)zHGet the names of parameters with registered gradients (including unset).)rVrXhas_gradrNr5s r/ grad_nameszModel.grad_namess.t'7'7Ot4==;NdOPPOs66cHt|jjS)z9Get the names of registered dimensions (including unset).)rVr;rWrRs r/ dim_nameszModel.dim_namesTZZ__&''r1cHt|jjS)z>Get the names of registered node references (including unset).)rVr@rWrRs r/ ref_nameszModel.ref_namesr_r1 operatorsc#K|jjt|}d|jj|yw)aBind arbitrary binary functions to Python operators, for use in any `Model` instance. Can (and should) be used as a contextmanager. EXAMPLE: with Model.define_operators({">>": chain}): model = Relu(512) >> Relu(512) >> Softmax() N)_context_operatorssetrJreset)clsrbtokens r/define_operatorszModel.define_operatorss9&&**4 ?;  $$U+sAAcB||jvry|j|yy)zCheck whether the model has a dimension of a given name. If the dimension is registered but the value is unset, returns None. FNT)r;r[s r/has_dimz Model.has_dim( tzz ! ZZ  )r1c||jvrtd|d|jd|j|}|d|d|jd}t||S)z4Retrieve the value of a dimension of the given name.zCannot get dimension ' ' for model ''z': value unset)r;KeyErrorr5 ValueErrorrNr5rOerrs r/get_dimz Model.get_dimsh tzz !3D6tyykQRST T 4  =*4& dii[WCS/ !Lr1F)forcerOrucb||jvrtd|d|jd|j|}td|jj D}|duxr||k7xr | xs|xr|}|r#d|d|jd|d|}t |||j|<y) zSet a value for a dimension.zCannot set unknown dimension 'rn'.c38K|]\}}t|ywr+)bool).0xys r/ z Model.set_dim..sFTQaFsNzAttempt to change dimension 'z' from z to )r;rpr5anyr?rLrq)rNr5rOru old_value has_paramsinvalid_changerss r/set_dimz Model.set_dims tzz !0mDII;bQ JJt$ FT-=-=-C-C-EFF #4/FI4F I --:  1$}TYYKwW`VaaefkelmCS/ !  4r1cJ|j|r|j|SdS)z=Retrieve the value of a dimension of the given name, or None.N)rkrtr[s r/ maybe_get_dimzModel.maybe_get_dim!%)\\$%7t||D!ATAr1cB||jvry|j|yy)zCheck whether the model has a weights parameter of the given name. Returns None if the parameter is registered but currently unset. FNT)r?r[s r/ has_paramzModel.has_params, t'' '   d # /r1c&||jvrtd|d|jd|jj |j |std|d|jd|jj |j |S)z%Retrieve a weights parameter by name.zUnknown param: 'rnrwz Parameter 'z' has not been allocated yet.)r?rpr5r:rr7 get_paramr[s r/rzModel.get_params t'' '-dV= 2NO O||%%dggt4dV= ;XY ||%%dggt44r1cJ|j|r|j|SdS)z.Retrieve a weights parameter by name, or None.N)rrr[s r/maybe_get_paramzModel.maybe_get_params!'+~~d';t~~d#EEr1c|d|j|<y|jj|j||d|j|<y)z Set a weights parameter's value.NT)r?r:rMr7rNr5rOs r/rMzModel.set_paramsA =%)D  T " LL " "477D% 8%)D  T "r1cN|jj|j|S)z@Check whether the model has a non-zero gradient for a parameter.)r:rZr7r[s r/rZzModel.has_grad||$$TWWd33r1cN|jj|j|S)zGet a gradient from the model.)r:get_gradr7r[s r/rzModel.get_gradrr1cR|jj|j||y)z#Set a gradient value for the model.N)r:set_gradr7rs r/rzModel.set_grad dggtU3r1cJ|j|r|j|SdS)z%Retrieve a gradient by name, or None.N)rZrr[s r/maybe_get_gradzModel.maybe_get_grads!&*mmD&9t}}T"CtCr1cR|jj|j||y)z1Increment the gradient of a parameter by a value.N)r:inc_gradr7rs r/rzModel.inc_grad rr1cB||jvry|j|yy)zCheck whether the model has a reference of a given name. If the reference is registered but the value is unset, returns None. FNT)r@r[s r/has_refz Model.has_refrlr1c||jvrtd|d|jd|j|}|d|d|jd}t||S)z4Retrieve the value of a reference of the given name.zCannot get reference 'rnrwz': value unset.)r@rpr5rqrrs r/get_refz Model.get_refsh tzz !3D6tyykQSTU U 4  =*4& dii[XCS/ !Lr1cJ|j|r|j|SdS)z8Retrieve the value of a reference if it exists, or None.N)rrr[s r/ maybe_get_refzModel.maybe_get_ref&rr1c|||j|<y||jvr||j|<ytd)zSet a value for a reference.Nz)Cannot add reference to node not in tree.)r@walkrqrs r/set_refz Model.set_ref*s= =$DJJt  diik !$DJJt HI Ir1Xis_trainc*|j|||S)z~Call the model's `forward` function, returning the output and a callback to compute the gradients via backpropagation.rr8)rNrrs r/__call__zModel.__call__3szz$Hz55r1Yctjr"t|j|j|||j |j ||||S)zFinish initialization of the model, optionally providing a batch of example input and output data to perform shape inference.)rr)rgetr r5r8r9)rNrrs r/ initializezModel.initialize8sG    %diiQ B 99 IIda1I % r1c*|j||dS)apRun the model over a batch of data, returning the output and a callback to complete the backward pass. A tuple (Y, finish_update), where Y is a batch of output data, and finish_update is a callback that takes the gradient with respect to the output and an optimizer function, and returns the gradient with respect to the input. TrrrNrs r/ begin_updatezModel.begin_updateAszz$Dz11r1c0|j||ddS)zCall the model's `forward` function with `is_train=False`, and return only the output, instead of the `(output, callback)` tuple. Frrrrs r/predictz Model.predictJszz$Ez2155r1 optimizerc t|jD]$}|jD]}|j|&|jD]n}|jD]]}|j |s||j |f|j ||j|\}}|j||_py)zUpdate parameters with current gradients. The optimizer is called with each parameter and gradient of the model. N) rrD finish_updaterXrZr7rrrM)rNrnodeshimr5paramgrads r/rzModel.finish_updatePsIIK .D  .""9- . .IIK 0D(( 0==&"+$)=t}}T?R#KE4NN4/  0 0r1c#"Ki}|jD]>}|j|f}||vs|j|||<|j|||@t j 5}|j D]"}|j|j|$|jD]"}|j|j|$dddd|r+|jD]\}}|j||yy#1swY7xYww)zContext manager to temporarily set the model's parameters to specified values. The params are a dictionary keyed by model IDs, whose values are arrays of weight values. N) rXr7rrM contextlib ExitStackrC enter_context use_paramsrDrL) rNrBbackupr5keystacklayerrrs r/rzModel.use_params_s $$ 2D77D/Cf}#~~d3t tVC[1  2  ! ! # u >##E$4$4V$<= >  =##DOOF$;< =    %||~ , etU+ ,    s#$D?D&A'D 6DD Dbfsorderrc|dk(r|jS|dk(r|jdS|dk(r|jdStd)zIterate out layers of the model. Nodes are returned in breadth-first order by default. Other possible orders are "dfs_pre" (depth-first search in preorder) and "dfs_post" (depth-first search in postorder).rdfs_preF) post_orderdfs_postTz5Invalid order, must be one of: bfs, dfs_pre, dfs_post) _walk_bfs _walk_dfsrq)rNrs r/rz Model.walkvsV E>>># # i >>U>3 3 j >>T>2 2TU Ur1c#K|g}t}|D]I}t||vr|jt|||j|jKyw)z/Iterate out layers of the model, breadth-first.N)rer7addextendrC)rNqueueseenrs r/rzModel._walk_bfssW &D$x4 HHRX J LL %  &sAArc#Kt}|g}t|j|t|<|s||rd t |t|d}t||vr8|s||j |t|j|t|<|rcyy#t $r|r|d|jY)wxYww)z-Iterate out layers of the model, depth-first.N)rJiterrCr7nextappend StopIterationpop)rNrrr next_childs r/rzModel._walk_dfss-1Vdkk*RXJ !$r%)}"56 *~-%((LL,+/ 0A0A+BDJ(! )O  s/7CABCC"CCCCcjt|jD];}||jvs|jj|||jvr*=t |j}|D]>}|j D]-}|j |}|||vs|j|d/@y)a]Remove a node from all layers lists, and then update references. References that no longer point to a node within the tree will be set to `None`. For instance, let's say a node has its grandchild as a reference. If the child is removed, the grandchild reference will be left dangling, so will be set to None. N)rKrrCremovererarr)rNrchildtreer5refs r/ remove_nodezModel.remove_nodes$))+& *E%,,& ##D)%,,& *499; -D -ll4(?s$LLt, - -r1)r9c8t|d|t|d|y)Nr8r9)rI)rNrGr9s r/replace_callbackszModel.replace_callbackss gw'fd#r1oldnewcd}t|jdD]e}||urd} |jDcgc] }||ur|n| c}|_|jD](}|j ||us|j ||*g|Scc}w)zzReplace a node anywhere it occurs within the model. Returns a boolean indicating whether the replacement was made.FrrT)rKrr<rarr)rNrrrrrr5s r/ replace_nodezModel.replace_nodes 45 0Ds{AE  7<5Cr\rrr#) rNrrBr5 copied_layersr copied_layer copied_shimsr copied_shimcopieds r/rz Model._copys <D$$ RD37>>$3G4>>$/TF4L R&( [[ 3E%yD $$T%bi%AB${{40 ".RY$$\2  3 JJ 1D$x4##DtBtH~$>?"iik !,RX##K0  1$) II JJ==(tzz*-- ,  $ OO >D OOD$--"5":":"< = >E6""r1gpu_idcddl}|jjj|5|j t dddy#1swYyxYw)z)Transfer the model to a given GPU device.rN)cupy.cuda.devicecudadeviceDevice_to_opsr)rNrcupys r/to_gpuz Model.to_gpus@ YY   $ $V , $ LL # $ $ $s A  Ac6|jty)zTransfer the model to CPU.N)rrrRs r/to_cpuz Model.to_cpus XZ r1c |jD]}||_|jD]}|j|r0|j ||j |j ||j|sV|j||j |j||jD](}|j|j|j*y)z Common method for to_cpu/to_gpu.N)rr6rXrrM asarray_frrZrrrD to_device device_type device_id)rNr6rr5rs r/rz Model._to_opssIIK ?DDH(( L>>$'NN4t~~d7K)LM==&MM$ dmmD6I(JK  L   ?s > ? ?r1c|j}t|jjd}t t ||}t j|S)aaSerialize the model to a bytes representation. Models are usually serialized using msgpack, so you should be able to call msgpack.loads() on the data and get back a dictionary with the contents. Serialization should round-trip identically, i.e. the same bytes should result from loading and serializing a model. <) byte_order)to_dictrr6to_numpyrrsrsly msgpack_dumps)rNmsg to_numpy_les r/to_byteszModel.to_bytes&sDllndhh//C@  [#>""3''r1pathct|tr t|n|}|jd5}|j |j dddy#1swYyxYw)zSerialize the model to disk. Most models will serialize to a single file, which should just be the bytes contents of model.to_bytes(). wbN) isinstancestrropenwriter )rNr file_s r/to_diskz Model.to_disk3sL(c2tDz YYt_ ) KK  ( ) ) )s AA!cggggd}t|j}t|Dcic]\}}|j|}}}t|D]\}}i}g}|jD]\}|j |sd||<|j |} | j|vr|| j||<L|j|^|rtd|i} |jD])} |j| r|j| nd| | <+|dj||j| |d|D]L}i} |jjD]\}} t| | ||| |<|dj| N|D]<}|dj|j"Dcgc]}|j%c}>|D]i}i}|j&D]B}|j)|r*t+t,t.|j1|||<>d||<D|dj|k|Scc}}w#t $rYwxYwcc}w) zSerialize the model to a dict representation. Serialization should round-trip identically, i.e. the same dict should result from loading and serializing a model. )nodesrErBrDNzCannot get references: r)indexr5rArFrErDrB)rKr enumerater7rarrrrqr^rkrtr5rErLserialize_attr TypeErrorrDr rXrrr rr)rNrrir node_to_irF invalid_refsr5rrAdimrErOrrBs r/rz Model.to_dict;st*,bBQSTTYY[! 09/?@GAtTWWaZ@ @ ' GAt-/D&(L 2||D)!%DJ,,t,Cvv*%.svv%6T $++D1 2 #:<.!IJJD~~ M15c1BDLL-S  M L  TYYdK # ( 'DE#zz//1  e"0tT"JE$K  L   & ' JD L  TZZ HT H I J )D46F(( (>>$'#'(:DNN4>r1cd|jjvr td|jjd||S)z-Apply the function bound to the '-' operator.-zUndefined operator: -r@rAs r/__sub__z Model.__sub__rCr1cd|jjvr td|jjd||S)z-Apply the function bound to the '*' operator.*zUndefined operator: *r@rAs r/__mul__z Model.__mul__rCr1cd|jjvr td|jjd||S)z-Apply the function bound to the '@' operator.@zUndefined operator: @r@rAs r/ __matmul__zModel.__matmul__rCr1cd|jjvr td|jjd||Sz-Apply the function bound to the '/' operator./zUndefined operator: /r@rAs r/__div__z Model.__div__ rCr1cd|jjvr td|jjd||SrNr@rAs r/ __truediv__zModel.__truediv__rCr1cd|jjvr td|jjd||S)z.Apply the function bound to the '//' operator.z//zUndefined operator: //r@rAs r/ __floordiv__zModel.__floordiv__H t..224 445 52t&&**,T24??r1cd|jjvr td|jjd||S)z-Apply the function bound to the '%' operator.%zUndefined operator: %r@rAs r/__mod__z Model.__mod__rCr1c d|jjvr td|jjd||S)z.Apply the function bound to the '**' operator.z**zUndefined operator: **r@)rNr=r.s r/__pow__z Model.__pow__"rUr1cd|jjvr td|jjd||S)z.Apply the function bound to the '<<' operator.z<>' operator.z>>zUndefined operator: >>r@rAs r/ __rshift__zModel.__rshift__.rUr1cd|jjvr td|jjd||S)z-Apply the function bound to the '&' operator.&zUndefined operator: &r@rAs r/__and__z Model.__and__4rCr1cd|jjvr td|jjd||S)z-Apply the function bound to the '^' operator.^zUndefined operator: ^r@rAs r/__xor__z Model.__xor__:rCr1cd|jjvr td|jjd||S)z-Apply the function bound to the '|' operator.|zUndefined operator: |r@rAs r/__or__z Model.__or__@rCr1)NN)F)rr$r)Nr+)r)N)k__name__ __module__ __qualname____doc__r3int__annotations__ threadingLockr4r&rdrrrrrr r rrry __slots__rr rrrrPpropertyrCrDrErrXr\r^ra classmethodrcontextmanagerrirkrtrrrrrrMrZrrrrrrrrr!r"rrrrrrrr rrrrrrrr#rrrrrbytesr rrrr"r&r!r7r6r9rBrFrIrLrPrRrTrXrZr\r^rardrgr,r1r/r$r$1s/9Is%3Y^^%5NINN5* I H G O N  Xc]" ## '] J cNc8D>)** I($()+02$& "-/26%,%,%, x %, 3 %& %,S(8,,-%,!%,Dz%,CH~%,3))*%,eHg-. /%,NW  tDztCH~ .U38_..QE#s(OQQ(5c?(((5c?(( ,c8m)< , , C HTN  C C ?D!C!!t!! B#B(3-B c htn 5c5h5FCFHX,>F*c*(8*<**4S4T44S4X44S44d4D3D8H+=D4S44d4 C HTN  C G B#B(7*;BJCJ(9JdJ6#66%h2G6 HSMXd^w2c2eD(D63;2G,G&H2666 0y 0T 0,eCHox&?!@,,,$) VS VXg-> V &8G, &DXg5F,-$@D$$*28*<$ $ g$* tE#s(OU8X;M5N$NO 5UHL'#'##DeGTM.B)B$CD'# '#R$S$T$! ?3 ?4 ? (% ()E$),))77r #U #w # +eD#I. +7 +Tg<GK >%c "2 >t >t >CG 6 64 64 6:>--$-$-^?S?W? ?S?W? ?S?W? ??? ?S?W? ??? @#@'@ ?S?W? @S@w@ @@@ @@@ ?S?W? ?S?W? ?C?G?r1_rOr5c,tj|S)zSerialize an attribute value (defaults to msgpack). You can register custom serializers using the @serialize_attr.register decorator with the type to serialize, e.g.: @serialize_attr.register(MyCustomObject). )rrrurOr5r(s r/rrG   u %%r1c,tj|S)zDeserialize an attribute value (defaults to msgpack). You can register custom deserializers using the @deserialize_attr.register decorator with the type to deserialize, e.g.: @deserialize_attr.register(MyCustomObject). )rrrws r/r)r)Prxr1_ModelTmappingc|jD]V}|j|vs||j}|jD]#\}}||jvs||j|<%X|S)zWalk over the model's nodes, changing the value of attributes using the provided mapping, which maps node names to attr names to attr values. )rr5rLrE)r(r{rrEr,rOs r/change_attr_valuesr}\sm - 99 DII&E${{} - e4::%',DJJt$ -- Lr1 dropout_ratedropcz|jD]'}|D] }||jvs||j|<")|S)zWalk over the model's nodes, setting the dropout rate. You can specify one or more attribute names, by default it looks for ["dropout_rate"]. )rrE)r(rrErr,s r/set_dropout_raterisI ( (Dtzz!#' 4  (( Lr1wrapperc~t|jD]}|j|||||S)zORecursively wrap a model and its submodules. The model is updated in-place.)rKrr)r(rrs r/wrap_model_recursiverts>UZZ\"0 4/0 5>r1)r$rr)r}rr)r(r$r)r$)=rr functoolsrn contextvarsrpathlibrtypingrrrrr r r r r rrrrrrbackendsrrrrr optimizersrrDrtypesrutilrrrrr r!r"r#r&rJrmr0r$singledispatchrrtrr)rzr}floatrr__all__r,r1r/rs "" JJ!envw'&01Db&Q:d#QS ?GCI S ?l &c&#&S&&5&& &&C&s&5&S&& )5 ) g S$sCx.5H0I g :H8HG5W%'9I0Jw r1