暾jhdZddlmZddlZddlZddlZddlZddlm Z m Z ddl Z e rddl mZmZddlmZej$eZdZdZGd d e Zdd Zdd Zdd ZGddej6ZGddej6Zy)zdImplements reading and writing to/from HDFS via the Hadoop ``hdfs`` CLI (must be on your ``$PATH``).) annotationsN) TYPE_CHECKING TypedDict)ReadableBufferWriteableBuffer)TransportParams)hdfsviewfs)zhdfs:///path/filezhdfs://host/path/filezhdfs://host:port/path/filezviewfs:///path/filezviewfs://host/path/filec"eZdZUded<ded<y)_HDFSUristrschemeuri_pathN)__name__ __module__ __qualname____annotations__Y/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/smart_open/hdfs.pyr r &s KMrr ctjj|}|jtvsJ|j r|n |j }|r|dk(rd|}t||j|dS)zBParse an ``hdfs://`` or ``viewfs://`` URI into its path component./zinvalid HDFS URI: )rr)urllibparseurlsplitrSCHEMESnetlocpath RuntimeError) uri_as_string split_urirmsgs r parse_urir#+sq %%m4I   w && & !* 0 0}innH x3"="343&&H ==rctjjt|t |}t|d|}|dj dd|_|S)z;Open an HDFS URI using the given mode and transport params.rr) smart_openutils check_kwargsopenr#splitname)urimodetransport_params parsed_urifobjs ropen_urir1;sT!!$(893J  :& -D:&,,S1"5DI Krcd|dk(r t|S|dk(r t|Sd|d}t|)aIOpen an HDFS `uri` for reading or writing. Args: uri: The HDFS path to open. mode: The mode for opening the object. Must be either "rb" or "wb". Returns: A file-like object for reading from or writing to the HDFS file. Raises: NotImplementedError: If ``mode`` is not supported. rbwbzhdfs support for mode z not implemented)CliRawInputBaseCliRawOutputBaseNotImplementedError)r,r-r"s rr)r)EsB t|s## t|$$ "4(*: ;C c ""rceZdZUdZded<dZded<ddZddZedd Z dd Z dd Z dd Z ddd Z dddZddZy)r5zReads bytes from HDFS via the "hdfs dfs" command-line interface. Implements the io.RawIOBase interface of the standard library. r r+Nsubprocess.Popen[bytes] | None_subc||_tjddd|jgtj|_y)Nr dfsz-cat)stdout_uri subprocessPopenPIPEr:selfr,s r__init__zCliRawInputBase.__init__cs1 $$feVTYY%GPZP_P_` rcztjd|j}||jd|_yyzFlush and close this stream.z close: calledN)loggerdebugr: terminaterDsubs rclosezCliRawInputBase.closejs1 _%ii ? MMODI rc|jduSz$Return True if the stream is closed.Nr:rDs rclosedzCliRawInputBase.closedryyD  rc|jduS)z+Return True if the stream can be read from.NrPrQs rreadablezCliRawInputBase.readablewyy$$rcyz2Return False; HDFS streams do not support seeking.FrrQs rseekablezCliRawInputBase.seekable{rc"tj) Unsupported.ioUnsupportedOperationrQs rdetachzCliRawInputBase.detachs%%%rc||j}|J|jJ|d}|jj|S)z6Read up to size bytes from the object and return them.r%)r:r=read)rDsizerLs rrbzCliRawInputBase.readsBiizz%%% <Dzzt$$rc&|j|S)zThis is the same as read().)rc)rb)rDrcs rread1zCliRawInputBase.read1syydy##rct|jd}|jt|}|sy||dt|t|S)zIRead up to ``len(b)`` bytes into `b` and return the number of bytes read.BrN) memoryviewcastrblen)rDbmvdatas rreadintozCliRawInputBase.readintosG ]   $yyR!;SY4yrr,r returnNonerprqrpboolrpz io.RawIOBase)r%)rcz int | Nonerpbytes)rkrrpint)rrr__doc__rr:rErMpropertyrRrUrYr`rbrernrrrr5r5ZsV I+/D (/a!!%&%$rr5cteZdZUdZded<dZded<ddZddZedd Z dd Z dd Z dd Z dd Z ddZy)r6zWrites bytes to HDFS via the "hdfs dfs" command-line interface. Implements the io.RawIOBase interface of the standard library. r r+Nr9r:c||_tjddddd|jgtj|_y)Nr r<z-putz-f-)stdinr>rCs rrEzCliRawOutputBase.__init__s5 $$feVT3 %RZdZiZij rctjd|j}|P|jJ|j |jj |j d|_yyrG)rHrIr:r}flushrMwaitrKs rrMzCliRawOutputBase.closesX _%ii ?99( (( JJL IIOO  HHJDI rc|jduSrOrPrQs rrRzCliRawOutputBase.closedrSrct|j}|J|jJ|jjy)z8Flush the underlying ``hdfs dfs -put`` subprocess stdin.N)r:r}rrKs rrzCliRawOutputBase.flushs3iiyy$$$ rc|jduS)z(Return True if this object is writeable.NrPrQs r writeablezCliRawOutputBase.writeablerVrcyrXrrQs rrYzCliRawOutputBase.seekablerZrct|j}|J|jJ|jj|S)aWrite the given buffer to the underlying raw stream. Returns the number of bytes written, as required by :class:`io.RawIOBase`. Without this return value, callers that wrap this stream and rely on the documented ``write`` contract (for example, ``ray._private.external_storage._write_multiple_objects``, which asserts ``written_bytes == payload_len``) fail with an ``AssertionError`` because ``write`` would otherwise implicitly return ``None``. )r:r}write)rDrkrLs rrzCliRawOutputBase.writes:iiyy$$$yyq!!rc0d}tj|)r\zdetach() not supportedr])rDr"s rr`zCliRawOutputBase.detachs&%%c**rrorrrs)rkrrprwru)rrrrxrr:rErMryrRrrrYrr`rrrr6r6sQ I+/D (/k !!%"&+rr6)r r rpr )r,r r-r r.rrp"CliRawInputBase | CliRawOutputBase)r,r r-r rpr)rx __future__rr^loggingr@ urllib.parsertypingrrsmart_open.utilsr& _typeshedrrsmart_open._typingr getLoggerrrHr URI_EXAMPLESr r#r1r) RawIOBaser5r6rrrrsk" +92   8 $  y > #*@bll@FB+r||B+r