recfunctions.pyc
1 o 2 [��c�� � @ s� d Z ddlZddlZddlmZ ddlmZmZ ddlmZ ddl m 3 Z 4 ddl m Z ddl mZ ejjjZg d�Zd d 5 � Ze e�d d � �Zd d� Zdd� Zdd� Zdd� Zdpdd�Zdpdd�Zdqdd�Zd d � Zd d � Z drd!d"�Z dsd#d$�Z dtd%d&�Z dud'd(�Z e e � dvd*d+��Z!dqd,d-�Z"e e"�dsd.d/��Z#dsd0d1�Z$d2d3� Z%e e%�d4d5� �Z&d6d7� Z'e e'�d8d9� �Z( dud:d;�Z)e e)� dwd<d=��Z*dtd>d?�Z+e e+�dtd@dA��Z,dqdBdC�Z-e e-�dxdDdE��Z.dydFdG�Z/ dzdHdI�Z0e e0�d{dKdL��Z1 d|dMdN�Z2e e2� Jd}dOdP��Z3dQdR� Z4e e4�dSdT� �Z5dtdUdV�Z6e e6�d~dWdX��Z7dYdZ� Z8e e8�d[d\� �Z9 dud]d^�Z:e e:� dd_d`��Z; dzdadb�Z<e e<�d�dcdd��Z= d�dedf�Z>e e>� i d�djdk��Z? dudldm�Z@e e@� i d�dndo��ZAdS )�z� 6 Collection of utilities to manipulate structured arrays. 7 8 Most of these functions were initially implemented by John Hunter for 9 matplotlib. They have been rewritten and extended for convenience. 10 11 � N)�ndarray�recarray)� MaskedArray)� MaskedRecords)�array_function_dispatch)�_is_string_like)� append_fields�apply_along_fields�assign_fields_by_name� drop_fields�find_duplicates� flatten_descr�get_fieldstructure� get_names�get_names_flat�join_by� merge_arrays�rec_append_fields�rec_drop_fields�rec_join�recursive_fill_fields� rename_fields� repack_fields�require_fields� stack_arrays�structured_to_unstructured�unstructured_to_structuredc C � | |fS �N� )�input�outputr r ��C:\Users\Jacks.GUTTSPC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\numpy\lib\recfunctions.py�!_recursive_fill_fields_dispatcher � r# c C sf |j }|jD ]*}z| | }W n ty Y qw |j jdur&t||| � q||| dt|��< q|S )aj 12 Fills fields from output with fields from input, 13 with support for nested structures. 14 15 Parameters 16 ---------- 17 input : ndarray 18 Input array. 19 output : ndarray 20 Output array. 21 22 Notes 23 ----- 24 * `output` should be at least the same size as `input` 25 26 Examples 27 -------- 28 >>> from numpy.lib import recfunctions as rfn 29 >>> a = np.array([(1, 10.), (2, 20.)], dtype=[('A', np.int64), ('B', np.float64)]) 30 >>> b = np.zeros((3,), dtype=a.dtype) 31 >>> rfn.recursive_fill_fields(a, b) 32 array([(1, 10.), (2, 20.), (0, 0.)], dtype=[('A', '<i8'), ('B', '<f8')]) 33 34 N)�dtype�names� 35 ValueErrorr �len)r r! �newdtype�field�currentr r r" r # s 36 � r c s6 � j du r 37 d� fgS � fdd�� j D �}dd� |D �S )aR 38 Produce a list of name/dtype pairs corresponding to the dtype fields 39 40 Similar to dtype.descr, but the second item of each tuple is a dtype, not a 41 string. As a result, this handles subarray dtypes 42 43 Can be passed to the dtype constructor to reconstruct the dtype, noting that 44 this (deliberately) discards field offsets. 45 46 Examples 47 -------- 48 >>> dt = np.dtype([(('a', 'A'), np.int64), ('b', np.double, 3)]) 49 >>> dt.descr 50 [(('a', 'A'), '<i8'), ('b', '<f8', (3,))] 51 >>> _get_fieldspec(dt) 52 [(('a', 'A'), dtype('int64')), ('b', dtype(('<f8', (3,))))] 53 54 N� c 3 s � | ] 55 }|� j | fV qd S r )�fields)�.0�name�r% r r" � <genexpr>a s � z!_get_fieldspec.<locals>.<genexpr>c S s4 g | ]\}}t |�d kr|n|d |f|d f�qS )� r �r( )r. r/ �fr r r" � 56 <listcomp>c s "��z"_get_fieldspec.<locals>.<listcomp>�r& )r% r- r r0 r" �_get_fieldspecJ s 57 58 �r7 c C sP g }| j }|D ] }| | }|j dur |�|tt|��f� q|�|� qt|�S )aG 59 Returns the field names of the input datatype as a tuple. Input datatype 60 must have fields otherwise error is raised. 61 62 Parameters 63 ---------- 64 adtype : dtype 65 Input datatype 66 67 Examples 68 -------- 69 >>> from numpy.lib import recfunctions as rfn 70 >>> rfn.get_names(np.empty((1,), dtype=[('A', int)]).dtype) 71 ('A',) 72 >>> rfn.get_names(np.empty((1,), dtype=[('A',int), ('B', float)]).dtype) 73 ('A', 'B') 74 >>> adtype = np.dtype([('a', int), ('b', [('ba', int), ('bb', int)])]) 75 >>> rfn.get_names(adtype) 76 ('a', ('b', ('ba', 'bb'))) 77 N)r& �append�tupler ��adtypeZ listnamesr& r/ r+ r r r" r i s 78 r c C sF g }| j }|D ]}|�|� | | }|j dur |�t|�� qt|�S )a� 79 Returns the field names of the input datatype as a tuple. Input datatype 80 must have fields otherwise error is raised. 81 Nested structure are flattened beforehand. 82 83 Parameters 84 ---------- 85 adtype : dtype 86 Input datatype 87 88 Examples 89 -------- 90 >>> from numpy.lib import recfunctions as rfn 91 >>> rfn.get_names_flat(np.empty((1,), dtype=[('A', int)]).dtype) is None 92 False 93 >>> rfn.get_names_flat(np.empty((1,), dtype=[('A',int), ('B', str)]).dtype) 94 ('A', 'B') 95 >>> adtype = np.dtype([('a', int), ('b', [('ba', int), ('bb', int)])]) 96 >>> rfn.get_names_flat(adtype) 97 ('a', 'b', 'ba', 'bb') 98 N)r& r8 �extendr r9 r: r r r" r � s 99 100 �r c C sd | j }|du r d| ffS g }|D ] }| j| \}}|j dur&|�t|�� q|�||f� qt|�S )aD 101 Flatten a structured data-type description. 102 103 Examples 104 -------- 105 >>> from numpy.lib import recfunctions as rfn 106 >>> ndtype = np.dtype([('a', '<i4'), ('b', [('ba', '<f8'), ('bb', '<i4')])]) 107 >>> rfn.flatten_descr(ndtype) 108 (('a', dtype('int32')), ('ba', dtype('float64')), ('bb', dtype('int32'))) 109 110 Nr, )r& r- r< r r8 r9 )�ndtyper& �descrr* �typ�_r r r" r � s 111 112 r Fc C st g }|r| D ] 113 }|� t|j�� qn#| D ] }|j}|jd ur-t|j�dkr-|� t|�� q|�d|f� qt�|�S )N� r, )r< r r% r& r( r7 r8 �np)� seqarrays�flattenr) �ar+ r r r" � 114 _zip_dtype� s � 115 rF c C s t | |d�jS )z� 116 Combine the dtype description of a series of arrays. 117 118 Parameters 119 ---------- 120 seqarrays : sequence of arrays 121 Sequence of arrays 122 flatten : {boolean}, optional 123 Whether to collapse nested descriptions. 124 �rD )rF r> )rC rD r r r" � 125 _zip_descr� s rH c C s� |du ri }| j }|D ]A}| | }|j dur,|r |g||< ng ||< |�t|||�� q dd� |�|g �p6g D �}|rA|�|� n|rF|g}|pIg ||< q |S )ab 126 Returns a dictionary with fields indexing lists of their parent fields. 127 128 This function is used to simplify access to fields nested in other fields. 129 130 Parameters 131 ---------- 132 adtype : np.dtype 133 Input datatype 134 lastname : optional 135 Last processed field name (used internally during recursion). 136 parents : dictionary 137 Dictionary of parent fields (used interbally during recursion). 138 139 Examples 140 -------- 141 >>> from numpy.lib import recfunctions as rfn 142 >>> ndtype = np.dtype([('A', int), 143 ... ('B', [('BA', int), 144 ... ('BB', [('BBA', int), ('BBB', int)])])]) 145 >>> rfn.get_fieldstructure(ndtype) 146 ... # XXX: possible regression, order of BBA and BBB is swapped 147 {'A': [], 'B': [], 'BA': ['B'], 'BB': ['B'], 'BBA': ['B', 'BB'], 'BBB': ['B', 'BB']} 148 149 Nc S s g | ]}|�qS r r )r. r@ r r r" r5 s z&get_fieldstructure.<locals>.<listcomp>)r& �updater �getr8 )r; �lastname�parentsr& r/ r+ Z 150 lastparentr r r" r � s" 151 r c c s6 � | D ]}t |tj�rtt|��E dH q|V qdS )zu 152 Returns an iterator of concatenated fields from a sequence of arrays, 153 collapsing any nested structure. 154 155 N)� 156 isinstancerB �void�_izip_fields_flatr9 ��iterable�elementr r r" rO s � �rO c c sf � | D ]-}t |d�rt|t�st|�E dH qt|tj�r-tt|��dkr-t|�E dH q|V qdS )zP 157 Returns an iterator of concatenated fields from a sequence of arrays. 158 159 �__iter__NrA )�hasattrrM �str� _izip_fieldsrB rN r( r9 rP r r r" rV s � 160 � �rV Tc c s8 � |rt }nt}tj| d|i�D ] }t||��V qdS )a* 161 Returns an iterator of concatenated items from a sequence of arrays. 162 163 Parameters 164 ---------- 165 seqarrays : sequence of arrays 166 Sequence of arrays. 167 fill_value : {None, integer} 168 Value used to pad shorter iterables. 169 flatten : {True, False}, 170 Whether to 171 � fillvalueN)rO rV � itertools� zip_longestr9 )rC � 172 fill_valuerD Zzipfunc�tupr r r" � _izip_records- s ��r\ c C s@ t | t�sd}|r|r| �t�} | S t�| �} |r | �t�} | S )z� 173 Private function: return a recarray, a ndarray, a MaskedArray 174 or a MaskedRecords depending on the input parameters 175 F)rM r �viewr �ma�filledr )r! �usemask� 176 asrecarrayr r r" � _fix_outputE s 177 178 179 � 180 rb c C sX | j j}| j| j| j}}}|pi �� D ]\}}||v r)|||< ||| || < q| S )zp 181 Update the fill_value and masked data of `output` 182 from the default given in a dictionary defaults. 183 )r% r& �data�maskrZ �items)r! �defaultsr& rc rd rZ �k�vr r r" � _fix_defaultsV s �ri c C � | S r r )rC rZ rD r` ra r r r" �_merge_arrays_dispatcherd � rk �����c C s� t | �dkr t�| d �} t| ttjf�rP| j}|jdu r%t�d|fg�}|r0t| fdd�|krL| � � } |r>|r;t 184 }n 185 t }n|rCt }nt}| j ||d�S | f} ndd � | D �} td 186 d � | D ��}t|�}t| |d�} g } 187 g } |r�t| |�D ]c\} } || }| � � �� }t�| �� � }|r�t|| j�}t|ttjf�r�t |j�dkr�|�� d }d}ntj|| jdd �}tjd |jd�}nd}d}| 188 �t�||g| �� | �t�||g| �� qutt| 189 |d��}tjtj || |d�t t| |d��d�}|r�|� t 190 �}|S t| |�D ]H\} } || }| � � �� }|�r;t|| j�}t|ttjf��r:t |j�dk�r1|�� d }n tj|| jdd �}nd}| 191 �t�||g| �� �qtj tt| 192 |d��| |d�}|�ra|� t �}|S )aR 193 Merge arrays field by field. 194 195 Parameters 196 ---------- 197 seqarrays : sequence of ndarrays 198 Sequence of arrays 199 fill_value : {float}, optional 200 Filling value used to pad missing data on the shorter arrays. 201 flatten : {False, True}, optional 202 Whether to collapse nested fields. 203 usemask : {False, True}, optional 204 Whether to return a masked array or not. 205 asrecarray : {False, True}, optional 206 Whether to return a recarray (MaskedRecords) or not. 207 208 Examples 209 -------- 210 >>> from numpy.lib import recfunctions as rfn 211 >>> rfn.merge_arrays((np.array([1, 2]), np.array([10., 20., 30.]))) 212 array([( 1, 10.), ( 2, 20.), (-1, 30.)], 213 dtype=[('f0', '<i8'), ('f1', '<f8')]) 214 215 >>> rfn.merge_arrays((np.array([1, 2], dtype=np.int64), 216 ... np.array([10., 20., 30.])), usemask=False) 217 array([(1, 10.0), (2, 20.0), (-1, 30.0)], 218 dtype=[('f0', '<i8'), ('f1', '<f8')]) 219 >>> rfn.merge_arrays((np.array([1, 2]).view([('a', np.int64)]), 220 ... np.array([10., 20., 30.])), 221 ... usemask=False, asrecarray=True) 222 rec.array([( 1, 10.), ( 2, 20.), (-1, 30.)], 223 dtype=[('a', '<i8'), ('f1', '<f8')]) 224 225 Notes 226 ----- 227 * Without a mask, the missing value will be filled with something, 228 depending on what its corresponding type: 229 230 * ``-1`` for integers 231 * ``-1.0`` for floating point numbers 232 * ``'-'`` for characters 233 * ``'-1'`` for strings 234 * ``True`` for boolean values 235 * XXX: I just obtained these values empirically 236 rA r Nr, TrG )r% �typec S s g | ]}t �|��qS r )rB � 237 asanyarray)r. �_mr r r" r5 � � z merge_arrays.<locals>.<listcomp>c s s � | ]}|j V qd S r )�size�r. rE r r r" r1 � s � z merge_arrays.<locals>.<genexpr>)r% �ndmin)rA r0 )r% �count)rd ) r( rB ro rM r rN r% r& rF �ravelr r r r] r9 �max�zip� __array__r^ � getmaskarray�_check_fill_value�item�array�onesr8 rX �chainr\ �fromiter�list)rC rZ rD r` ra ZseqdtypeZseqtype�sizes� maxlengthr) ZseqdataZseqmaskrE �nZ nbmissingrc rd �fvalZfmskr! r r r" r i s� 1 238 �� 239 � � � 240 r c C � | fS r r )�base� 241 drop_namesr` ra r r r" �_drop_fields_dispatcher� � r� c sX t |�r|g}nt|�}� fdd�� � | j|�}tj| j|d�}t| |�}t|||d�S )a 242 Return a new array with fields in `drop_names` dropped. 243 244 Nested fields are supported. 245 246 .. versionchanged:: 1.18.0 247 `drop_fields` returns an array with 0 fields if all fields are dropped, 248 rather than returning ``None`` as it did previously. 249 250 Parameters 251 ---------- 252 base : array 253 Input array 254 drop_names : string or sequence 255 String or sequence of strings corresponding to the names of the 256 fields to drop. 257 usemask : {False, True}, optional 258 Whether to return a masked array or not. 259 asrecarray : string or sequence, optional 260 Whether to return a recarray or a mrecarray (`asrecarray=True`) or 261 a plain ndarray or masked array with flexible dtype. The default 262 is False. 263 264 Examples 265 -------- 266 >>> from numpy.lib import recfunctions as rfn 267 >>> a = np.array([(1, (2, 3.0)), (4, (5, 6.0))], 268 ... dtype=[('a', np.int64), ('b', [('ba', np.double), ('bb', np.int64)])]) 269 >>> rfn.drop_fields(a, 'a') 270 array([((2., 3),), ((5., 6),)], 271 dtype=[('b', [('ba', '<f8'), ('bb', '<i8')])]) 272 >>> rfn.drop_fields(a, 'ba') 273 array([(1, (3,)), (4, (6,))], dtype=[('a', '<i8'), ('b', [('bb', '<i8')])]) 274 >>> rfn.drop_fields(a, ['ba', 'bb']) 275 array([(1,), (4,)], dtype=[('a', '<i8')]) 276 c s` | j }g }|D ]&}| | }||v rq|j d ur&� ||�}|r%|�||f� q|�||f� q|S r )r& r8 )r= r� r& r) r/ r+ r> �� _drop_descrr r" r� s 277 278 �z drop_fields.<locals>._drop_descrr0 �r` ra )r �setr% rB �empty�shaper rb )r� r� r` ra r) r! r r� r" r � s & 279 r c s: � fdd�|D �}t j� j|d�}t� |�}t|||d�S )a� 280 Return a new array keeping only the fields in `keep_names`, 281 and preserving the order of those fields. 282 283 Parameters 284 ---------- 285 base : array 286 Input array 287 keep_names : string or sequence 288 String or sequence of strings corresponding to the names of the 289 fields to keep. Order of the names will be preserved. 290 usemask : {False, True}, optional 291 Whether to return a masked array or not. 292 asrecarray : string or sequence, optional 293 Whether to return a recarray or a mrecarray (`asrecarray=True`) or 294 a plain ndarray or masked array with flexible dtype. The default 295 is False. 296 c s g | ] }|� j | f�qS r r0 �r. r� �r� r r" r5 I � z _keep_fields.<locals>.<listcomp>r0 r� )rB r� r� r rb )r� Z 297 keep_namesr` ra r) r! r r� r" � _keep_fields6 s 298 r� c C r� r r �r� r� r r r" �_rec_drop_fields_dispatcherO r� r� c C s t | |ddd�S )zK 299 Returns a new numpy.recarray with fields in `drop_names` dropped. 300 FTr� )r r� r r r" r S s r c C r� r r )r� � 301 namemapperr r r" �_rename_fields_dispatcher[ r� r� c s" � fdd�� � | j |�}| �|�S )a� 302 Rename the fields from a flexible-datatype ndarray or recarray. 303 304 Nested fields are supported. 305 306 Parameters 307 ---------- 308 base : ndarray 309 Input array whose fields must be modified. 310 namemapper : dictionary 311 Dictionary mapping old field names to their new version. 312 313 Examples 314 -------- 315 >>> from numpy.lib import recfunctions as rfn 316 >>> a = np.array([(1, (2, [3.0, 30.])), (4, (5, [6.0, 60.]))], 317 ... dtype=[('a', int),('b', [('ba', float), ('bb', (float, 2))])]) 318 >>> rfn.rename_fields(a, {'a':'A', 'bb':'BB'}) 319 array([(1, (2., [ 3., 30.])), (4, (5., [ 6., 60.]))], 320 dtype=[('A', '<i8'), ('b', [('ba', '<f8'), ('BB', '<f8', (2,))])]) 321 322 c sV g }| j D ]#}|�||�}| | }|j d ur!|�|� ||�f� q|�||f� q|S r )r& rJ r8 )r= r� r) r/ �newnamer+ ��_recursive_rename_fieldsr r" r� w s 323 324 �z/rename_fields.<locals>._recursive_rename_fields)r% r] )r� r� r) r r� r" r _ s 325 r c c � � | V |E d H d S r r )r� r& rc �dtypesrZ r` ra r r r" �_append_fields_dispatcher� s �r� c C sP t |ttf�rt|�t|�krd}t|��n t |t�r!|g}|g}|du r7dd� |D �}dd� t||�D �}n0t |ttf�sA|g}t|�t|�kr\t|�dkrV|t|� }nd}t|��dd� t|||�D �}t| ||d �} t|�dkr}t|d 326 ||d �}n|�� }t j 327 t t| �t|��t | j �t |j � d �}t| |�}t||�}t|||d �S )a 328 Add new fields to an existing array. 329 330 The names of the fields are given with the `names` arguments, 331 the corresponding values with the `data` arguments. 332 If a single field is appended, `names`, `data` and `dtypes` do not have 333 to be lists but just values. 334 335 Parameters 336 ---------- 337 base : array 338 Input array to extend. 339 names : string, sequence 340 String or sequence of strings corresponding to the names 341 of the new fields. 342 data : array or sequence of arrays 343 Array or sequence of arrays storing the fields to add to the base. 344 dtypes : sequence of datatypes, optional 345 Datatype or sequence of datatypes. 346 If None, the datatypes are estimated from the `data`. 347 fill_value : {float}, optional 348 Filling value used to pad missing data on the shorter arrays. 349 usemask : {False, True}, optional 350 Whether to return a masked array or not. 351 asrecarray : {False, True}, optional 352 Whether to return a recarray (MaskedRecords) or not. 353 354 z7The number of arrays does not match the number of namesNc S s g | ] 355 }t j|d dd��qS )FT)�copy�subok)rB r} rs r r r" r5 � s z!append_fields.<locals>.<listcomp>c S s" g | ] \}}|� ||jfg��qS r )r] r% )r. r/ rE r r r" r5 � s " rA z5The dtypes argument must be None, a dtype, or a list.c S s0 g | ]\}}}t j|d d|d��||fg��qS )FT)r� r� r% )rB r} r] )r. rE r� �dr r r" r5 � s "�)r` rZ T)rD r` rZ r0 r� )rM r9 r� r( r' rU rx r �popr^ � 356 masked_allrw r7 r% r rb ) r� r& rc r� rZ r` ra �msgr! r r r" r � sD � 357 358 � �� 359 360 r c c r� r r �r� r& rc r� r r r" � _rec_append_fields_dispatcher� s �r� c C s t | |||ddd�S )aM 361 Add new fields to an existing array. 362 363 The names of the fields are given with the `names` arguments, 364 the corresponding values with the `data` arguments. 365 If a single field is appended, `names`, `data` and `dtypes` do not have 366 to be lists but just values. 367 368 Parameters 369 ---------- 370 base : array 371 Input array to extend. 372 names : string, sequence 373 String or sequence of strings corresponding to the names 374 of the new fields. 375 data : array or sequence of arrays 376 Array or sequence of arrays storing the fields to add to the base. 377 dtypes : sequence of datatypes, optional 378 Datatype or sequence of datatypes. 379 If None, the datatypes are estimated from the `data`. 380 381 See Also 382 -------- 383 append_fields 384 385 Returns 386 ------- 387 appended_array : np.recarray 388 TF)rc r� ra r` )r r� r r r" r � s 389 �r c C r� r r )rE �align�recurser r r" �_repack_fields_dispatcher� r� r� c C s� t | tj�st| j||d�}| j|dd�S | jdu r | S g }| jD ]*}| j| }|r4t|d |dd�}n|d }t|�dkrD|d |f}|�||f� q!tj||d �}t�| j |f�S ) 390 a 391 Re-pack the fields of a structured array or dtype in memory. 392 393 The memory layout of structured datatypes allows fields at arbitrary 394 byte offsets. This means the fields can be separated by padding bytes, 395 their offsets can be non-monotonically increasing, and they can overlap. 396 397 This method removes any overlaps and reorders the fields in memory so they 398 have increasing byte offsets, and adds or removes padding bytes depending 399 on the `align` option, which behaves like the `align` option to 400 `numpy.dtype`. 401 402 If `align=False`, this method produces a "packed" memory layout in which 403 each field starts at the byte the previous field ended, and any padding 404 bytes are removed. 405 406 If `align=True`, this methods produces an "aligned" memory layout in which 407 each field's offset is a multiple of its alignment, and the total itemsize 408 is a multiple of the largest alignment, by adding padding bytes as needed. 409 410 Parameters 411 ---------- 412 a : ndarray or dtype 413 array or dtype for which to repack the fields. 414 align : boolean 415 If true, use an "aligned" memory layout, otherwise use a "packed" layout. 416 recurse : boolean 417 If True, also repack nested structures. 418 419 Returns 420 ------- 421 repacked : ndarray or dtype 422 Copy of `a` with fields repacked, or `a` itself if no repacking was 423 needed. 424 425 Examples 426 -------- 427 428 >>> from numpy.lib import recfunctions as rfn 429 >>> def print_offsets(d): 430 ... print("offsets:", [d.fields[name][1] for name in d.names]) 431 ... print("itemsize:", d.itemsize) 432 ... 433 >>> dt = np.dtype('u1, <i8, <f8', align=True) 434 >>> dt 435 dtype({'names': ['f0', 'f1', 'f2'], 'formats': ['u1', '<i8', '<f8'], 'offsets': [0, 8, 16], 'itemsize': 24}, align=True) 436 >>> print_offsets(dt) 437 offsets: [0, 8, 16] 438 itemsize: 24 439 >>> packed_dt = rfn.repack_fields(dt) 440 >>> packed_dt 441 dtype([('f0', 'u1'), ('f1', '<i8'), ('f2', '<f8')]) 442 >>> print_offsets(packed_dt) 443 offsets: [0, 1, 9] 444 itemsize: 17 445 446 )r� r� F)r� Nr T� r2 �r� ) 447 rM rB r% r �astyper& r- r( r8 rn )rE r� r� �dtZ fieldinfor/ r[ �fmtr r r" r s < 448 449 450 r c 451 s� dd� }g }| j D ]T}| j| }|d |d }}||�\}}|j du r5|�t�||ff�||| f� q t||| �} |j�t|�D ]� � dkrO|�| � qC|�� �fdd�| D �� qCq |S )z� 452 Returns a flat list of (dtype, count, offset) tuples of all the 453 scalar fields in the dtype "dt", including nested fields, in left 454 to right order. 455 c S s: d}| j dkr| j D ]}||9 }q 456 | j} | j dks| |fS )NrA r )r� r� )r� ru rr r r r" � 457 count_elem\ s 458 459 460 461 �z+_get_fields_and_offsets.<locals>.count_elemr rA Nc s$ g | ]\}}}|||� � f�qS r r )r. r� �c�o��irr r r" r5 u s $ z+_get_fields_and_offsets.<locals>.<listcomp>) r& r- r8 rB r% �_get_fields_and_offsets�itemsize�ranger< ) 462 r� �offsetr� r- r/ r* Zf_dtZf_offsetr� � subfieldsr r� r" r� S s 463 464 465 " �r� c C r� r r )�arrr% r� �castingr r r" �&_structured_to_unstructured_dispatchery � r� �unsafec s� | j jdu r 466 td��t| j �}t|�}|dkr |du r td��|dkr'td��t|� \}}}dd� t|�D �} |du rFtj dd� |D �� � n|� t� | ||| j j 467 d ��} 468 | � | 469 �} t� | � fd 470 d�|D �d ��} | j | ||d �} | � � t |�ff�S ) a� 471 Converts an n-D structured array into an (n+1)-D unstructured array. 472 473 The new array will have a new last dimension equal in size to the 474 number of field-elements of the input array. If not supplied, the output 475 datatype is determined from the numpy type promotion rules applied to all 476 the field datatypes. 477 478 Nested fields, as well as each element of any subarray fields, all count 479 as a single field-elements. 480 481 Parameters 482 ---------- 483 arr : ndarray 484 Structured array or dtype to convert. Cannot contain object datatype. 485 dtype : dtype, optional 486 The dtype of the output unstructured array. 487 copy : bool, optional 488 See copy argument to `numpy.ndarray.astype`. If true, always return a 489 copy. If false, and `dtype` requirements are satisfied, a view is 490 returned. 491 casting : {'no', 'equiv', 'safe', 'same_kind', 'unsafe'}, optional 492 See casting argument of `numpy.ndarray.astype`. Controls what kind of 493 data casting may occur. 494 495 Returns 496 ------- 497 unstructured : ndarray 498 Unstructured array with one more dimension. 499 500 Examples 501 -------- 502 503 >>> from numpy.lib import recfunctions as rfn 504 >>> a = np.zeros(4, dtype=[('a', 'i4'), ('b', 'f4,u2'), ('c', 'f4', 2)]) 505 >>> a 506 array([(0, (0., 0), [0., 0.]), (0, (0., 0), [0., 0.]), 507 (0, (0., 0), [0., 0.]), (0, (0., 0), [0., 0.])], 508 dtype=[('a', '<i4'), ('b', [('f0', '<f4'), ('f1', '<u2')]), ('c', '<f4', (2,))]) 509 >>> rfn.structured_to_unstructured(a) 510 array([[0., 0., 0., 0., 0.], 511 [0., 0., 0., 0., 0.], 512 [0., 0., 0., 0., 0.], 513 [0., 0., 0., 0., 0.]]) 514 515 >>> b = np.array([(1, 2, 5), (4, 5, 7), (7, 8 ,11), (10, 11, 12)], 516 ... dtype=[('x', 'i4'), ('y', 'f4'), ('z', 'f8')]) 517 >>> np.mean(rfn.structured_to_unstructured(b[['x', 'z']]), axis=-1) 518 array([ 3. , 5.5, 9. , 11. ]) 519 520 N� arr must be a structured arrayr z(arr has no fields. Unable to guess dtypez#arr with no fields is not supportedc S � g | ]}d � |��qS �zf{}��formatr� r r r" r5 � rq z.structured_to_unstructured.<locals>.<listcomp>c S � g | ]}|j �qS r r� �r. r� r r r" r5 � � �r& �formats�offsetsr� c s g | ]}� |j f�qS r )r� r� �� out_dtyper r" r5 � rq �r& r� �r� r� )r% r& r' r� r( �NotImplementedErrorrx r� rB � result_typer� r] r� �sum) r� r% r� r� r- �n_fields�dts�countsr� r& �flattened_fields� packed_fieldsr r� r" r } s0 5 521 � 522 �r c C r� r r )r� r% r& r� r� r� r r r" �&_unstructured_to_structured_dispatcher� r� r� c sr � j dkr td��� j d }|dkrtd��|du rA|du r'dd� t|�D �}tj� fd d�|D �|d 523 �}t|�}t|� \} } 524 } n<|durItd ��t�|�}t|�}t|�dkrag g g } } 525 } nt|� \} } 526 } |t | 527 �krrtd ��|}|r}|j 528 s}td ��dd� tt|��D �}t�|� fdd�| D �d��} t� � �� | �� t�|| | |j d��} � j| ||d�� � � |�d S )a 529 Converts an n-D unstructured array into an (n-1)-D structured array. 530 531 The last dimension of the input array is converted into a structure, with 532 number of field-elements equal to the size of the last dimension of the 533 input array. By default all output fields have the input array's dtype, but 534 an output structured dtype with an equal number of fields-elements can be 535 supplied instead. 536 537 Nested fields, as well as each element of any subarray fields, all count 538 towards the number of field-elements. 539 540 Parameters 541 ---------- 542 arr : ndarray 543 Unstructured array or dtype to convert. 544 dtype : dtype, optional 545 The structured dtype of the output array 546 names : list of strings, optional 547 If dtype is not supplied, this specifies the field names for the output 548 dtype, in order. The field dtypes will be the same as the input array. 549 align : boolean, optional 550 Whether to create an aligned memory layout. 551 copy : bool, optional 552 See copy argument to `numpy.ndarray.astype`. If true, always return a 553 copy. If false, and `dtype` requirements are satisfied, a view is 554 returned. 555 casting : {'no', 'equiv', 'safe', 'same_kind', 'unsafe'}, optional 556 See casting argument of `numpy.ndarray.astype`. Controls what kind of 557 data casting may occur. 558 559 Returns 560 ------- 561 structured : ndarray 562 Structured array with fewer dimensions. 563 564 Examples 565 -------- 566 567 >>> from numpy.lib import recfunctions as rfn 568 >>> dt = np.dtype([('a', 'i4'), ('b', 'f4,u2'), ('c', 'f4', 2)]) 569 >>> a = np.arange(20).reshape((4,5)) 570 >>> a 571 array([[ 0, 1, 2, 3, 4], 572 [ 5, 6, 7, 8, 9], 573 [10, 11, 12, 13, 14], 574 [15, 16, 17, 18, 19]]) 575 >>> rfn.unstructured_to_structured(a, dt) 576 array([( 0, ( 1., 2), [ 3., 4.]), ( 5, ( 6., 7), [ 8., 9.]), 577 (10, (11., 12), [13., 14.]), (15, (16., 17), [18., 19.])], 578 dtype=[('a', '<i4'), ('b', [('f0', '<f4'), ('f1', '<u2')]), ('c', '<f4', (2,))]) 579 580 r z$arr must have at least one dimensionrm r z&last axis with size 0 is not supportedNc S r� r� r� r� r r r" r5 rq z.unstructured_to_structured.<locals>.<listcomp>c s g | ]}|� j f�qS r r0 r� �r� r r" r5 rq r� z!don't supply both dtype and nameszVThe length of the last dimension of arr must be equal to the number of fields in dtypez'align was True but dtype is not alignedc S r� r� r� r� r r r" r5 4 rq c s g | ]}� j |jf�qS r )r% r� r� r� r r" r5 : � r� r� r� ).r )r� r' r� r� rB r% r� rx r( r� �isalignedstruct�ascontiguousarrayr] r� r� )r� r% r& r� r� r� Zn_elemr� r- r� r� r� r� r� r r� r" r � sF 581 8 582 583 584 ��r c C s |fS r r )�funcr� r r r" � _apply_along_fields_dispatcherG r� r� c C s( |j jdu r 585 td��t|�}| |dd�S )a@ 586 Apply function 'func' as a reduction across fields of a structured array. 587 588 This is similar to `apply_along_axis`, but treats the fields of a 589 structured array as an extra axis. The fields are all first cast to a 590 common type following the type-promotion rules from `numpy.result_type` 591 applied to the field's dtypes. 592 593 Parameters 594 ---------- 595 func : function 596 Function to apply on the "field" dimension. This function must 597 support an `axis` argument, like np.mean, np.sum, etc. 598 arr : ndarray 599 Structured array for which to apply func. 600 601 Returns 602 ------- 603 out : ndarray 604 Result of the recution operation 605 606 Examples 607 -------- 608 609 >>> from numpy.lib import recfunctions as rfn 610 >>> b = np.array([(1, 2, 5), (4, 5, 7), (7, 8 ,11), (10, 11, 12)], 611 ... dtype=[('x', 'i4'), ('y', 'f4'), ('z', 'f8')]) 612 >>> rfn.apply_along_fields(np.mean, b) 613 array([ 2.66666667, 5.33333333, 8.66666667, 11. ]) 614 >>> rfn.apply_along_fields(np.mean, b[['x', 'z']]) 615 array([ 3. , 5.5, 9. , 11. ]) 616 617 Nr� rm )�axis)r% r& r' r )r� r� �uarrr r r" r J s # r c C r r r )�dst�src�zero_unassignedr r r" �!_assign_fields_by_name_dispatcheru r$ r� c C sX | j jdu r || d< dS | j jD ]}||j jvr |r d| |< qt| | || |� qdS )a� 618 Assigns values from one structured array to another by field name. 619 620 Normally in numpy >= 1.14, assignment of one structured array to another 621 copies fields "by position", meaning that the first field from the src is 622 copied to the first field of the dst, and so on, regardless of field name. 623 624 This function instead copies "by field name", such that fields in the dst 625 are assigned from the identically named field in the src. This applies 626 recursively for nested structures. This is how structure assignment worked 627 in numpy >= 1.6 to <= 1.13. 628 629 Parameters 630 ---------- 631 dst : ndarray 632 src : ndarray 633 The source and destination arrays during assignment. 634 zero_unassigned : bool, optional 635 If True, fields in the dst for which there was no matching 636 field in the src are filled with the value 0 (zero). This 637 was the behavior of numpy <= 1.13. If False, those fields 638 are not modified. 639 N.r )r% r& r 640 )r� r� r� r/ r r r" r 641 x s ���r 642 c C r� r r )r} �required_dtyper r r" �_require_fields_dispatcher� r� r� c C s t j| j|d�}t|| � |S )a� 643 Casts a structured array to a new dtype using assignment by field-name. 644 645 This function assigns from the old to the new array by name, so the 646 value of a field in the output array is the value of the field with the 647 same name in the source array. This has the effect of creating a new 648 ndarray containing only the fields "required" by the required_dtype. 649 650 If a field name in the required_dtype does not exist in the 651 input array, that field is created and set to 0 in the output array. 652 653 Parameters 654 ---------- 655 a : ndarray 656 array to cast 657 required_dtype : dtype 658 datatype for output array 659 660 Returns 661 ------- 662 out : ndarray 663 array with the new dtype, with field values copied from the fields in 664 the input array with the same name 665 666 Examples 667 -------- 668 669 >>> from numpy.lib import recfunctions as rfn 670 >>> a = np.ones(4, dtype=[('a', 'i4'), ('b', 'f8'), ('c', 'u1')]) 671 >>> rfn.require_fields(a, [('b', 'f4'), ('c', 'u1')]) 672 array([(1., 1), (1., 1), (1., 1), (1., 1)], 673 dtype=[('b', '<f4'), ('c', 'u1')]) 674 >>> rfn.require_fields(a, [('b', 'f4'), ('newf', 'u1')]) 675 array([(1., 0), (1., 0), (1., 0), (1., 0)], 676 dtype=[('b', '<f4'), ('newf', 'u1')]) 677 678 r0 )rB r� r� r 679 )r} r� �outr r r" r � s ' 680 r c C rj r r )�arraysrf r` ra � autoconvertr r r" �_stack_arrays_dispatcher� rl r� c C s� t | t�r| S t| �dkr| d S dd� | D �}dd� |D �}dd� |D �}dd� |D �}|d } t| �} 681 dd� | 682 D �} |dd � D ]?} t| �D ]8\} }| | vr]| 683 �| |f� | �| � qH| �| �}| 684 | \}}|rt| t||�f| 685 |< qH||kr�td 686 ||f ��qHqBt| 687 �dkr�t� |�}nYt� 688 t � |�f| 689 �}t � t jd|f �}g }t|||d d � |dd � �D ]3\}}}}|jj} | d u r�||d t|� ||�< q�|D ]}|| || ||�< ||vr�|�|� q�q�tt||�||d �S )a� 690 Superposes arrays fields by fields 691 692 Parameters 693 ---------- 694 arrays : array or sequence 695 Sequence of input arrays. 696 defaults : dictionary, optional 697 Dictionary mapping field names to the corresponding default values. 698 usemask : {True, False}, optional 699 Whether to return a MaskedArray (or MaskedRecords is 700 `asrecarray==True`) or a ndarray. 701 asrecarray : {False, True}, optional 702 Whether to return a recarray (or MaskedRecords if `usemask==True`) 703 or just a flexible-type ndarray. 704 autoconvert : {False, True}, optional 705 Whether automatically cast the type of the field to the maximum. 706 707 Examples 708 -------- 709 >>> from numpy.lib import recfunctions as rfn 710 >>> x = np.array([1, 2,]) 711 >>> rfn.stack_arrays(x) is x 712 True 713 >>> z = np.array([('A', 1), ('B', 2)], dtype=[('A', '|S3'), ('B', float)]) 714 >>> zz = np.array([('a', 10., 100.), ('b', 20., 200.), ('c', 30., 300.)], 715 ... dtype=[('A', '|S3'), ('B', np.double), ('C', np.double)]) 716 >>> test = rfn.stack_arrays((z,zz)) 717 >>> test 718 masked_array(data=[(b'A', 1.0, --), (b'B', 2.0, --), (b'a', 10.0, 100.0), 719 (b'b', 20.0, 200.0), (b'c', 30.0, 300.0)], 720 mask=[(False, False, True), (False, False, True), 721 (False, False, False), (False, False, False), 722 (False, False, False)], 723 fill_value=(b'N/A', 1.e+20, 1.e+20), 724 dtype=[('A', 'S3'), ('B', '<f8'), ('C', '<f8')]) 725 726 rA r c S s g | ] }t �|��� �qS r )rB ro rv rs r r r" r5 � r� z stack_arrays.<locals>.<listcomp>c S s g | ]}t |��qS r r3 rs r r r" r5 � c S r� r r0 rs r r r" r5 r� c S r� r r6 )r. r� r r r" r5 r� c S s g | ]\}}|�qS r r )r. r� r� r r r" r5 r� Nz Incompatible type '%s' <> '%s'rm zf%ir� )rM r r( r7 r8 �indexrw � TypeErrorr^ � concatenater� rB r� �cumsum�r_rx r% r& rb ri )r� rf r` ra r� rC Znrecordsr= �fldnamesZdtype_lZnewdescrr& Zdtype_n�fname�fdtype�nameidxr@ �cdtyper! r� �seenrE r� r� �jr/ r r r" r � sX 727 ) 728 ��� * 729 �� 730 �r c C r� r r )rE �key� 731 ignoremask� return_indexr r r" �_find_duplicates_dispatcher* r� r� c C s� t �| ��� } t| j�}| }|r || D ]}|| }q|| }|�� }|| }|�� } | dd� | dd� k} 732 |rD|j} d| 733 | dd� < t �dg| 734 f�} 735 | 736 dd� | 737 dd� | 738 dd�< | | | 739 } |rj| || 740 fS | S )a� 741 Find the duplicates in a structured array along a given key 742 743 Parameters 744 ---------- 745 a : array-like 746 Input array 747 key : {string, None}, optional 748 Name of the fields along which to check the duplicates. 749 If None, the search is performed by records 750 ignoremask : {True, False}, optional 751 Whether masked data should be discarded or considered as duplicates. 752 return_index : {False, True}, optional 753 Whether to return the indices of the duplicated values. 754 755 Examples 756 -------- 757 >>> from numpy.lib import recfunctions as rfn 758 >>> ndtype = [('a', int)] 759 >>> a = np.ma.array([1, 1, 1, 2, 2, 3, 3], 760 ... mask=[0, 0, 1, 0, 0, 0, 1]).view(ndtype) 761 >>> rfn.find_duplicates(a, ignoremask=True, return_index=True) 762 (masked_array(data=[(1,), (1,), (2,), (2,)], 763 mask=[(False,), (False,), (False,), (False,)], 764 fill_value=(999999,), 765 dtype=[('a', '<i8')]), array([0, 1, 3, 4])) 766 Nrm rA F) rB ro rv r r% �argsortr_ � 767 recordmaskr� ) rE r� r� r� r- r� r4 ZsortidxZ 768 sortedbaseZ 769 sorteddata�flagZ 770 sortedmask� 771 duplicatesr r r" r / s( 772 773 r c C � ||fS r r ) r� �r1�r2�jointype� r1postfix� r2postfixrf r` ra r r r" �_join_by_dispatcheri � r �inner�1�2c - s� |dvr 774 t d| ��t� t�r� f� tt� ��t� �kr/t� fdd�t� �D ��} t d| ��� D ]} 775 | 776 |jjvr?t d| 777 ��| 778 |jjvrKt d| 779 ��q1|� � }|� � }t|�} |jj|jj} } t| �t| �@ t� � }|r}|s}|s}d}|d 7 }t |��� fd 780 d �| D �}t 781 ||�}t 782 ||�}t � ||f�}|j � d �}|| }t � d g|dd� |dd� kf�}|dd� |dd� |dd�< || }||| k }||| k | }t|�t|�}}|dkr�d\}} nN|dk�r|| } t� || | | k f�}t� || | | k | f�}t|�| t|�| }} n |dk�r5|| } t� || | | k f�}t|�| d}} || || } } t|j�} t|j�D ]\}!}"|!� v�rX| �|!|"f� �qHt|j�D ]P\}!}"tdd� | D ��}#z|#�|!�}$W n t �y� | �|!|"f� Y �q_w | |$ \}%}&|!� v �r�|!t|"|&�f| |$< �q_|!| |&f|!| |"fg| |$|$d �< �q_t�| �} t||�}'t j|'| | f| d�}(|(jj}#| D ]<})| |) }*|)|#v�s�|)| v �r�|�s�|)� v�r�|)|7 })|(|) }+|*d|� |+d|�< |dv �r|*|d� |+|'|'| �< �q�| D ]>})| |) }*|)|#v�s#|)| v �r'|�s'|)� v�r'|)|7 })|(|) }+|*d|� |+d|�< |dk�rH| �rH|*|d� |+| d�< �q |(j� d � t||d�},tt|(|�fi |,��S )a< 783 Join arrays `r1` and `r2` on key `key`. 784 785 The key should be either a string or a sequence of string corresponding 786 to the fields used to join the array. An exception is raised if the 787 `key` field cannot be found in the two input arrays. Neither `r1` nor 788 `r2` should have any duplicates along `key`: the presence of duplicates 789 will make the output quite unreliable. Note that duplicates are not 790 looked for by the algorithm. 791 792 Parameters 793 ---------- 794 key : {string, sequence} 795 A string or a sequence of strings corresponding to the fields used 796 for comparison. 797 r1, r2 : arrays 798 Structured arrays. 799 jointype : {'inner', 'outer', 'leftouter'}, optional 800 If 'inner', returns the elements common to both r1 and r2. 801 If 'outer', returns the common elements as well as the elements of 802 r1 not in r2 and the elements of not in r2. 803 If 'leftouter', returns the common elements and the elements of r1 804 not in r2. 805 r1postfix : string, optional 806 String appended to the names of the fields of r1 that are present 807 in r2 but absent of the key. 808 r2postfix : string, optional 809 String appended to the names of the fields of r2 that are present 810 in r1 but absent of the key. 811 defaults : {dictionary}, optional 812 Dictionary mapping field names to the corresponding default values. 813 usemask : {True, False}, optional 814 Whether to return a MaskedArray (or MaskedRecords is 815 `asrecarray==True`) or a ndarray. 816 asrecarray : {False, True}, optional 817 Whether to return a recarray (or MaskedRecords if `usemask==True`) 818 or just a flexible-type ndarray. 819 820 Notes 821 ----- 822 * The output is sorted along the key. 823 * A temporary array is formed by dropping the fields not in the key for 824 the two arrays and concatenating the result. This array is then 825 sorted, and the common entries selected. The output is constructed by 826 filling the fields with the selected entries. Matching is not 827 preserved if there are some duplicates... 828 829 )r �outer� leftouterzWThe 'jointype' argument should be in 'inner', 'outer' or 'leftouter' (got '%s' instead)c 3 s, � | ]\}}|� |d d� v r|V qdS )rA Nr )r. r� �x�r� r r" r1 � s �* zjoin_by.<locals>.<genexpr>zduplicate join key %rz r1 does not have key field %rz r2 does not have key field %rz8r1 and r2 contain common names, r1postfix and r2postfix zcan't both be emptyc s g | ]}|� v r|�qS r r r� r r r" r5 � r� zjoin_by.<locals>.<listcomp>)�orderFrA Nrm r )r r r r r c s s � | ]\}}|V qd S r r )r. r/ r% r r r" r1 � s � r0 )r r r� )r' rM rU r( r� �next� enumerater% r& rv r� r^ r� r� rB r7 r8 r� r� rw r� �sort�dictrb ri )-r� r� r� r r r rf r` ra �dupr/ �nb1Zr1namesZr2names� 830 collisionsr� �key1Zr1kZr2k�auxZidx_sortZflag_in�idx_inZidx_1Zidx_2Zr1cmnZr2cmnZr1spcZr2spc�idx_out�s1�s2r= r� r� r& r� r@ r� Zcmnr! r4 �selectedr+ �kwargsr r r" r o s� 4�� 831 � 832 833 $ 834 835 836 837 838 839 840 �� 841 842 843 � 844 845 $ 846 �$� r c C r� r r )r� r� r� r r r rf r r r" �_rec_join_dispatcher# r r c C s( t ||||ddd�}t| ||fi |��S )z� 847 Join arrays `r1` and `r2` on keys. 848 Alternative to join_by, that always returns a np.recarray. 849 850 See Also 851 -------- 852 join_by : equivalent function 853 FT)r r r rf r` ra )r r )r� r� r� r r r rf r r r r" r ) s �r )F)NN)NT)TFr )NNNN)rm FFF)Nrm TF)FF)r )NNN)NFr� )NNNNN)NNFFr� )T)NTFF)NTF)NNNNNN)r r r NTF)r r r N)B�__doc__rX �numpyrB �numpy.mar^ r r r �numpy.ma.mrecordsr �numpy.core.overridesr �numpy.lib._iotoolsr �corer{ �__all__r# r r7 r r r rF rH r rO rV r\ rb ri rk r r� r r� r� r r� r r� r r� r r� r r� r� r r� r r� r r� r 854 r� r r� r r� r r r r r r r r r" �<module> s� 855 856 & 857 858 859 / 860 861 862 863 � � 864 865 @ 866 867 ' 868 � � 869 F 870 " 871 R& 872 � Z 873 � �j 874 875 * % 876 + 877 � �X 878 � : 879 � � 5 880 ��