api.pyc
1 o 2 ��Yc�J � @ s d dl Z d dlZd dlmZ d dlmZmZ d dlmZm Z m 3 Z 4 m Z m Z ddl mZmZmZmZ ddlmZmZmZmZ ddlmZ dd lmZmZ dd 5 l m Z m Z m Z m Z m!Z!m"Z"m#Z# e �$d �Z%e �&� Z'e'�(e �)d �� d&de*de+de+de,de e 6 e- de e 7 e- de.de.defdd �Z/ d&d e de+de+de,de e 8 e- de e 9 e- de.de.defd d �Z0 d&d d!de+de+de,de e 10 e- de e 11 e- de.de.defd"d#�Z1 d'd d!de+de+de,de e 12 e- de e 13 e- de.defd$d%�Z2dS )(� N)�PathLike)�basename�splitext)�Any�BinaryIO�List�Optional�Set� )�coherence_ratio�encoding_languages�mb_encoding_languages�merge_coherence_ratios)�IANA_SUPPORTED�TOO_BIG_SEQUENCE�TOO_SMALL_SEQUENCE�TRACE)� 14 mess_ratio)� CharsetMatch�CharsetMatches)�any_specified_encoding�cut_sequence_chunks� iana_name�identify_sig_or_bom� is_cp_similar�is_multi_byte_encoding�should_strip_sig_or_bom�charset_normalizerz)%(asctime)s | %(levelname)s | %(message)s� � 皙�����?TF� sequences�steps� 15 chunk_size� threshold� cp_isolation� cp_exclusion�preemptive_behaviour�explain�returnc - C s� t | ttf�std�t| ����|r tj}t�t � t� 16 t � t | �} | dkrGt� d� |r;t�t � t� 17 |p9tj� tt| dddg d�g�S |dur]t�t d d 18 �|�� d d � |D �}ng }|durut�t d d 19 �|�� dd � |D �}ng }| || kr�t�t d||| � d}| }|dkr�| | |k r�t| | �}t | �tk } 20 t | �tk} | 21 r�t�t d�| �� n | r�t�t d�| �� g } |r�t| �nd} | dur�| �| � t�t d| � t� }g }g }d}d}d}t� }t| �\}}|du�r| �|� t�t dt |�|� | �d� d| v�r| �d� | t D �]�}|�r!||v�r!�q|�r+||v �r+�q||v �r2�q|� |� d}||k}|�oCt |�}|dv �rU|�sUt�t d|� �qzt |�}W n t t!f�yo t�t d|� Y �qw z9| �r�|du �r�t"|du �r�| dtd�� n | t |�td�� |d� nt"|du �r�| n| t |�d� |d�}W n+ t#t$f�y� } z t | t$��s�t�t d|t"| �� |�|� W Y d} ~ �qd} ~ ww d} |D ] } t%|| ��r�d } n�q�| �r�t�t d || � �qt&|�s�dnt |�| t| | ��} |�o|du�ot |�| k } | �r t�t d |� tt | �d �}!t'|!d �}!d}"d}#g }$g }%z9t(| || ||||||� D ]*}&|$�|&� |%�t)|&|�� |%d! |k�rY|"d7 }"|"|!k�sf|�rh|du �rh n�q?W n! t#�y� } zt�t d"|t"| �� |!}"d }#W Y d} ~ nd} ~ ww |#�s�| �r�|�s�z| td#�d� j*|d$d%� W n# t#�y� } zt�t d&|t"| �� |�|� W Y d} ~ �qd} ~ ww |%�r�t+|%�t |%� nd}'|'|k�s�|"|!k�r |�|� t�t d'||"t,|'d( d)d*�� |dd| fv �r|#�st| ||dg |�}(|| k�r|(}n 22 |dk�r|(}n|(}�qt�t d+|t,|'d( d)d*�� |�s2t-|�})nt.|�})|)�rEt�t d,�|t"|)��� g }*|dk�re|$D ]}&t/|&d-|)�r[d.�|)�nd�}+|*�|+� �qNt0|*�},|,�rvt�t d/�|,|�� |�t| ||'||,|�� || ddfv �r�|'d-k �r�t� d0|� |�r�t�t � t� 23 |� t|| g� S ||k�r�t� d1|� |�r�t�t � t� 24 |� t|| g� S �qt |�dk�r&|�s�|�s�|�r�t�t d2� |�r�t� d3|j1� |�|� n2|�r�|du �s|�r |�r |j2|j2k�s|du�rt� d4� |�|� n |�r&t� d5� |�|� |�r8t� d6|�3� j1t |�d � nt� d7� |�rJt�t � t� 25 |� |S )8ae 26 Given a raw bytes sequence, return the best possibles charset usable to render str objects. 27 If there is no results, it is a strong indicator that the source is binary/not text. 28 By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence. 29 And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will. 30 31 The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page 32 but never take it for granted. Can improve the performance. 33 34 You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that 35 purpose. 36 37 This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32. 38 By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain' 39 toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging. 40 Custom logging format and handler can be set manually. 41 z4Expected object of type bytes or bytearray, got: {0}r z<Encoding detection on empty bytes, assuming utf_8 intention.�utf_8g F� Nz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, c S � g | ]}t |d ��qS �F�r ��.0�cp� r2 ��C:\Users\Jacks.GUTTSPC\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\charset_normalizer\api.py� 42 <listcomp>[ � z from_bytes.<locals>.<listcomp>zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c S r, r- r. r/ r2 r2 r3 r4 f r5 z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r 43 z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.�ascii> �utf_16�utf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecoderg �� A)�encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.� � �����zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sg j�@�strict)�errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.�d � )�ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g�������?�,z We detected language {} using {}z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4� 44 isinstance� bytearray�bytes� TypeError�format�type�logger�level� 45 addHandler�explain_handler�setLevelr �len�debug� removeHandler�logging�WARNINGr r �log�join�intr r r �append�setr r �addr r �ModuleNotFoundError� ImportError�str�UnicodeDecodeError� LookupErrorr �range�maxr r �decode�sum�roundr r r r r9 � fingerprint�best)-r! r" r# r$ r% r& r' r( Zprevious_logger_level�lengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodings�specified_encoding�testedZtested_but_hard_failureZtested_but_soft_failureZfallback_asciiZ fallback_u8Zfallback_specified�resultsZ sig_encoding� sig_payload� encoding_iana�decoded_payload�bom_or_sig_available�strip_sig_or_bom�is_multi_byte_decoder�eZsimilar_soft_failure_testZencoding_soft_failed�r_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZlazy_str_hard_failureZ md_chunksZ md_ratios�chunk�mean_mess_ratioZfallback_entryZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedr2 r2 r3 � 46 from_bytes# s� �� 47 48 49 50 �� � ����� 51 � 52 53 � 54 55 56 57 58 �� �� 59 �� 60 ��� � 61 �� �� 62 �� 63 �� 64 � 65 66 � 67 ���� ��� � 68 �� 69 70 � 71 � � 72 73 � 74 �� 75 ����� 76 � 77 78 79 � 80 81 � �� � 82 �� 83 84 85 86 87 � 88 89 90 rs �fpc C s t | �� |||||||�S )z� 91 Same thing than the function from_bytes but using a file pointer that is already ready. 92 Will not close the file pointer. 93 )rs �read)rt r" r# r$ r% r& r' r( r2 r2 r3 �from_fp� s �rv �pathz PathLike[Any]c 94 C sD t | d��}t||||||||�W d � S 1 sw Y dS )z� 95 Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode. 96 Can raise IOError. 97 �rbN)�openrv ) rw r" r# r$ r% r& r' r( rt r2 r2 r3 � from_path� s �$�rz c C s� t �dt� t| ||||||�}t| �}tt|��} t|�dkr'td� |���|� 98 � } 99 | d d| 100 j 7 < t d� t | ��|d�| ���d��} | �| 101 �� � W d � | 102 S 1 sZw Y | 103 S ) zi 104 Take a (text-based) file path and try to create another file next to it, this time using UTF-8. 105 z2normalize is deprecated and will be removed in 3.0r z;Unable to normalize "{}", no encoding charset seems to fit.�-z{}r+ �wbN)�warnings�warn�DeprecationWarningrz r �listr rN �IOErrorrG rd r9 ry r[ �replacerT �write�output) rw r" r# r$ r% r& r' rh �filenameZtarget_extensions�resultrt r2 r2 r3 � normalize s@ �� 106 �� � 107 ��r� )r r r NNTF)r r r NNT)3rQ r} �osr �os.pathr r �typingr r r r r �cdr r r r �constantr r r r �mdr �modelsr r �utilsr r r r r r r � getLoggerrI � StreamHandlerrL � setFormatter� FormatterrE rU �floatr[ �boolrs rv rz r� r2 r2 r2 r3 �<module> s� $ 108 ������ 109 � 110 ��� 111 � G����� 112 � 113 ��� 114 � ����� 115 � 116 ��� 117 � ����� 118 � 119 ���