o "i/@sUddlmZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZd d lmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$e ed dtddZ%e ed duddZ&e ed dvddZ'e(dde)DZ*de+d<dde*DZ,de+d<e ed dwd!d"Z-e ed dud#d$Z.e ed dud%d&Z/e ed dud'd(Z0e ed dud)d*Z1e ed dud+d,Z2e ed dud-d.Z3e ed dud/d0Z4e ed dud1d2Z5e ed dud3d4Z6e ed dud5d6Z7e ed dud7d8Z8e ed dud9d:Z9e ed dud;d<Z:e ed dud=d>Z;e edydzdHdIZ?e dJd d{dLdMZ@d|dOdPZAd}dRdSZBd~ddWdXZCdd\d]ZDdd^d_ZEd`ejFdafddedfZG dddrdsZHdS)) annotationsN) bisect_right)IncrementalDecoder)aliases) lru_cache)findall) Generator)MultibyteIncrementalDecoder)ENCODING_MARKSIANA_SUPPORTED_SIMILARRE_POSSIBLE_ENCODING_INDICATIONUNICODE_RANGES_COMBINEDUNICODE_SECONDARY_RANGE_KEYWORDUTF8_MAXIMAL_ALLOCATIONCOMMON_CJK_CHARACTERS_LATIN_CJK_HANGUL _KATAKANA _HIRAGANA_THAI_ARABIC_ARABIC_ISOLATED_FORM_ACCENT_KEYWORDS _ACCENTUATED)maxsize characterstrreturnintcCszt|}Wn tyYdSwd}d|vr|tO}d|vr$|tO}d|vr,|tO}d|vr4|tO}d|vr<|tO}d|vrD|tO}d|vrT|t O}d |vrT|t O}t D] }||vrc|t O}|SqV|S) zRCompute all name-based classification flags with a single unicodedata.name() call.rLATINCJKHANGULKATAKANAHIRAGANATHAIARABICz ISOLATED FORM) unicodedataname ValueErrorrrrrrrrrrr)rdescflagskwr.\/var/www/html/voicebot/backend/venv/lib/python3.10/site-packages/charset_normalizer/utils.py_character_flags&s: r0boolcCtt|t@SN)r1r0rrr.r.r/is_accentuatedIr5cCs.t|}|s |S|d}tt|ddS)N r)r( decompositionsplitchrr )r decomposedcodesr.r.r/ remove_accentNs  r>ccs"|] \}}|j|j|fVqdSr3)startstop).0r) ord_ranger.r.r/ [s  rCzlist[tuple[int, int, str]]_UNICODE_RANGES_SORTEDcCsg|]}|dqS)rr.)rAer.r.r/ _srFz list[int]_UNICODE_RANGE_STARTS str | NonecCs<t|}tt|d}|dkrt|\}}}||kr|SdS)zK Retrieve the Unicode range official name from a single character. r rN)ordrrGrD)r character_ordidxr?r@r)r.r.r/ unicode_rangebsrLcCr2r3)r1r0rr4r.r.r/is_latinsr6rMcCs2t|}d|vr dSt|}|durdSd|vS)NPTF Punctuationr(categoryrLrcharacter_categorycharacter_ranger.r.r/is_punctuationxs rUcCsBt|}d|vs d|vrdSt|}|durdSd|vo |dkS)NSNTFFormsLorPrRr.r.r/ is_symbols rZcCs$t|}|dur dSd|vpd|vS)NF Emoticons Pictographs)rL)rrTr.r.r/ is_emoticonsr]cCs.|s|dvr dSt|}d|vp|dvS)N>｜+<>TZ>PcPdPo)isspacer(rQ)rrSr.r.r/ is_separators rgcCs||kSr3)islowerisupperr4r.r.r/is_case_variabler6rjcCr2r3)r1r0rr4r.r.r/is_cjkr6rkcCr2r3)r1r0rr4r.r.r/ is_hiraganar6rlcCr2r3)r1r0rr4r.r.r/ is_katakanar6rmcCr2r3)r1r0rr4r.r.r/ is_hangulr6rncCr2r3)r1r0rr4r.r.r/is_thair6rocCr2r3)r1r0rr4r.r.r/ is_arabicr6rpcCr2r3)r1r0rr4r.r.r/is_arabic_isolated_formr6rqcCs|tvSr3)rr4r.r.r/is_cjk_uncommonsrr range_namecstfddtDS)Nc3s|]}|vVqdSr3r.)rAkeywordrsr.r/rCsz-is_unicode_range_secondary..)anyrrur.rur/is_unicode_range_secondarysrwcCs(|duo|duo|dko|dkS)NFu)rf isprintabler4r.r.r/is_unprintables  rz sequencebytes search_zonecCst|ttfs tt|}tt|dt||jddd}t|dkr&dS|D]'}| dd}t D]\}}||krD|S||krN|Sq6q(dS)zW Extract using ASCII-only decoder any specified encoding in the first n-bytes. Nasciiignoreerrorsr-_) isinstancer} bytearray TypeErrorlenrr mindecodelowerreplaceritems)r|r~seq_lenresultsspecified_encodingencoding_alias encoding_ianar.r.r/any_specified_encodings&   rr)cCs |dvpttd|jtS)zQ Verify is a specific encoding is a multi byte one based on it IANA name > utf_7utf_8utf_16utf_32 utf_16_be utf_16_le utf_32_be utf_32_le utf_8_sig encodings.) issubclass importlib import_modulerr )r)r.r.r/is_multi_byte_encodings  rtuple[str | None, bytes]cCsJtD] }t|}t|tr|g}|D]}||r!||fSqqdS)z9 Identify and extract SIG/BOM in given sequence. )N)r rr} startswith)r| iana_encodingmarksmarkr.r.r/identify_sig_or_boms  rrcCs|dvS)N>rrr.)rr.r.r/should_strip_sig_or_bom.srTcp_namestrictcCsN|dd}tD]\}}|||fvr|Sq |r%td|d|S)zIReturns the Python normalized encoding name (Not the IANA official name).rrzUnable to retrieve IANA for '')rrrrr*)rrrrr.r.r/ iana_name2s r iana_name_a iana_name_bfloatc Cst|st|r dStd|j}td|j}|dd}|dd}d}tdD]}t|g}||||krA|d7}q,|dS)Ngrrrrr )rrrrranger}r) rr decoder_a decoder_bid_aid_bcharacter_match_counti to_be_decodedr.r.r/ cp_similarityCs    rcCs|tvo |t|vS)z Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using the function cp_similarity. )r )rrr.r.r/ is_cp_similarWs rcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)slevel format_stringNonecCs:t|}||t}|t|||dSr3)logging getLoggersetLevel StreamHandler setFormatter Formatter addHandler)r)rrloggerhandlerr.r.r/set_logging_handlerbs  r sequencesroffsetsr chunk_sizebom_or_sig_availablestrip_sig_or_bom sig_payloadis_multi_byte_decoderdecoded_payloadGenerator[str, None, None]c cs&|r|dur|D]} || | |} | sdS| Vq dS|D]p} | |} | t|dkr/q || | |} |rA|durA|| } | j||rHdndd} |r| dkrt|d} |r| d| |vrt| | dd D]#}||| } |r{|dur{|| } | j|dd} | d| |vrnqi| Vq dS) NFrrrrr8)rrrr)rrrrrrrrrrchunk chunk_end cut_sequencechunk_partial_size_chkjr.r.r/cut_sequence_chunksosD       r)rrrr )rrrr1)rrrr)rrrrH)rsrrr1)r{)r|r}r~r rrH)r)rrr1)r|r}rr)rrrr1)T)rrrr1rr)rrrrrr)rrrrrr1)r)rrr rrrrr3)rr}rrrrrr rr1rr1rr}rr1rrHrr)I __future__rrrr(bisectrcodecsrencodings.aliasesr functoolsrrertypingr_multibytecodecr constantr r r rrrrrrrrrrrrrrr0r5r>sortedrrD__annotations__rGrLrMrUrZr]rgrjrkrlrmrnrorprqrrrrwrzrrrrrrrINFOrrr.r.r.r/s       L "