edZdZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZdd lmZmZdd lmZdd lmZdd lmZd dlmZd dlmZddlmZddlm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(GddeZ)e*dkrce&ej+dej, e)-dS#e $rZ.e/e.e0d dZ.[.wwxYwdS)zd This module contains SSA classes for sending e-mails with report and sending reports to ClickHouse z ssa.modulesN) EmailMessage)partial)OptionalAny)get_admin_email)is_cl_solo_edition)SessionResponse) HTTPAdapter)RequestException)Retry)Common) DecisionMaker) stat_server)SSAError)Mailerrender_report_table) read_sys_id sentry_init duration_cast format_dateceZdZdZfdZddeddfdZdedeefdZ dee fd Z dee fd Z dede fd Zed edefd ZdedefdZdede fdZxZS)StatisticsSenderz4 Send report to ClickHouse and over e-mails cLttjd|_|jdt t|_dtd|_ t|_ tdtdgtgdd}t| }t!|_|jd|t'|jjd |j_dS) N stat_senderzStatisticsSender enabled: %szhttps://z /api/clos-ssaPOST)iii)totalallowed_methodsstatus_forcelistbackoff_factor) max_retries )timeout)super__init__logging getLoggerloggerinfo __package__rsys_idr ch_endpointr mail_senderr frozensetr r sessionmountrrequest)self retry_confadapter __class__s N/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/modules/stat_sender.pyr(zStatisticsSender.__init__*s ' 66  7EEE!mm @k@@@!88+4fX+>+>,5ooo,F,F*+--- *555yy  :w///&t|';RHHH Nreportreturnc| t}|jr||||dS)zi Send given report over e-mail If no report given, get it from DecisionMaker API N)rget_json_reportsummary_notification_enabled email_reportclickhouse_report)r5r;s r9sendzStatisticsSender.send;sW >"__4466F  , &   f % % % v&&&&&r: report_viewc|}|r|drt|\}}|j|dt |d|}|tj|dd|dd||d d|dd |j ||Sd Sd S) z4 Create and send e-mail with report domains ssa_reportdate) recipienttemplaterGhtmljsonreport_z.json)subtypefilenamerJz.htmlN) get_mail_recipientgetrr0_messageradd_attachmentrKdumps_send)r5rCmail_to report_tablemailmsgs r9r@zStatisticsSender.email_reportGs())++  {y11 !4[!A!A L$"++!% V!455 ,C   tz+66'-(L+f2E(L(L(L  N N N   |'-(L+f2E(L(L(L  N N N   " "3 ' ' 'J#    r:c |p tS#t$r3}|jdt |Yd}~dSd}~wwxYw)z Retrieve a recipient's e-mail: 1. get address from a wmt-api utility 2. if command failed or address is empty, get address of server admin zget_admin_email failed with: %sN)wmt_api_report_emailr Exceptionr+errorstr)r5es r9rOz#StatisticsSender.get_mail_recipient`sy  I,,..C/2C2C C I I I K  ?Q H H H H H H H H H Is!$ A!(AA!ctdrdnd} tjd|dgdddj}nX#tjt ttf$r3}|j dt|Yd }~d Sd }~wwxYw tj |d d S#tj$r3}|j d t|Yd }~d Sd }~wt $r|j d |Yd SwxYw)zD Retrieve a recipient's e-mail address from WMT API T)skip_jwt_checkz wmt-api-solozwmt-apiz&/usr/share/web-monitoring-tool/wmtbin/z --config-get)checktextcapture_outputzwmt-api utility failed: %sNconfig report_emailz!wmt-api returned invalid json: %sz(wmt-api returned unexpected response: %s)r subprocessrunstdoutstripCalledProcessErrorAttributeErrorOSError ValueErrorr+r\r]rKloadsrPJSONDecodeError)r5_util api_responser^s r9rZz%StatisticsSender.wmt_api_report_emaills#5#!#!#!0&/  %>A%AA!# %%%&,EEGG L -~w    K  :CFF C C C FFFFF   ,:l++//99==nMM M# & & & K  A!!ff & & & & & & & & & , , , K  H* , , , , , , ,s54A "B -(BB $9CE-(D(EEc |dr|jd|j |j|j|||}nV#t$rI}|j d|d|jitdt||d}~wwxYw| |S|jd d S) z+ Send report to ClickHouse rEzSending POST request to %s)rKzPOST failed with %sendpointextraz'Failed to POST data to SSA API server: Nz*Report is empty, not sending to ClickHouseF) rPr+r,r/r2post_ch_packclickhouse_formatr r\rr]_process_response)r5rCrespr^s r9rAz"StatisticsSender.clickhouse_reports5 ??9 % %  K  9!- / / / O|(()9 **;77A9A9(::# O O O !!"7)3T5E(F"HHHFc!ffFFHHMNO O ))$// / K  I J J J5sAA?? C AC  Cvaluec"t|S)z< Pack given value into data field of a dict )data)dict)r{s r9rwzStatisticsSender._ch_packs r:original_reportc ft}|dD]}||j|d|d|d|dd|dDd|S) zd Format local report for sending to ClickHouse (required structures differ) rEname slow_urls slow_reqs total_reqsc g|]p}|d|dt|dt|dddqS)rreqs_numaverage_duration correlationr)urlcount_requests avg_durationr)rPrfloat).0us r9 z6StatisticsSender.clickhouse_format..s !uuV}}*+%% *;*;(5EE"455)7)7',QUU=!-D-D'E'E r:urls) system_iddomaincount_slow_urlscount_slow_requeststotal_requestsdetails)listrPappendr.)r5r ch_reportrs r9rxz"StatisticsSender.clickhouse_formats FF %)))44  F   ![ **V,,#)::k#:#:'-zz+'>'>"(**\":": &zz&11      r:responsec|js1|jd|j|jd|jidS|jd|j|j|j|}|ddkr'|jd|dd |idS|jd d S) z Check received response :param response: a requests.Response object :return: True in case of success, False otherwise z&Unable to connect to server with %s:%s resp_textrtFz[%s:%s] Response received %sstatusokz Received response with status %srzSent to ClickHouse successfullyT) rr+r\ status_codereasonrbr,rrK)r5rresults r9ryz"StatisticsSender._process_responses { + K  F&2HO%0(-$@  B B B5 K  ;%18?%\ + + + ( t # # K  @$X.z66J  L L L5 :;;;tr:)N)__name__ __module__ __qualname____doc__r(r~rBrrr@r]rOrZboolrA staticmethodrrwrrxr ry __classcell__)r8s@r9rr%slIIIII" ' '4 '4 ' ' ' ',1G2 IHSM I I I I,hsm,,,,4Td(     \ $2(tr:r__main__zstat_sender_standalone.log)rNlevel)1rr-rKr)rf email.messager functoolsrtypingrrclcommon.cpapirclcommon.lib.cleditionrrequestsr r requests.adaptersr requests.exceptionsr $requests.packages.urllib3.util.retryr commonrdecision_makerrinternal.constantsrinternal.exceptionsrinternal.mailerrrinternal.utilsrrrrrr basicConfigINFOrBexcprint SystemExitr:r9rsU &&&&&& ******555555&&&&&&&&))))))000000666666)))))),,,,,,******99999999lllllvlll^ zKMMMG!=%l,,,,!!!!!  c jmm s-C C+C&&C+