KMam@sdZddlmZdZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZmZddlmZdd lmZmZdd lmZdd lmZd d dZGdddeZddZddZddZddZej ddfkrpddl!Z!e!j"Z#ddZ$ddZ%ne"Z#eZ$eZ%ddZ&Gd d!d!eZ'Gd"d#d#e(Z)Gd$d%d%eZ*Gd&d'd'e)Z+dS)(zRefactoring framework. Used as a main program, this can refactor any number of files and/or recursively descend down directories. Imported as a module, this provides infrastructure to write your own refactoring tool. )with_statementz#Guido van Rossum N)chain)drivertokenizetoken) find_root)pytreepygram) btm_utils) btm_matcherTcCst|ggdg}tjj|j}g}xhttj|D]Q}|jdrI|jdrI|r|dd}|j |ddqIW|S)zEReturn a sorted list of all available fix names in the given package.*fix_z.pyN) __import__ospathdirname__file__sortedlistdir startswithendswithappend)Z fixer_pkgZ remove_prefixZpkgZ fixer_dirZ fix_namesnamer5/opt/alt/python35/lib64/python3.5/lib2to3/refactor.pyget_all_fix_names!src@seZdZdS) _EveryNodeN)__name__ __module__ __qualname__rrrrr .s r cCst|tjtjfr:|jdkr0t|jhSt|tjrh|jrbt|jStt|tj rt }x5|jD]*}x!|D]}|j t|qWqW|St d|dS)zf Accepts a pytree Pattern Node and returns a set of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s) isinstancer Z NodePatternZ LeafPatterntyper ZNegatedPatternZcontent_get_head_typesZWildcardPatternsetupdate Exception)Zpatrpxrrrr&2s     r&c Cstjt}g}x|D]}|jryt|j}Wntk r_|j|YqXxU|D]}||j|qgWq|jdk r||jj|q|j|qWx:tt j j j t j j D]}||j|qWt|S)z^ Accepts a list of fixers and returns a dictionary of head node type --> fixer list. N) collections defaultdictlistpatternr&r rZ _accept_typerr python_grammarZ symbol2numbervaluestokensextenddict)Z fixer_listZ head_nodesZeveryfixerZheadsZ node_typerrr_get_headnode_dictNs"    r7cs fddtdDS)zN Return the fully qualified names for fixers in the package pkg_name. csg|]}d|qS).r).0fix_name)pkg_namerr ks z+get_fixers_from_package..F)r)r;r)r;rget_fixers_from_packagegsr=cCs|S)Nr)objrrr _identitynsr?rcCs|jddS)Nz  )replace)inputrrr_from_system_newlinesusrCcCs*tjdkr"|jdtjS|SdS)Nr@)rlineseprA)rBrrr_to_system_newlineswsrEc sd}tjtj|jfdd}ttjtjtj h}t }yVxO|\}}||krq`q`|tj kr|rPd}q`|tj kr|dkr|\}}|tj ks|dkrP|\}}|tj ks|dkrP|\}}|tj krJ|dkrJ|\}}x^|tj kr|j||\}}|tj ks|d krP|\}}qMWq`Pq`WWntk rYnXt|S) NFcst}|d|dfS)Nrr)next)tok)genrradvances z(_detect_future_features..advanceTfrom __future__import(,)rgenerate_tokensioStringIOreadline frozensetrNEWLINENLCOMMENTr'STRINGNAMEOPadd StopIteration)sourceZhave_docstringrIignorefeaturestpvaluer)rHr_detect_future_featuressD      rac@seZdZdZdS) FixerErrorzA fixer could not be loaded.N)r!r"r#__doc__rrrrrbs rbc@sieZdZddddiZdZdZddddZd d Zd d Zd dZ ddZ ddZ ddddZ ddddZ ddZddddZddZdddZdd Zd!d"Zdddd#d$Zdd%d&Zd'Zd(Zd)d*Zd+d,Zd-d.Zd/d0Zd1d2Zd3d4ZdS)5RefactoringToolprint_functionFwrite_unchanged_filesZFixrNcCs||_|pg|_|jj|_|dk rF|jj||jdrbtj|_n tj |_|jj d|_ g|_ t jd|_g|_d|_tj|jdtjd|j|_|j\|_|_g|_tj|_g|_g|_xzt|j|jD]c}|j rQ|jj!|q/||jkrs|jj"|q/||jkr/|jj"|q/Wt#|j|_$t#|j|_%dS)zInitializer. Args: fixer_names: a list of fixers to import options: a dict with configuration. explicit: a list of fixers to run even if they are explicit. NrerfrdFconvertlogger)&fixersexplicit_default_optionscopyoptionsr(r !python_grammar_no_print_statementgrammarr1getrferrorsloggingZ getLoggerrh fixer_logwroterZDriverr rg get_fixers pre_order post_orderfilesbmZ BottomMatcherBMZ bmi_pre_orderZbmi_post_orderrZ BM_compatibleZ add_fixerrr7bmi_pre_order_headsbmi_post_order_heads)selfZ fixer_namesrmrjr6rrr__init__s<            zRefactoringTool.__init__c Csg}g}x|jD]}t|iidg}|jddd}|j|jru|t|jd}|jd}|jdjdd|D}yt ||}Wn(t k rt d ||fYnX||j |j } | jr4|jd k r4||jkr4|jd |q|jd || jd krc|j| q| jdkr|j| qt d| jqWtjd} |jd| |jd| ||fS)aInspects the options to load the requested patterns and handlers. Returns: (pre_order, post_order), where pre_order is the list of fixers that want a pre-order AST traversal, and post_order is the list that want post-order traversal. r r8rN_cSsg|]}|jqSr)title)r9r+rrrr<s z.RefactoringTool.get_fixers..zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sZpreZpostzIllegal fixer order: %rZ run_orderkey)rirrsplitr FILE_PREFIXlensplit CLASS_PREFIXjoingetattrAttributeErrorrbrmrsrj log_message log_debugorderroperator attrgettersort) r}Zpre_order_fixersZpost_order_fixersZ fix_mod_pathmodr:partsZ class_nameZ fix_classr6Zkey_funcrrrrus8# zRefactoringTool.get_fixerscOsdS)zCalled when an error occurs.Nr)r}msgargskwdsrrr log_errorszRefactoringTool.log_errorcGs$|r||}|jj|dS)zHook to log a message.N)rhinfo)r}rrrrrrs zRefactoringTool.log_messagecGs$|r||}|jj|dS)N)rhdebug)r}rrrrrrs zRefactoringTool.log_debugcCsdS)zTCalled with the old version, new version, and filename of a refactored file.Nr)r}old_textnew_textfilenameequalrrr print_output!szRefactoringTool.print_outputcCsPxI|D]A}tjj|r5|j|||q|j|||qWdS)z)Refactor a list of files and directories.N)rrisdir refactor_dir refactor_file)r}itemswrite doctests_onlyZ dir_or_filerrrrefactor&s zRefactoringTool.refactorc Cstjd}xtj|D]\}}}|jd||j|jxb|D]Z}|jd rWtjj|d|krWtjj||} |j | ||qWWdd|D|dd.N) rextsepwalkrrrrsplitextrr) r}Zdir_namerrZpy_extdirpathZdirnames filenamesrfullnamerrrr/s    zRefactoringTool.refactor_dircCsyt|d}Wn<tk rQ}z|jd||dSWYdd}~XnXztj|jd}Wd|jXt|dd|}t|j |fSWdQRXdS)zG Do our best to decode a Python source file correctly. rbzCan't open %s: %sNrr*encoding)NN) openOSErrorrrdetect_encodingrRclose_open_with_encodingrCread)r}rferrrrrr_read_python_sourceCs z#RefactoringTool._read_python_sourcecCs|j|\}}|dkr%dS|d7}|r|jd||j||}|jsl||kr|j|||||q|jd|ni|j||}|js|r|jr|jt|dd |d|d|n|jd|dS) zRefactors a file.Nr@zRefactoring doctests in %szNo doctest changes in %srrrzNo changes in %sr)rrrefactor_docstringrfprocessed_filerefactor_string was_changedstr)r}rrrrBroutputtreerrrrSs  zRefactoringTool.refactor_filecCst|}d|kr'tj|j_zby|jj|}WnEtk r}z%|jd||jj |dSWYdd}~XnXWd|j|j_X||_ |j d||j |||S)aFRefactor a given input string. Args: data: a string holding the code to be refactored. name: a human-readable name for use in error/log messages. Returns: An AST corresponding to the refactored input stream; None if there were errors during the parse. rezCan't parse %s: %s: %sNzRefactoring %s) rar rnrroZ parse_stringr)r __class__r!future_featuresr refactor_tree)r}datarr^rrrrrrjs    zRefactoringTool.refactor_stringcCstjj}|ro|jd|j|d}|jsI||kr_|j|d|q|jdnS|j|d}|js|r|jr|jt |d|n |jddS)NzRefactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin) sysstdinrrrrfrrrr)r}rrBrrrrrrefactor_stdins zRefactoringTool.refactor_stdinc Csx-t|j|jD]}|j||qW|j|j|j|j|j|j|jj|j }xt |j rLx|jj D]}||kr||r||j dtjjdd|jr||j dtjjxGt||D]5}|||kr3||j|yt|Wntk rXw YnX|jrt||jkrtq |j|}|r |j||}|dk r |j|x6|jD](}|jsg|_|jj|qW|jj|j }x9|D]1} | |kr(g|| <|| j|| q Wq WqWq}Wx-t|j|jD]}|j||qcW|jS)aRefactors a parse tree (modifying the tree in place). For compatible patterns the bottom matcher module is used. Otherwise the tree is traversed node-to-node for matches. Args: tree: a pytree.Node instance representing the root of the tree to be refactored. name: a human-readable name for this tree. Returns: True if the tree was modified, False otherwise. rreverseTN)rrvrwZ start_tree traverse_byr{r|rzZrunZleavesanyr2rirr ZBaseZdepthZkeep_line_orderZ get_linenor/remover ValueErrorZfixers_appliedmatch transformrArr4Z finish_treer) r}rrr6Z match_setnoderesultsnewZ new_matchesZfxrrrrrsJ          %zRefactoringTool.refactor_treecCs}|s dSxl|D]d}x[||jD]L}|j|}|r%|j||}|dk r%|j||}q%WqWdS)aTraverse an AST, applying a set of fixers to each node. This is a helper method for refactor_tree(). Args: fixers: a list of fixer instances. traversal: a generator that yields AST nodes. Returns: None N)r%rrrA)r}riZ traversalrr6rrrrrrs    zRefactoringTool.traverse_bycCs|jj||dkr?|j|d}|dkr?dS||k}|j|||||r|jd||jsdS|r|j||||n|jd|dS)zR Called when a file has been refactored and there may be changes. NrzNo changes to %szNot writing changes to %s)rxrrrrrf write_file)r}rrrrrrrrrrs    zRefactoringTool.processed_filec%Csyt|dd|}Wn<tk rW}z|jd||dSWYdd}~XnXzVy|jt|Wn8tk r}z|jd||WYdd}~XnXWd|jX|jd|d|_dS)zWrites a string to a file. It first shows a unified diff between the old text and the new text, and then rewrites the file; the latter is only done if the write option is set. wrzCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)rrrrrErrrt)r}rrrrrrrrrrs* zRefactoringTool.write_filez>>> z... c Csg}d}d}d}d}x.|jddD]}|d7}|jj|jr|dk r|j|j|||||}|g}|j|j} |d| }q1|dk r|j||js|||jjdkr|j |q1|dk r2|j|j||||d}d}|j |q1W|dk rz|j|j||||dj |S)aRefactors a docstring, looking for doctests. This returns a modified version of the input string. It looks for doctests, which start with a ">>>" prompt, and may be continued with "..." prompts, as long as the "..." is indented the same as the ">>>". (Unfortunately we can't use the doctest module's parser, since, like most parsers, it is not geared towards preserving the original source.) NrkeependsTrr@r) splitlineslstriprPS1r4refactor_doctestfindPS2rstriprr) r}rBrresultblockZ block_linenoindentlinenolineirrrr(s:          z"RefactoringTool.refactor_docstringc s^yj||}Wntk r}zgjjtjrmx'|D]}jd|jdqJWjd|||j j ||SWYdd}~XnXj ||rZt |j dd}|d|d||dd} }|d jds|d d7<j|jdg}|rZ|fd d |D7}|S) zRefactors one doctest. A doctest is given as a block of lines, the first of which starts with ">>>" (possibly indented), while the remaining lines start with "..." (identically indented). z Source: %sr@z+Can't parse docstring in %s line %s: %s: %sNrTrrcs!g|]}j|qSr)r)r9r)rr}rrr<ms z4RefactoringTool.refactor_doctest..rr) parse_blockr)rhZ isEnabledForrrDEBUGrrrrr!rrrrrpop) r}rrrrrrrrZclippedr)rr}rrSs$  ) z RefactoringTool.refactor_doctestcCs|jrd}nd}|js4|jd|n1|jd|x|jD]}|j|qNW|jr|jdx|jD]}|j|qW|jrt|jdkr|jdn|jdt|jx-|jD]"\}}}|j|||qWdS) Nwerez need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:rzThere was 1 error:zThere were %d errors:)rtrxrrsrqr)r}rfilemessagerrrrrr summarizeps$      zRefactoringTool.summarizecCs1|jj|j|||}t|_|S)zParses a block into a tree. This is necessary to get correct line number / offset information in the parser diagnostics and embedded into the parse tree. )rZ parse_tokens wrap_toksrSr)r}rrrrrrrrs! zRefactoringTool.parse_blockc cstj|j||j}xe|D]]\}}\}}\} } } ||d7}| |d7} ||||f| | f| fVq%WdS)z;Wraps a tokenize stream to systematically modify start/end.rN)rrO gen_lines__next__) r}rrrr3r%r`Zline0Zcol0Zline1Zcol1Z line_textrrrrs (zRefactoringTool.wrap_toksccs||j}||j}|}xo|D]g}|j|rT|t|dVn4||jdkrrdVntd||f|}q'Wx dVqWdS)zGenerates lines as expected by tokenize from a list of lines. This strips the first len(indent + self.PS1) characters off each line. Nr@zline=%r, prefix=%rr)rrrrrAssertionError)r}rrZprefix1Zprefix2prefixrrrrrs    zRefactoringTool.gen_lines)r!r"r#rkrrr~rurrrrrrrrrrrrrrrrrrrrrrrrrrrds:  4 (        O  +   rdc@seZdZdS)MultiprocessingUnsupportedN)r!r"r#rrrrrs rcsaeZdZfddZdddfddZfddZfd d ZS) MultiprocessRefactoringToolcs/tt|j||d|_d|_dS)N)superrr~queue output_lock)r}rkwargs)rrrr~s z$MultiprocessRefactoringTool.__init__FrcsU|dkr(ttj|||SyddlWntk rRtYnXjdk rntdj_j _ fddt |D}z;x|D]}|j qWttj|||Wdjj x$t |D]}jjdqWx$|D]}|jr'|j q'Wd_XdS)Nrrz already doing multiple processescs%g|]}jdjqS)target)ZProcess_child)r9r)multiprocessingr}rrr<s z8MultiprocessRefactoringTool.refactor..)rrrr ImportErrorrr RuntimeErrorZ JoinableQueueZLockrrangestartrputZis_alive)r}rrrZ num_processesZ processesr+r)r)rr}rrs2          z$MultiprocessRefactoringTool.refactorc so|jj}xY|dk rj|\}}ztt|j||Wd|jjX|jj}qWdS)N)rrprrrZ task_done)r}Ztaskrr)rrrrs z"MultiprocessRefactoringTool._childcsE|jdk r(|jj||fntt|j||SdS)N)rrrrr)r}rr)rrrrsz)MultiprocessRefactoringTool.refactor_file)r!r"r#r~rrrrr)rrrs  r),rcrKr __author__rrrrrr-rP itertoolsrZpgen2rrrZ fixer_utilrrr r r Zbur ryrr)r r&r7r=r? version_infocodecsrrrCrErarbobjectrdrrrrrr sF               (