0{Yb@sddlZddlmZddlZddlZddlZddlZddlZyddlZWne k rddl ZYnXddl Z ddl m Z ddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlmZm Z m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,dd l-m.Z.m/Z/dd l0m1Z1m2Z2ej3e4Z5ej6d Z7ej6d ej8Z9ej6d Z:dZ;dddZ<GdddeZ=Gddde>Z?Gddde?Z@Gddde?ZAGddde>ZBGddde?ZCGddde?ZDGdd d e?ZEGd!d"d"e?ZFGd#d$d$e?ZGeGeEeCd%d&d'd(d)ZHeHjIZIej6d*ZJGd+d,d,e>ZKdS)-N)BytesIO)DistlibException)urljoinurlparse urlunparse url2pathname pathname2urlqueuequoteunescape string_types build_openerHTTPRedirectHandlerRequest HTTPErrorURLError) DistributionDistributionPath make_dist)Metadata)cached_propertyparse_credentials ensure_slashsplit_filenameget_project_dataparse_requirementparse_name_and_version ServerProxy) get_schemeUnsupportedVersionError)Wheel is_compatiblez^(\w+)=([a-f0-9]+)z;\s*charset\s*=\s*(.*)\s*$ztext/html|application/x(ht)?mlzhttp://python.org/pypicCs1|dkrt}nt|dd}|jS)z Return all distribution names known by an index. :param url: The URL of the index. :return: A list of all known distribution names. Ntimeoutg@) DEFAULT_INDEXr list_packages)urlclientr(7/tmp/pip-ok8sy_ze-build/pip/_vendor/distlib/locators.pyget_all_distribution_names)s  r*c@s0eZdZdZddZeZZZdS)RedirectHandlerzE A class to work around a bug in some Python 3.2.x releases. c Csd}x(dD] }||kr ||}Pq q W|dkrAdSt|}|jdkrt|j|}t|dr|j||q|||.same_projectNzegg=z %s: version hint in fragment: %rr/z.whlTr\versionror.r&z, cSs/g|]%}djt|ddqS).N)joinlist).0vr(r(r) s z8Locator.convert_url_to_download_info..zpython-versionzinvalid path for wheel: %sz No match for project/version: %sz %s_digest)NNr)rrs startswithrhri HASHER_HASHmatchgroupsendswithr!r" wheel_tagsr\rwrorrzpyver Exceptionwarningdownloadable_extensionsrcrdlenr)r5r&rprur_r0rbreparamsqueryfragmalgodigestZorigpathwheelincludeeroextrfr\rwrr(r(r)convert_url_to_download_infosj           '&   z$Locator.convert_url_to_download_infocCsEd}x8dD]0}d|}||kr |||f}Pq q W|S)z Get a digest from a dictionary by looking at keys of the form 'algo_digest'. Returns a 2-tuple (algo, digest) if found, else None. Currently looks only for SHA256, then MD5. Nsha256md5z %s_digest)zsha256zmd5r()r5infor_rr<r(r(r) _get_digests   zLocator._get_digestcCs|jd}|jd}||kr@||}|j}n!t||d|j}|j}|j||_|j|dkr|j|j|d|_n||_|||= 1.0, < 2.0, != 1.3)' :param prereleases: If ``True``, allow pre-release versions to be located. Otherwise, pre-release versions are not returned. :return: A :class:`Distribution` instance, or ``None`` if no such distribution could be located. NzNot a valid requirement: %rzmatcher: %s (%s)z%s did not match %rz%skipping pre-release version %s of %szerror matching %s with %rrr<zsorted list: %sr)rrrr0rS requirementrhrityper>r`r\ version_classr is_prereleaseappendrrrsortedr<extras) r5r prereleasesr_rr0rSversionsslistZvclskr(r(r)locate7s>         zLocator.locate)rFrGrHrIrJrK)rLrMrN)rO)rN)r>r?r@rAsource_extensionsbinary_extensionsexcluded_extensionsrrrTrVrXrZpropertyr0r]r^r`rgrnrrrrrr(r(r(r)rESs*          M  rEcs@eZdZdZfddZddZddZS)PyPIRPCLocatorz This locator uses XML-RPC to locate distributions. It therefore cannot be used with simple mirrors (that only mirror file content). c s8tt|j|||_t|dd|_dS)z Initialise an instance. :param url: The URL to use for XML-RPC. :param kwargs: Passed to the superclass constructor. r#g@N)superrrTbase_urlrr')r5r&kwargs) __class__r(r)rTms zPyPIRPCLocator.__init__cCst|jjS)zJ Return all the distribution names known to this locator. )setr'r%)r5r(r(r)r^xsz%PyPIRPCLocator.get_distribution_namesc Csi}|jj|d}x|D]}|jj||}|jj||}td|j}|d|_|d|_|jd|_ |jdg|_ |jd|_ t |}|r"|d} | d |_ |j| |_||_|||r?r@rArTr^r]r(r()rr)rhs  rcs@eZdZdZfddZddZddZS)PyPIJSONLocatorzw This locator uses PyPI's JSON interface. It's very limited in functionality nad probably not worth using. c s)tt|j|t||_dS)N)rrrTrr)r5r&r)rr(r)rTszPyPIJSONLocator.__init__cCstddS)zJ Return all the distribution names known to this locator. zNot available from this locatorN)r[)r5r(r(r)r^sz&PyPIJSONLocator.get_distribution_namesc Csbi}t|jdt|}y|jj|}|jj}tj|}t d|j }|d}|d|_ |d|_ |j d|_|j dg|_|j d|_t|}|d } | r%| d } | d |_|j| |_||_|||j r?r@rArTr^r]r(r()rr)rs  rc@seZdZdZejdejejBejBZ ejdejejBZ ddZ ejdejZ e ddZd S) Pagez4 This class represents a scraped HTML page. z (rel\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s ]*))\s+)? href\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s ]*)) (\s+rel\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s ]*)))? z!]+)cCsM||_||_|_|jj|j}|rI|jd|_ndS)zk Initialise an instance with the Unicode page contents and the URL they came from. rN)rrr&_basesearchgroup)r5rr&rr(r(r)rTs  z Page.__init__z[^a-z0-9$&+,/:;=?@.#%_\\|-]cCs dd}t}x|jj|jD]}|jd}|dpy|dpy|dpy|dpy|dpy|d }|d p|d p|d }t|j|}t|}|jj d d|}|j ||fq+Wt |ddddd}|S)z Return the URLs of all the links on a page together with information about their "rel" attribute, for determining which ones to treat as downloads and which ones to queue for further scraping. cSs@t|\}}}}}}t||t||||fS)zTidy up an URL.)rrr )r&r0rbrerrrr(r(r)cleanszPage.links..cleanr.Zrel1Zrel2Zrel3Zrel4Zrel5Zrel6rjrkZurl3cSsdt|jdS)Nz%%%2xr)ordr)rr(r(r)szPage.links..r<cSs|dS)Nrr()rfr(r(r)rsreverseT) r_hreffinditerr groupdictrrr _clean_resubaddr)r5rr_rrrelr&r(r(r)linkss   z Page.linksN)r>r?r@rArecompileISXrrrTrrrr(r(r(r)rs  rcseZdZdZiejd6ddd6ddd6Zdd fd d Zd d ZddZ ddZ e j de j ZddZddZddZddZddZe j dZddZS) SimpleScrapingLocatorz A locator which scrapes HTML pages to locate downloads for a distribution. This runs multiple threads to do the I/O; performance is at least as good as pip's PackageFinder, which works in an analogous fashion. deflatecCstjdttjS)Nfileobj)gzipGzipFilerrr)br(r(r)rszSimpleScrapingLocator.rcCs|S)Nr()rr(r(r)rsnoneN c stt|j|t||_||_i|_t|_t j |_ t|_ d|_ ||_tj|_tj|_dS)a Initialise an instance. :param url: The root URL to use for scraping. :param timeout: The timeout, in seconds, to be applied to requests. This defaults to ``None`` (no timeout specified). :param num_workers: The number of worker threads you want to do I/O, This defaults to 10. :param kwargs: Passed to the superclass. FN)rrrTrrr# _page_cacher_seenr Queue _to_fetch _bad_hostsskip_externals num_workers threadingRLock_lock_gplock)r5r&r#rr)rr(r)rTs       zSimpleScrapingLocator.__init__cCscg|_xSt|jD]B}tjd|j}|jd|j|jj|qWdS)z Threads are created only when get_project is called, and terminate before it returns. They are there primarily to parallelise I/O (i.e. fetching web pages). targetTN) _threadsrangerrThread_fetch setDaemonstartr)r5irfr(r(r)_prepare_threadss    z&SimpleScrapingLocator._prepare_threadscCsOx!|jD]}|jjdq Wx|jD]}|jq.Wg|_dS)zu Tell all the threads to terminate (by sending a sentinel value) and wait for them to do so. N)rrputrz)r5rfr(r(r) _wait_threads,s z#SimpleScrapingLocator._wait_threadscCsi}|j||_||_t|jdt|}|jj|jj|j z1t j d||j j ||j jWd|jX|`WdQX|S)Nz%s/z Queueing %s)rr_rprrr rrUrrrhrirrrzr)r5r\r_r&r(r(r)r]9s        z"SimpleScrapingLocator._get_projectz<\b(linux-(i\d86|x86_64|arm\w+)|win(32|-amd64)|macosx-?\d+)\bcCs|jj|S)zD Does an URL refer to a platform-specific download? )platform_dependentr)r5r&r(r(r)_is_platform_dependentNsz,SimpleScrapingLocator._is_platform_dependentc Csp|j|rd}n|j||j}tjd|||rl|j|j|j|WdQXn|S)a% See if an URL is a suitable download for a project. If it is, register information in the result dictionary (for _get_project) about the specific version it's for. Note that the return value isn't actually used other than as a boolean value. Nzprocess_download: %s -> %s)rrrprhrirrr_)r5r&rr(r(r)_process_downloadTs   z'SimpleScrapingLocator._process_downloadc Cst|\}}}}}}|j|j|j|jrGd}n|jrl|j|j rld}n|j|jsd}ny|d krd}nd|dkrd}nO|j|rd}n7|j ddd } | j d krd}nd }t j d |||||S)z Determine whether a link URL from a referring page and with a particular "rel" attribute should be queued for scraping. Fhomepagedownloadhttpraftp:rr localhostTz#should_queue: %s (%s) from %s -> %s)rzdownload)zhttpzhttpszftp) rrrrrrrrrsplitrsrhri) r5linkZreferrerrr0rbrerqr_hostr(r(r) _should_queuehs*           z#SimpleScrapingLocator._should_queuec Csx|jj}z|r|j|}|dkr<wnx|jD]y\}}||jkrF|jj||j| r|j|||rtj d|||jj |qqFqFWnWd|jj X|sPqqWdS)z Get a URL to fetch from the work queue, get the HTML page, examine its links for download candidates and candidates for further scraping. This is a handy method to run in a thread. NzQueueing %s from %s) rrget_pagerrrrrrhrir task_done)r5r&pagerrr(r(r)rs" !zSimpleScrapingLocator._fetchcCst|\}}}}}}|dkrZtjjt|rZtt|d}n||jkr|j|}tj d||n|j ddd}d}||j krtj d||nAt |d id d 6}zy8tj d ||j j|d |j} tj d|| j} | jdd} tj| r | j} | j} | jd}|r|j|}|| } nd}tj| }|r|jd}ny| j|} Wn!tk r| jd} YnXt| | }||j| ]*>([^<]+)r?r@rAzlib decompressrrTrrr]rrrrrrrrrrr^r(r()rr)rs$           ;rcsLeZdZdZfddZddZddZdd ZS) DirectoryLocatorz? This class locates distributions in a directory tree. c so|jdd|_tt|j|tjj|}tjj|sbt d|n||_ dS)a Initialise an instance. :param path: The root of the directory tree to search. :param kwargs: Passed to the superclass constructor, except for: * recursive - if True (the default), subdirectories are recursed into. If False, only the top-level directory is searched, recursiveTzNot a directory: %rN) rrrrrTr reabspathr rbase_dir)r5rer)rr(r)rTs zDirectoryLocator.__init__cCs|j|jS)z Should a filename be considered as a candidate for a distribution archive? As well as the filename, the directory which contains it is provided, though not used by the current implementation. )rr)r5roparentr(r(r)should_includeszDirectoryLocator.should_includec Csi}xtj|jD]\}}}x|D]}|j||r/tjj||}tddttjj|dddf}|j ||}|r|j ||qq/q/W|j sPqqW|S)Nr r.) r walkrrrerzrr rrrr) r5r\r_rootdirsfilesfnr&rr(r(r)r]s"   zDirectoryLocator._get_projectc Cst}xtj|jD]\}}}x|D]}|j||r2tjj||}tddttjj |dddf}|j |d}|r|j |dqq2q2W|j sPqqW|S)zJ Return all the distribution names known to this locator. r r.Nr\) rr rrrrerzrr rrrr)r5r_r r!r"r#r&rr(r(r)r^s "   z'DirectoryLocator.get_distribution_names)r>r?r@rArTrr]r^r(r()rr)rs   rc@s.eZdZdZddZddZdS) JSONLocatora This locator uses special extended metadata (not available on PyPI) and is the basis of performant dependency resolution in distlib. Other locators require archive downloads before dependencies can be determined! As you might imagine, that can be slow. cCstddS)zJ Return all the distribution names known to this locator. zNot available from this locatorN)r[)r5r(r(r)r^2sz"JSONLocator.get_distribution_namescCs i}t|}|r x|jdgD]}|ddks+|ddkrWq+nt|d|dd|jdd d |j}|j}|d |_d |kr|d rd |d f|_n|jdi|_|jdi|_|||j r?r@rAr^r]r(r(r(r)r$+s  r$cs4eZdZdZfddZddZS)DistPathLocatorz This locator finds installed distributions in a path. It can be useful for adding to an :class:`AggregatingLocator`. c s8tt|j|t|ts+t||_dS)zs Initialise an instance. :param distpath: A :class:`DistributionPath` instance to search. N)rr*rT isinstancerAssertionErrordistpath)r5r-r)rr(r)rTUszDistPathLocator.__init__cCs;|jj|}|dkr'i}ni||j6}|S)N)r-get_distributionrw)r5r\rr_r(r(r)r]_s   zDistPathLocator._get_project)r>r?r@rArTr]r(r()rr)r*Ps  r*csseZdZdZfddZfddZddZeej j eZ dd Z d d Z S) AggregatingLocatorzI This class allows you to chain and/or merge a list of locators. cs8|jdd|_||_tt|j|dS)a Initialise an instance. :param locators: The list of locators to search. :param kwargs: Passed to the superclass constructor, except for: * merge - if False (the default), the first successful search from any of the locators is returned. If True, the results from all locators are merged (this can be slow). mergeFN)rr0locatorsrr/rT)r5r1r)rr(r)rTls  zAggregatingLocator.__init__cs5tt|jx|jD]}|jqWdS)N)rr/rVr1)r5r)rr(r)rV|szAggregatingLocator.clear_cachecCs*||_x|jD]}||_qWdS)N)rWr1r0)r5rYrr(r(r)rZs zAggregatingLocator._set_schemecCsi}x|jD]}|j|}|r|jrD|j|q|jdkr\d}n3d}x*|D]"}|jj|rid}PqiqiW|r|}PqqqW|S)NTF)r1r`r0updaterSr)r5r\r_rrfoundrr(r(r)r]s"     zAggregatingLocator._get_projectc CsJt}x:|jD]/}y||jO}Wqtk rAYqXqW|S)zJ Return all the distribution names known to this locator. )rr1r^r[)r5r_rr(r(r)r^s   z)AggregatingLocator.get_distribution_names) r>r?r@rArTrVrZrrEr0fgetr]r^r(r()rr)r/hs   r/zhttps://pypi.python.org/simple/r#g@r0legacyz1(?P[\w-]+)\s*\(\s*(==\s*)?(?P[^)]+)\)$c@sseZdZdZdddZddZddZd d Zd d Zd dZ ddddZ dS)DependencyFinderz0 Locate dependencies for distributions. NcCs(|p t|_t|jj|_dS)zf Initialise an instance, using the specified locator to locate distributions. N)default_locatorrrr0)r5rr(r(r)rTszDependencyFinder.__init__cCstjd||j}||j|<||j||jf= 1.0) while C requires (B >= 1.1). For successful replacement, ``provider`` must meet all the requirements which ``other`` fulfills. :param provider: The provider we are trying to replace with. :param other: The provider we're trying to replace. :param problems: If False is returned, this will contain what problems prevented replacement. This is currently a tuple of the literal string 'cantreplace', ``provider``, ``other`` and the set of requirements that ``provider`` couldn't fulfill. :return: True if we can replace ``other`` with ``provider``, else False. Z cantreplaceFT) reqtsrrCrrwrrAr<r>) r5rDotherproblemsZrlist unmatchedr@rSr_r(r(r)try_to_replaces        # zDependencyFinder.try_to_replaceFcCsi|_i|_i|_i|_t|p0g}d|krk|jd|tdddgO}nt|tr|}}tj d|nK|j j |d|}}|dkrt d|ntj d |d |_ t}t|g}t|g}x|r|j}|j} | |jkrO|j|n/|j| } | |kr~|j|| |n|j|jB} |j} t} ||krxAdD]6}d|}||kr| t|d|O} qqWn| | B| B}x|D]}|j|}|s+tj d||j j |d|}|dkrv| rv|j j |dd }n|dkrtj d||jd|fq+|j|j}}||f|jkr|j|n|j||| kr+||kr+|j|tj d|jq+nxw|D]o}|j} | |jkrr|jj|tj|q2|j| } | |kr2|j|| |q2q2WqWqWt|jj}x<|D]4}||k|_|jrtj d|jqqWtj d|||fS)a Find a distribution and all distributions it depends on. :param requirement: The requirement specifying the distribution to find, or a Distribution instance. :param meta_extras: A list of meta extras such as :test:, :build: and so on. :param prereleases: If ``True``, allow pre-release versions to be returned - otherwise, don't return prereleases unless they're all that's available. Return a set of :class:`Distribution` instances and a set of problems. The distributions returned should be such that they have the :attr:`required` attribute set to ``True`` if they were from the ``requirement`` passed to ``find()``, and they have the :attr:`build_time_dependency` attribute set to ``True`` unless they are post-installation dependencies of the ``requirement``. The problems should be a tuple consisting of the string ``'unsatisfied'`` and the requirement which couldn't be satisfied by any distribution known to the locator. z:*:z:test:z:build:z:dev:zpassed %s as requirementrNzUnable to locate %rz located %sTtestbuilddevz:%s:z %s_requireszNo providers found for %rzCannot satisfy %rZ unsatisfiedzAdding %s to install_distsz#%s is a build-time dependency only.zfind done for %s)ztestzbuildzdev)r;r9r8rFrr?r+rrhrirrr requestedrr<r>rJZ run_requiresZ meta_requiresZbuild_requiresgetattrrErrwZname_and_versionr<valuesZbuild_time_dependency)r5rZ meta_extrasrrZodistrHtodoZ install_distsr\rGZireqtsZsreqtsZereqtsr<rZ all_reqtsrZ providersrDnr}r=r9r(r(r)find>s                      !       "  "   zDependencyFinder.find) r>r?r@rArTr>rArCrErJrSr(r(r(r)r6s      'r6)Lriorrloggingr rcrr ImportErrordummy_threadingrr.rcompatrrrrr r r r r rrr3rrrZdatabaserrrrrutilrrrrrrrrrwrr rr!r" getLoggerr>rhrrrrrr$r*r+objectrErrrrrr$r*r/r7rNAME_VERSION_REr6r(r(r(r)sV        ^: +*:A%N