Descriptor Remote
*****************

Module for remotely retrieving descriptors from directory authorities
and mirrors. This is the simplest method for getting current tor
descriptor information…

   import stem.descriptor.remote

   for desc in stem.descriptor.remote.get_server_descriptors():
     if desc.exit_policy.is_exiting_allowed():
       print('  %s (%s)' % (desc.nickname, desc.fingerprint))

More custom downloading behavior can be done through the
"DescriptorDownloader" class, which issues "Query" instances to get
you descriptor content. For example…

   from stem.descriptor.remote import DescriptorDownloader

   downloader = DescriptorDownloader(
     use_mirrors = True,
     timeout = 10,
   )

   query = downloader.get_server_descriptors()

   print('Exit Relays:')

   try:
     for desc in query.run():
       if desc.exit_policy.is_exiting_allowed():
         print('  %s (%s)' % (desc.nickname, desc.fingerprint))

     print
     print('Query took %0.2f seconds' % query.runtime)
   except Exception as exc:
     print('Unable to retrieve the server descriptors: %s' % exc)

   get_instance - Provides a singleton DescriptorDownloader used for...
     |- their_server_descriptor - provides the server descriptor of the relay we download from
     |- get_server_descriptors - provides present server descriptors
     |- get_extrainfo_descriptors - provides present extrainfo descriptors
     |- get_microdescriptors - provides present microdescriptors with the given digests
     |- get_consensus - provides the present consensus or router status entries
     |- get_bandwidth_file - provides bandwidth heuristics used to make the next consensus
     +- get_detached_signatures - authority signatures used to make the next consensus

   Query - Asynchronous request to download tor descriptors
     |- start - issues the query if it isn't already running
     +- run - blocks until the request is finished and provides the results

   DescriptorDownloader - Configurable class for issuing queries
     |- use_directory_mirrors - use directory mirrors to download future descriptors
     |- their_server_descriptor - provides the server descriptor of the relay we download from
     |- get_server_descriptors - provides present server descriptors
     |- get_extrainfo_descriptors - provides present extrainfo descriptors
     |- get_microdescriptors - provides present microdescriptors with the given digests
     |- get_consensus - provides the present consensus or router status entries
     |- get_vote - provides an authority's vote for the next consensus
     |- get_key_certificates - provides present authority key certificates
     |- get_bandwidth_file - provides bandwidth heuristics used to make the next consensus
     |- get_detached_signatures - authority signatures used to make the next consensus
     +- query - request an arbitrary descriptor resource

New in version 1.1.0.

stem.descriptor.remote.MAX_FINGERPRINTS

   Maximum number of descriptors that can requested at a time by their
   fingerprints.

stem.descriptor.remote.MAX_MICRODESCRIPTOR_HASHES

   Maximum number of microdescriptors that can requested at a time by
   their hashes.

stem.descriptor.remote.Compression(enum)

   Compression when downloading descriptors.

   New in version 1.7.0.

   +-----------------+-------------------------------------------------------------------------------------------------------------------------------------------+
   | Compression     | Description                                                                                                                               |
   +=================+===========================================================================================================================================+
   | **PLAINTEXT**   | Uncompressed data.                                                                                                                        |
   +-----------------+-------------------------------------------------------------------------------------------------------------------------------------------+
   | **GZIP**        | GZip compression.                                                                                                                         |
   +-----------------+-------------------------------------------------------------------------------------------------------------------------------------------+
   | **ZSTD**        | Zstandard compression, this requires the zstandard module.                                                                                |
   +-----------------+-------------------------------------------------------------------------------------------------------------------------------------------+
   | **LZMA**        | LZMA compression, this requires the ‘lzma module <https://docs.python.org/3/library/lzma.html>`_.                                         |
   +-----------------+-------------------------------------------------------------------------------------------------------------------------------------------+

stem.descriptor.remote.get_instance()

   Provides the singleton "DescriptorDownloader" used for this
   module’s shorthand functions.

   New in version 1.5.0.

   Returns:
      singleton "DescriptorDownloader" instance

stem.descriptor.remote.their_server_descriptor(**query_args)

   Provides the server descriptor of the relay we’re downloading from.

   New in version 1.7.0.

   Parameters:
      **query_args** – additional arguments for the "Query"
      constructor

   Returns:
      "Query" for the server descriptors

stem.descriptor.remote.get_server_descriptors(fingerprints=None, **query_args)

   Shorthand for "get_server_descriptors()" on our singleton instance.

   New in version 1.5.0.

stem.descriptor.remote.get_extrainfo_descriptors(fingerprints=None, **query_args)

   Shorthand for "get_extrainfo_descriptors()" on our singleton
   instance.

   New in version 1.5.0.

stem.descriptor.remote.get_microdescriptors(hashes, **query_args)

   Shorthand for "get_microdescriptors()" on our singleton instance.

   New in version 1.8.0.

stem.descriptor.remote.get_consensus(authority_v3ident=None, microdescriptor=False, **query_args)

   Shorthand for "get_consensus()" on our singleton instance.

   New in version 1.5.0.

stem.descriptor.remote.get_bandwidth_file(**query_args)

   Shorthand for "get_bandwidth_file()" on our singleton instance.

   New in version 1.8.0.

stem.descriptor.remote.get_detached_signatures(**query_args)

   Shorthand for "get_detached_signatures()" on our singleton
   instance.

   New in version 1.8.0.

class stem.descriptor.remote.Query(resource, descriptor_type=None, endpoints=None, compression=('gzip', ), retries=2, fall_back_to_authority=False, timeout=None, start=True, block=False, validate=False, document_handler='ENTRIES', **kwargs)

   Bases: "object"

   Asynchronous request for descriptor content from a directory
   authority or mirror. These can either be made through the
   "DescriptorDownloader" or directly for more advanced usage.

   To block on the response and get results either call "run()" or
   iterate over the Query. The "run()" method pass along any errors
   that arise…

      from stem.descriptor.remote import Query

      query = Query(
        '/tor/server/all',
        timeout = 30,
      )

      print('Current relays:')

      try:
        for desc in Query('/tor/server/all', 'server-descriptor 1.0').run():
          print(desc.fingerprint)
      except Exception as exc:
        print('Unable to retrieve the server descriptors: %s' % exc)

   … while iterating fails silently…

      print('Current relays:')

      for desc in Query('/tor/server/all', 'server-descriptor 1.0'):
        print(desc.fingerprint)

   In either case exceptions are available via our ‘error’ attribute.

   Tor provides quite a few different descriptor resources via its
   directory protocol (see section 4.2 and later of the dir-spec).
   Commonly useful ones include…

   +-------------------------------------------------+-------------------------------------------------------+
   | Resource                                        | Description                                           |
   +=================================================+=======================================================+
   | /tor/server/all                                 | all present server descriptors                        |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/server/fp/<fp1>+<fp2>+<fp3>                | server descriptors with the given fingerprints        |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/extra/all                                  | all present extrainfo descriptors                     |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/extra/fp/<fp1>+<fp2>+<fp3>                 | extrainfo descriptors with the given fingerprints     |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/micro/d/<hash1>-<hash2>                    | microdescriptors with the given hashes                |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/status-vote/current/consensus              | present consensus                                     |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/status-vote/current/consensus-microdesc    | present microdescriptor consensus                     |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/status-vote/next/bandwidth                 | bandwidth authority heuristics for the next consenus  |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/status-vote/next/consensus-signatures      | detached signature, used for making the next consenus |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/keys/all                                   | key certificates for the authorities                  |
   +-------------------------------------------------+-------------------------------------------------------+
   | /tor/keys/fp/<v3ident1>+<v3ident2>              | key certificates for specific authorities             |
   +-------------------------------------------------+-------------------------------------------------------+

   **ZSTD** compression requires zstandard, and **LZMA** requires the
   lzma module.

   For legacy reasons if our resource has a ‘.z’ suffix then our
   **compression** argument is overwritten with Compression.GZIP.

   Changed in version 1.7.0: Added support for downloading from
   ORPorts.

   Changed in version 1.7.0: Added the compression argument.

   Changed in version 1.7.0: Added the reply_headers attribute.The
   class this provides changed between Python versions. In python2
   this was called httplib.HTTPMessage, whereas in python3 the class
   was renamed to http.client.HTTPMessage.

   Changed in version 1.7.0: Endpoints are now expected to be
   "DirPort" or "ORPort" instances. Usage of tuples for this argument
   is deprecated and will be removed in the future.

   Changed in version 1.7.0: Avoid downloading from tor26. This
   directory authority throttles its DirPort to such an extent that
   requests either time out or take on the order of minutes.

   Changed in version 1.7.0: Avoid downloading from Bifroest. This is
   the bridge authority so it doesn’t vote in the consensus, and
   apparently times out frequently.

   Changed in version 1.8.0: Serge has replaced Bifroest as our bridge
   authority. Avoiding descriptor downloads from it instead.

   Changed in version 1.8.0: Defaulting to gzip compression rather
   than plaintext downloads.

   Changed in version 1.8.0: Using "Compression" for our compression
   argument, usage of strings or this module’s Compression enum is
   deprecated and will be removed in stem 2.x.

   Variables:
      * **resource** (*str*) – resource being fetched, such as
        ‘/tor/server/all’

      * **descriptor_type** (*str*) – type of descriptors being
        fetched (for options see "parse_file()"), this is guessed from
        the resource if **None**

      * **endpoints** (*list*) – "DirPort" or "ORPort" of the
        authority or mirror we’re querying, this uses authorities if
        undefined

      * **compression** (*list*) – list of
        "stem.descriptor.Compression" we’re willing to accept, when
        none are mutually supported downloads fall back to
        Compression.PLAINTEXT

      * **retries** (*int*) – number of times to attempt the request
        if downloading it fails

      * **fall_back_to_authority** (*bool*) – when retrying request
        issues the last request to a directory authority if **True**

      * **content** (*str*) – downloaded descriptor content

      * **error** (*Exception*) – exception if a problem occured

      * **is_done** (*bool*) – flag that indicates if our request
        has finished

      * **start_time** (*float*) – unix timestamp when we first
        started running

      * **reply_headers** (*http.client.HTTPMessage*) – headers
        provided in the response, **None** if we haven’t yet made our
        request

      * **runtime** (*float*) – time our query took, this is
        **None** if it’s not yet finished

      * **validate** (*bool*) – checks the validity of the
        descriptor’s content if **True**, skips these checks otherwise

      * **document_handler**
        (*stem.descriptor.__init__.DocumentHandler*) – method in which
        to parse a "NetworkStatusDocument"

      * **kwargs** (*dict*) – additional arguments for the
        descriptor constructor

   Following are only applicable when downloading from a "DirPort"…

   Variables:
      * **timeout** (*float*) – duration before we’ll time out our
        request

      * **download_url** (*str*) – last url used to download the
        descriptor, this is unset until we’ve actually made a download
        attempt

   Parameters:
      * **start** (*bool*) – start making the request when
        constructed (default is **True**)

      * **block** (*bool*) – only return after the request has been
        completed, this is the same as running **query.run(True)**
        (default is **False**)

   start()

      Starts downloading the scriptors if we haven’t started already.

   run(suppress=False)

      Blocks until our request is complete then provides the
      descriptors. If we haven’t yet started our request then this
      does so.

      Parameters:
         **suppress** (*bool*) – avoids raising exceptions if **True**

      Returns:
         list for the requested "Descriptor" instances

      Raises:
         Using the iterator can fail with the following if
         **suppress** is **False**…

            * **ValueError** if the descriptor contents is malformed

            * "DownloadTimeout" if our request timed out

            * "DownloadFailed" if our request fails

class stem.descriptor.remote.DescriptorDownloader(use_mirrors=False, **default_args)

   Bases: "object"

   Configurable class that issues "Query" instances on your behalf.

   Parameters:
      * **use_mirrors** (*bool*) – downloads the present consensus
        and uses the directory mirrors to fetch future requests, this
        fails silently if the consensus cannot be downloaded

      * **default_args** – default arguments for the "Query"
        constructor

   use_directory_mirrors()

      Downloads the present consensus and configures ourselves to use
      directory mirrors, in addition to authorities.

      Returns:
         "NetworkStatusDocumentV3" from which we got the directory
         mirrors

      Raises:
         **Exception** if unable to determine the directory mirrors

   their_server_descriptor(**query_args)

      Provides the server descriptor of the relay we’re downloading
      from.

      New in version 1.7.0.

      Parameters:
         **query_args** – additional arguments for the "Query"
         constructor

      Returns:
         "Query" for the server descriptors

   get_server_descriptors(fingerprints=None, **query_args)

      Provides the server descriptors with the given fingerprints. If
      no fingerprints are provided then this returns all descriptors
      known by the relay.

      Parameters:
         * **fingerprints** (*str**,**list*) – fingerprint or list
           of fingerprints to be retrieved, gets all descriptors if
           **None**

         * **query_args** – additional arguments for the "Query"
           constructor

      Returns:
         "Query" for the server descriptors

      Raises:
         **ValueError** if we request more than 96 descriptors by
         their fingerprints (this is due to a limit on the url length
         by squid proxies).

   get_extrainfo_descriptors(fingerprints=None, **query_args)

      Provides the extrainfo descriptors with the given fingerprints.
      If no fingerprints are provided then this returns all
      descriptors in the present consensus.

      Parameters:
         * **fingerprints** (*str**,**list*) – fingerprint or list
           of fingerprints to be retrieved, gets all descriptors if
           **None**

         * **query_args** – additional arguments for the "Query"
           constructor

      Returns:
         "Query" for the extrainfo descriptors

      Raises:
         **ValueError** if we request more than 96 descriptors by
         their fingerprints (this is due to a limit on the url length
         by squid proxies).

   get_microdescriptors(hashes, **query_args)

      Provides the microdescriptors with the given hashes. To get
      these see the **microdescriptor_digest** attribute of
      "RouterStatusEntryMicroV3". Note that these are only provided
      via the **microdescriptor consensus**. For exampe…

         >>> import stem.descriptor.remote
         >>> consensus = stem.descriptor.remote.get_consensus(microdescriptor = True).run()
         >>> my_router_status_entry = list(filter(lambda desc: desc.nickname == 'caersidi', consensus))[0]
         >>> print(my_router_status_entry.microdescriptor_digest)
         IQI5X2A5p0WVN/MgwncqOaHF2f0HEGFEaxSON+uKRhU

         >>> my_microdescriptor = stem.descriptor.remote.get_microdescriptors([my_router_status_entry.microdescriptor_digest]).run()[0]
         >>> print(my_microdescriptor)
         onion-key
         -----BEGIN RSA PUBLIC KEY-----
         MIGJAoGBAOJo9yyVgG8ksEHQibqPIEbLieI6rh1EACRPiDiV21YObb+9QEHaR3Cf
         FNAzDbGhbvADLBB7EzuViL8w+eXQUOaIsJRdymh/wuUJ78bv5oEIJhthKq/Uqa4P
         wKHXSZixwAHfy8NASTX3kxu9dAHWU3Owb+4W4lR2hYM0ZpoYYkThAgMBAAE=
         -----END RSA PUBLIC KEY-----
         ntor-onion-key kWOHNd+2uBlMpcIUbbpFLiq/rry66Ep6MlwmNpwzcBg=
         id ed25519 xE/GeYImYAIB0RbzJXFL8kDLpDrj/ydCuCdvOgC4F/4

      Parameters:
         * **hashes** (*str**,**list*) – microdescriptor hash or
           list of hashes to be retrieved

         * **query_args** – additional arguments for the "Query"
           constructor

      Returns:
         "Query" for the microdescriptors

      Raises:
         **ValueError** if we request more than 92 microdescriptors by
         their hashes (this is due to a limit on the url length by
         squid proxies).

   get_consensus(authority_v3ident=None, microdescriptor=False, **query_args)

      Provides the present router status entries.

      Changed in version 1.5.0: Added the microdescriptor argument.

      Parameters:
         * **authority_v3ident** (*str*) – fingerprint of the
           authority key for which to get the consensus, see ‘v3ident’
           in tor’s config.c for the values.

         * **microdescriptor** (*bool*) – provides the
           microdescriptor consensus if **True**, standard consensus
           otherwise

         * **query_args** – additional arguments for the "Query"
           constructor

      Returns:
         "Query" for the router status entries

   get_vote(authority, **query_args)

      Provides the present vote for a given directory authority.

      Parameters:
         * **authority** (*stem.directory.Authority*) – authority
           for which to retrieve a vote for

         * **query_args** – additional arguments for the "Query"
           constructor

      Returns:
         "Query" for the router status entries

   get_key_certificates(authority_v3idents=None, **query_args)

      Provides the key certificates for authorities with the given
      fingerprints. If no fingerprints are provided then this returns
      all present key certificates.

      Parameters:
         * **authority_v3idents** (*str*) –

           fingerprint or list of fingerprints of the authority keys,
           see ‘v3ident’ in tor’s config.c for the values.

         * **query_args** – additional arguments for the "Query"
           constructor

      Returns:
         "Query" for the key certificates

      Raises:
         **ValueError** if we request more than 96 key certificates by
         their identity fingerprints (this is due to a limit on the
         url length by squid proxies).

   get_bandwidth_file(**query_args)

      Provides the bandwidth authority heuristics used to make the
      next consensus.

      New in version 1.8.0.

      Parameters:
         **query_args** – additional arguments for the "Query"
         constructor

      Returns:
         "Query" for the bandwidth authority heuristics

   get_detached_signatures(**query_args)

      Provides the detached signatures that will be used to make the
      next consensus. Please note that **these are only available
      during minutes 55-60 each hour**. If requested during minutes
      0-55 tor will not service these requests, and this will fail
      with a 404.

      For example…

         import stem.descriptor.remote

         detached_sigs = stem.descriptor.remote.get_detached_signatures().run()[0]

         for i, sig in enumerate(detached_sigs.signatures):
           print('Signature %i is from %s' % (i + 1, sig.identity))

      **When available (minutes 55-60 of the hour)**

         % python demo.py
         Signature 1 is from 0232AF901C31A04EE9848595AF9BB7620D4C5B2E
         Signature 2 is from 14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4
         Signature 3 is from 23D15D965BC35114467363C165C4F724B64B4F66
         ...

      **When unavailable (minutes 0-55 of the hour)**

         % python demo.py
         Traceback (most recent call last):
           File "demo.py", line 3, in
             detached_sigs = stem.descriptor.remote.get_detached_signatures().run()[0]
           File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 533, in run
             return list(self._run(suppress))
           File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 544, in _run
             raise self.error
         stem.DownloadFailed: Failed to download from http://154.35.175.225:80/tor/status-vote/next/consensus-signatures (HTTPError): Not found

      New in version 1.8.0.

      Parameters:
         **query_args** – additional arguments for the "Query"
         constructor

      Returns:
         "Query" for the detached signatures

   query(resource, **query_args)

      Issues a request for the given resource.

      Changed in version 1.7.0: The **fall_back_to_authority** default
      when using this method is now **False**, like the "Query" class.

      Parameters:
         * **resource** (*str*) – resource being fetched, such as
           ‘/tor/server/all’

         * **query_args** – additional arguments for the "Query"
           constructor

      Returns:
         "Query" for the descriptors

      Raises:
         **ValueError** if resource is clearly invalid or the
         descriptor type can’t be determined when ‘descriptor_type’ is
         **None**

stem.descriptor.remote.get_authorities()

   Provides cached Tor directory authority information. The directory
   information hardcoded into Tor and occasionally changes, so the
   information this provides might not necessarily match your version
   of tor.

   Deprecated since version 1.7.0: Use
   stem.directory.Authority.from_cache() instead.

   Returns:
      **dict** of **str** nicknames to "Authority" instances
