Core async functions.

Module Contents#

class async_retriever.utils.BaseRetriever(urls, file_paths=None, read_method=None, request_kwds=None, request_method='GET', cache_name=None)#

Base class for async retriever.

static generate_requests(urls, request_kwds, file_paths)#

Generate urls and keywords.


Create a cache folder in the current working directory.

async async_retriever.utils.delete_url(url, method='GET', cache_name=None, **kwargs)#

Delete cached response associated with url.


Create an event loop.

async async_retriever.utils.retriever(uid, url, s_kwds, session, read_type, r_kwds)#

Create an async request and return the response as binary.

  • uid (int) – ID of the URL for sorting after returning the results

  • url (str) – URL to be retrieved

  • s_kwds (dict) – Arguments to be passed to requests

  • session (ClientSession) – A ClientSession for sending the request

  • read_type (str) – Return response as text, bytes, or json.

  • r_kwds (dict) – Keywords to pass to the response read function. It is {"content_type": None} if read is json else an empty dict.


bytes – The retrieved response as binary.

async async_retriever.utils.stream_session(url, s_kwds, session, filepath)#

Stream the response to a file.