Downloader¶
- class parfive.Downloader(max_conn=5, progress=True, file_progress=True, loop=None, notebook=None, overwrite=False, headers=None, use_aiofiles=False)[source]¶
Bases:
object
Download files in parallel.
- Parameters
max_conn (
int
, optional) – The number of parallel download slots.progress (
bool
, optional) – IfTrue
show a main progress bar showing how many of the total files have been downloaded. IfFalse
, no progress bars will be shown at all.file_progress (
bool
, optional) – IfTrue
andprogress
is true, showmax_conn
progress bars detailing the progress of each individual file being downloaded.loop (
asyncio.AbstractEventLoop
, optional) – No longer used, and will be removed in a future release.notebook (
bool
, optional) – IfTrue
tqdm will be used in notebook mode. IfNone
an attempt will be made to detect the notebook and guess which progress bar to use.overwrite (
bool
orstr
, optional) – Determine how to handle downloading if a file already exists with the same name. IfFalse
the file download will be skipped and the path returned to the existing file, ifTrue
the file will be downloaded and the existing file will be overwritten, if'unique'
the filename will be modified to be unique.headers (
dict
) – Request headers to be passed to the server. AddsUser-Agent
information aboutparfive
,aiohttp
andpython
if not passed explicitely.
Attributes Summary
aiofiles requires a different default chunk size
The total number of files already queued for download.
aiofiles will be used if installed and must be explicitly enabled
Methods Summary
download
([timeouts])Download all files in the queue.
enqueue_file
(url[, path, filename, overwrite])Add a file to the download queue.
retry
(results)Retry any failed downloads in a results object.
run_download
([timeouts])Download all files in the queue.
simple_download
(urls, *[, path, overwrite])Download a series of URLs to a single destination.
Attributes Documentation
- default_chunk_size¶
aiofiles requires a different default chunk size
- queued_downloads¶
The total number of files already queued for download.
- use_aiofiles¶
aiofiles will be used if installed and must be explicitly enabled
PARFIVE_OVERWRITE_ENABLE_AIOFILES takes precedence if present, aiofiles will not be used
finally the Downloader’s constructor argument is considered.
Methods Documentation
- download(timeouts=None)[source]¶
Download all files in the queue.
- Parameters
timeouts (
dict
, optional) – Overrides for the default timeouts for http downloads. Supported keys are any accepted by theaiohttp.ClientTimeout
class. Defaults to no timeout for total session timeout (overriding the aiohttp 5 minute default) and 90 seconds for socket read timeout.- Returns
parfive.Results
– A list of files downloaded.
Notes
This is a synchronous version of
run_download
, anasyncio
event loop will be created to run the download (in it’s own thread if a loop is already running).The defaults for the
'total'
and'sock_read'
timeouts can be overridden by two environment variablesPARFIVE_TOTAL_TIMEOUT
andPARFIVE_SOCK_READ_TIMEOUT
.
- enqueue_file(url, path=None, filename=None, overwrite=None, **kwargs)[source]¶
Add a file to the download queue.
- Parameters
url (
str
) – The URL to retrieve.path (
str
, optional) – The directory to retrieve the file into, ifNone
defaults to the current directory.filename (
str
orcallable
, optional) – The filename to save the file as. Can also be a callable which takes two arguments the url and the response object from opening that URL, and returns the filename. (Note, for FTP downloads the response will beNone
.) IfNone
the HTTP headers will be read for the filename, or the last segment of the URL will be used.overwrite (
bool
orstr
, optional) – Determine how to handle downloading if a file already exists with the same name. IfFalse
the file download will be skipped and the path returned to the existing file, ifTrue
the file will be downloaded and the existing file will be overwritten, if'unique'
the filename will be modified to be unique. IfNone
the value set when constructing theDownloader
object will be used.kwargs (
dict
) – Extra keyword arguments are passed toaiohttp.ClientSession.get
oraioftp.Client.context
depending on the protocol.
Notes
Proxy URL is read from the environment variables
HTTP_PROXY
orHTTPS_PROXY
, depending on the protocol of theurl
passed. Proxy Authenticationproxy_auth
should be passed as aaiohttp.BasicAuth
object. Proxy Headersproxy_headers
should be passed asdict
object.
- retry(results)[source]¶
Retry any failed downloads in a results object.
Note
This will start a new event loop.
- Parameters
results (
parfive.Results
) – A previous results object, the.errors
property will be read and the downloads retried.- Returns
parfive.Results
– A modified version of the inputresults
with all the errors from this download attempt and any new files appended to the list of file paths.
- async run_download(timeouts=None)[source]¶
Download all files in the queue.
- Parameters
timeouts (
dict
, optional) – Overrides for the default timeouts for http downloads. Supported keys are any accepted by theaiohttp.ClientTimeout
class. Defaults to no timeout for total session timeout (overriding the aiohttp 5 minute default) and 90 seconds for socket read timeout.- Returns
parfive.Results
– A list of files downloaded.
Notes
The defaults for the
'total'
and'sock_read'
timeouts can be overridden by two environment variablesPARFIVE_TOTAL_TIMEOUT
andPARFIVE_SOCK_READ_TIMEOUT
.
- classmethod simple_download(urls, *, path='./', overwrite=None)[source]¶
Download a series of URLs to a single destination.
- Parameters
urls (iterable) – A sequence of URLs to download.
path (
pathlib.Path
, optional) – The destination directory for the downloaded files. Defaults to the current directory.overwrite (
bool
, optional) – Overwrite the files at the destination directory. IfFalse
the URL will not be downloaded if a file with the corresponding filename already exists.
- Returns
parfive.Results
– A list of files downloaded.