wget utility is the best option to download files from internet. wget can pretty much handle all complex download situations including large file downloads, recursive downloads, non-interactive downloads, multiple file downloads etc.,
In this article let us review how to use wget for various download scenarios using 15 awesome wget examples.
1. Download Single File with wget
The following example downloads a single file from internet and stores in the current directory.
$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
While downloading it will show a progress bar with the following information:
- %age of download completion (for e.g. 31% as shown below)
- Total amount of bytes downloaded so far (for e.g. 1,213,592 bytes as shown below)
- Current download speed (for e.g. 68.2K/s as shown below)
- Remaining time to download (for e.g. eta 34 seconds as shown below)
Download in progress:
$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2 Saving to: `strx25-0.9.2.1.tar.bz2.1' 31% [=================> 1,213,592 68.2K/s eta 34s
Download completed:
$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2 Saving to: `strx25-0.9.2.1.tar.bz2' 100%[======================>] 3,852,374 76.8K/s in 55s 2009-09-25 11:15:30 (68.7 KB/s) - `strx25-0.9.2.1.tar.bz2' saved [3852374/3852374]
2. Download and Store With a Different File name Using wget -O
By default wget will pick the filename from the last word after last forward slash, which may not be appropriate always.
Wrong: Following example will download and store the file with name: download_script.php?src_id=7701
$ wget http://www.vim.org/scripts/download_script.php?src_id=7701
Even though the downloaded file is in zip format, it will get stored in the file as shown below.
$ ls download_script.php?src_id=7701
Correct: To correct this issue, we can specify the output file name using the -O option as:
$ wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701
3. Specify Download Speed / Download Rate Using wget –limit-rate
While executing the wget, by default it will try to occupy full possible bandwidth. This might not be acceptable when you are downloading huge files on production servers. So, to avoid that we can limit the download speed using the –limit-rate as shown below.
In the following example, the download speed is limited to 200k
$ wget --limit-rate=200k http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
4. Continue the Incomplete Download Using wget -c
Restart a download which got stopped in the middle using wget -c option as shown below.
$ wget -c http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
This is very helpful when you have initiated a very big file download which got interrupted in the middle. Instead of starting the whole download again, you can start the download from where it got interrupted using option -c
Note: If a download is stopped in middle, when you restart the download again without the option -c, wget will append .1 to the filename automatically as a file with the previous name already exist. If a file with .1 already exist, it will download the file with .2 at the end.
5. Download in the Background Using wget -b
For a huge download, put the download in background using wget option -b as shown below.
$ wget -b http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2 Continuing in background, pid 1984. Output will be written to `wget-log'.
It will initiate the download and gives back the shell prompt to you. You can always check the status of the download using tail -f as shown below.
$ tail -f wget-log
Saving to: `strx25-0.9.2.1.tar.bz2.4'
0K .......... .......... .......... .......... .......... 1% 65.5K 57s
50K .......... .......... .......... .......... .......... 2% 85.9K 49s
100K .......... .......... .......... .......... .......... 3% 83.3K 47s
150K .......... .......... .......... .......... .......... 5% 86.6K 45s
200K .......... .......... .......... .......... .......... 6% 33.9K 56s
250K .......... .......... .......... .......... .......... 7% 182M 46s
300K .......... .......... .......... .......... .......... 9% 57.9K 47s
Also, make sure to review our previous multitail article on how to use tail command effectively to view multiple files.
6. Mask User Agent and Display wget like Browser Using wget –user-agent
Some websites can disallow you to download its page by identifying that the user agent is not a browser. So you can mask the user agent by using –user-agent options and show wget like a browser as shown below.
$ wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" URL-TO-DOWNLOAD
7. Test Download URL Using wget –spider
When you are going to do scheduled download, you should check whether download will happen fine or not at scheduled time. To do so, copy the line exactly from the schedule, and then add –spider option to check.
$ wget --spider DOWNLOAD-URL
If the URL given is correct, it will say
$ wget --spider download-url Spider mode enabled. Check if remote file exists. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Remote file exists and could contain further links, but recursion is disabled -- not retrieving.
This ensures that the downloading will get success at the scheduled time. But when you had give a wrong URL, you will get the following error.
$ wget --spider download-url Spider mode enabled. Check if remote file exists. HTTP request sent, awaiting response... 404 Not Found Remote file does not exist -- broken link!!!
You can use the spider option under following scenarios:
- Check before scheduling a download.
- Monitoring whether a website is available or not at certain intervals.
- Check a list of pages from your bookmark, and find out which pages are still exists.
8. Increase Total Number of Retry Attempts Using wget –tries
If the internet connection has problem, and if the download file is large there is a chance of failures in the download. By default wget retries 20 times to make the download successful.
If needed, you can increase retry attempts using –tries option as shown below.
$ wget --tries=75 DOWNLOAD-URL
9. Download Multiple Files / URLs Using Wget -i
First, store all the download files or URLs in a text file as:
$ cat > download-file-list.txt URL1 URL2 URL3 URL4
Next, give the download-file-list.txt as argument to wget using -i option as shown below.
$ wget -i download-file-list.txt
10. Download a Full Website Using wget –mirror
Following is the command line which you want to execute when you want to download a full website and made available for local viewing.
$ wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL
- –mirror : turn on options suitable for mirroring.
- -p : download all files that are necessary to properly display a given HTML page.
- –convert-links : after the download, convert the links in document for local viewing.
- -P ./LOCAL-DIR : save all the files and directories to the specified directory.
11. Reject Certain File Types while Downloading Using wget –reject
You have found a website which is useful, but don’t want to download the images you can specify the following.
$ wget --reject=gif WEBSITE-TO-BE-DOWNLOADED
12. Log messages to a log file instead of stderr Using wget -o
When you wanted the log to be redirected to a log file instead of the terminal.
$ wget -o download.log DOWNLOAD-URL
13. Quit Downloading When it Exceeds Certain Size Using wget -Q
When you want to stop download when it crosses 5 MB you can use the following wget command line.
$ wget -Q5m -i FILE-WHICH-HAS-URLS
Note: This quota will not get effect when you do a download a single URL. That is irrespective of the quota size everything will get downloaded when you specify a single file. This quota is applicable only for recursive downloads.
14. Download Only Certain File Types Using wget -r -A
You can use this under following situations:
- Download all images from a website
- Download all videos from a website
- Download all PDF files from a website
$ wget -r -A.pdf http://url-to-webpage-with-pdfs/
15. FTP Download With wget
You can use wget to perform FTP download as shown below.
Anonymous FTP download using Wget
$ wget ftp-url
FTP download using wget with username and password authentication.
$ wget --ftp-user=USERNAME --ftp-password=PASSWORD DOWNLOAD-URL
If you liked this article, please bookmark it with delicious or Stumble.
My name is Ramesh Natarajan. I will be posting instruction guides, how-to, troubleshooting tips and tricks on Linux, database, hardware, security and web. My focus is to write articles that will either teach you or help you resolve a problem. Read more about
Comments on this entry are closed.
Really nice guide, it’ll sure come handy.
Have fun!
Thanks for great tips 🙂
You can use wget to follow HTTP locations as shown here.
Awesome … very helpful.
Perfect timing, lol
I was just trying to remember how to use this application to download some documentation.
Thanks !
Quite good 🙂
very good, thanks.
Thanks for great information,
But I have one question,
Are we net to set any configure file before using wget ?
@bubblefish, kyanh, MihirJ, Orlin, Jaeho,
Thanks a lot for your comments. I’m glad you found this article helpful.
@runlevel0,
We can read your mind and post articles accordingly. 🙂 Just kidding.
@beparas,
There is no configuration file for wget. Once you’ve installed wget, just start using it using one of the examples mentioned in this article.
how to download whole directory, sub directory with wget.
@shashank: try -r and –level options
thanks for sharing..
I have used wget regularly for a long time, but never realized (or considered) that wget had command line options. Thanks for the great tutorial!
for ((i=10; i<= 99; i++ ))
do
wget http://url/some_picture_$i.jpg
done
if som1 needs wild cards just an idea how to do it if one is limited and cant use curls [from-to] range
Good sathiya.. even i referred this article for the -P option for downloading only one particular URL path to a directory..
good work.. keep it up.
for recursive download of index.html to only one directory level , with all required files, you can use this
wget -nc -nd -nH -l2 –convert-links -r /index.html
Great examples! Thanks!!
Q. Can wget download part of a site for offline viewing?
I like to download html documentation for offline use but the links are always wrong for local viewing. i end up creating regular expressions and using the substitute command in vim to update the html files to work locally.
I have tried wget’s –mirror option but it downloads the entire website, rather than just the few pages I want.
Is wget the simple way to get this task done, and if so, how?
Thanks in advance!
Just a note, I needed to put the web address in double quotes for this to work on my linux box.
O Man… you saved my time 🙂
Thanks you so much …
Thanks, Ramesh… the only thing lacking is a link to wget’s download site. 🙂
i’m sure make a great thankz for this tutor 🙂
hehe windows addict but wont to learning widget at linux to make mirror website 😀
xiix thankz broo
Is there any way to invoke wget programmatically in C/C++ without using system(). Which library contains this. If I get any idea, then I could possibly dyamically load the library for the same.
If someone can give hints on the same, it will be of great help.
Thanks in advance,
Chitra
thanks
its usefull
How do I download a file with a “%20” space in the URL?
For example http://dn.mozilla.com/firefox 1.5.1 32bit.exe or http://dn.mozilla.com/firefox%201.5.1%2032bit.exe ?
If wget can not do this and there is another command line tool which can do this please let me know.
Regards & TIA
How to download an excel file using wget in unix?Kinldy help.
Very Useful Tips!
Keep it up.
You solved my problem! I am going to bookmark this page. Thanks.
how can i get all files with the same file extension in a specific folder using wget?
for example I want to get all .ZIP files on domain.com/files
please help me.
to jehzlau
use wget -P domain.com/files -A zip url.downloads.com
fabulous tutorial for a beginner
That -c option to wget is the best!!, it helps to continue the download from where it left off before power failure.
Awesome article again 😉
-nc
–no-clobber
is a great option if you are using the wget -i , you can keep appending the contents of the file with newer URLs yet it wont download/overwrite the files that are already downloaded 😉
Thank you for your tips!
Thanks for the great tips! Very useful. Looking to direct my downloaded files to a specific directory – perhaps by file extension. TIA.
9. Download Multiple Files / URLs Using Wget -i
tell me what you do to rename these files?
This is very handy information. Awesome job! Thanks for sharing!
I use a fairly simple WGET command to verify if the IP Address and port are open or closed for a particular server (wget -a /tmp/ports.log -t1 -T1 10.178.30.45:443).
The issue I have is that there are a number of servers that I need to check, and each server links to other IP addresses/ports.
Currently I have several of these one liner type scripts deployed on each of the specific servers which require being run manually as and when required.
Ideally, I am looking to customize this by hopefully creating one script that can recursively read in perhaps a flat file of IP addresses/ports against a WGET command and report on only those that are not connected (ie: “failed: Connection timed out.”). Results to be written out to an output file.
Is it possible to create this single script in such a way that it can be run from a single location (like windows) rather than deploying the script and running it in each of the servers?
Thanks,
Evan.
If you have a download link (e.g.download_script.php?src_id=7701) but do know the extension of the file being provided (it could be zip, rar, dmg, gz, etc..), how do you know what to call the file in your -O argument?
Is there any way to get wget to resolve the file extensions automatically?
Where the is saved when downloaded using command “wget” ?????
2. Download and Store With a Different File name Using wget -O
How can I store with a different file name dynamiquely
exemple:
I wanna add the date to the name of file
wget -O File_%DATE%.txt ftp://username:pass@Host/folder/File.txt
that doesn’t work 🙁
P.S: I use wget on windows.
Thanks for your help
Regards
I found a solution of my prob, it’s only missed a “”:
wget -O “File_%DATE%”.txt ftp://username:pass@Host/folder/File.txt
I am using the wget command to download a file. But, the speed is too slow its happening B/S. Can we increase it to KB/S?. Is there any network settings I need to do in order to increase the speed?
To Arun: you can limite the speed (–limit-rate=value k) but you can’t increase it.
It depend with the connexion that you use.
Regards.
Nice article but there is one more interesting link include the things given here also
A cool trick. If you come across a site with download links using a PHP redirect that won’t work with WGET (you get an html file named *.php* instead of what you want) What you can do is use WGET to mirror the page with the links.
wget –mirror -p -A.php -P ./LOCAL-DIR WEBSITE-URL
It will start downloading all of the PHP on the page including the files behind the PHP redirects. Now what you can do is stop with CTRL+C once it starts downloading one of those files behind the PHP redirect. Above the Progress bar you’ll see a URL next to a Date stamp and above “Resolving [some address here]… [some IP]”, That’s the real location of that file. Using that you can now figure out the actual location of the file you want.
You could also just let WGET keep running until it’s downloaded all the files and just find the one you want from the dump, but depending on how many Download links there are, you could end up with a lot of really large files. If you want to do this I reccomend making sure the Download Loctaion has plenty of Free Space.
I don’t understand so much how
wget -r –dont-remove-listing ftp://username:pass@Host/folder works !!
I know that creat a list to check the ftp files BUT if there’s a new file on the ftp folder, i can’t download it with this methode.
-nc helps to download only the new files but it take much time if there is a lot of folders.
as there any other way please ?
Regards
Hello,
Is there a way to mirror a static copy of a PHP page that requires login and password authentication?
I tried this (using a cookies file), but had no luck:
Something else that I need to know pertaining to the syntax here…?
Thanks a lot!
Sirisha
What about downloading a file and saving it to a certain directory?
Thanks, I’ve been looking download with list file, and finally found it here
Very nice..! How do I specify which directory I am downloading files to?
you just need to be in the directory.
exemple:
cd c:\test
wget …
the files will be saved in the folder test
Regards
wget -r -A.zip -PE:\test ftp://username:password@ftp.xyz.com/directory/subdirectory
will download all zip files located in a certain subdirectory at the ftp server of xyz.com into the test folder on your E-drive.
Cheers
how can i download any thing in A specific path ??
this question is already get ansered
you just need to be in the directory.
exemple with wget in windows:
cd /var/www/html/
wget …
Regards
Great information!
Anyone able to use Wget to get past Digest Authentication and download the web page?
Hi,
Thanks a lot, very useful article
great article – use wget with caution. wget is a direct link to the other server [ file, web, etc… ] Especially if you use the -r option!! If you’re not watching the logs you could download 500g of data before you know it. Excellent intro to the topic. To add to this article I’d also suggest to look at the documentation.
Hello sir, i am new to linux but when i read ur posts its really interesting to learn and use commands of linux.
Its Nice to have u…………………..
thanks a lot
In example 2 can use –content-disposition option to save file in correct name.
$ wget –content-disposition http://www.vim.org/scripts/download_script.php?src_id=7701
just what i was looking for.. i had the older wget syntax.
thank you
Thanks for the excellent information!
Is there a way to pass an argument to the download link? I am looking to download build from Jenkins/Hudson server but the build number keeps auto-incrementing to a new number. Hence I have to update the URL with the new build number.
I am looking for a way to automate this process and not enter the build number (may be through a script?). Any help would be appreciated..
@Deepak
you have two choices:
1. replace the number incremented by *
2. do a script who increment the last number just before the wget line, and insert it in a variable..
Regards
Thanks ned!
Yeah I’m looking at option #2 and will report once I get it working..
@Deepak, here is a solution,
:: you put the variable to be increment in the file C:\test\increment_variable.txt
:: this line set the number in the file to a variable v
FOR /F %%a IN (C:\test\increment_variable.txt) DO SET v=%%a
:: increment the variable
SET /A increment_v=%v%+1
:: do your wget commande
wget … %increment_v%
:: replace the varible incremented in the file
echo %increment_v% > C:\test\increment_variable.txt
Let me know if that works ^^
Regards
Thanks for the quick tutorial.
how to download check file “check_memcached.tar.gz” from http://exchange.nagios.org/components/com_mtree/attachment.php?link_id=2182&cf_id=24 this link. When I use wget it’s download only “attachment.php?link_id=2182” this file and size is “0”. Please help me.
Thanks.
you need to put the extention of the file .tar.gz u want to download
–limit-rate is an unrecognized option
How to download all videos from a specif website?
Thanks,
that was really nice
Nice examples.
Some urls require authentication user, password. One may use :
wget –user –password URL
However, when the above above command is downloading the file, password can be viewed by other users on different sessions by simply typing ‘ps -eaf|grep wget’. To avoid this, use the below option which will ask password later and will password will not appear in the process list:
wget –user –ask-password URL
NOTE: not sure if this option for wget is available in all flavours of unix
very nicely explanied, thank a lot
Fantastic tool as well as this teaching article. Thank you most much for sharing knowledge !!!
Just tried “Download Multiple Files / URLs Using Wget -i” for 6 embeded mp3 files and it works like a charm !!!
I would like to Download all the .pdf and reject .zip files of a Website including there Original ‘Directory and Sub-Directory’ i am trying the following command
wget -r -l0 url
But, its downloading all the files of a url including ‘index.php, and .zip’ files…i want to reject ‘.php, and .zip’ files
awesome…
this article really help me.
thanks alot.
These guides were extremely helpful. Especially the demonstration of the user agent switch. Helped me a lot. thanks
Hello
Very userful info
But i have a few problems
1) How to download https files
2) How to download the files(when username and password is mentioned in the URL
Hi,
I have an interesting issue, I am trying to tackle. Consider this link
http://app.quotemedia.com/quotetools/clientForward?action=showHistory&symbol=BLC&targetURL=http://www.quotemedia.com/results.php&targetsym=qm_symbol
I need to the output of the data ie. values of open,high,low,close, vol, chg, et. et.
I tried several options however I am not able to get the data in a file, this is oe of the several commands i used.
wget –user-agent=”Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3″ -O c:/temp/sample.csv “http://app.quotemedia.com/quotetools/clientForward?action=showHistory&symbol=BLC&targetURL=http://www.quotemedia.com/results.php&targetsym=qm_symbol”
Any inputs will be appreciated.
Thanks.
I downloaded following file
http://sourceforge.net/projects/opsi/postdownload?source=dlp
file is 920 Mb approximately, I do not use any download manager it was a direct http download, after having approximately 600 Mb of download some how the download has broken, I have a file now opsi4.0.3-2-servervm.zip which is 600 Mb but this should have been 900 Mb is there any way to resume this one using wget
@abhishek
take a look to num 4,
4. Continue the Incomplete Download Using wget -c
Regards
can someone tell me how to view the contents of downloaded file in unix…
hello, how do we retry wget after when we see 500 Internal Server Error? Regards
Does this work for wget for windows too? I am using the command to downlaod all PDF files and it tries to download the entire website. help!
Hi, Can you please let me know how can we download the pdf files only from the root folder and i need to eliminate the files from sub folders
Thank you 🙂 Great tips!
I am using wget to download a file and checking the maximum network speed I can get!
I am getting the network speed as 268 MB/s. Can you please tell me some other method for this? Or can you please tell me a file of greater size, so that I can get the total speed available?
Thanks mate, i needed to download all files from the ubuntu archives and wasn’t getting anywhere until i read your post. Really helpful. Keep up the good work. 🙂
How can I use wget to download my facebook page wtih all dialogues?
Some sites needs authentication. I tried using following command:
wget –user=username –password=$pass!word “http://…………………”
But it didnot work. As you can notice password has special characters $ and !. May be because of this it is failing to authenticate. I tried using password in single quote as well. Can some one please help me in this ?
Have u tried adding –debug to get more information? I would also try adding double quotes around the string values.
hello there, thanks for great tips. I just want to ask, if there is any option for downloading for example 100 websites using wget but each website will be downloaded only in 50MB (or any other size). I tried using quota and i have .txt fie with urls of my webpages i want to download, but it only download 50MB of first website and thats all. Thanks for answer.
Im try to download a zip file with following requirements
1. custom name to the file
2. unzip the file
rit now, im using 2 separate commands
wget -O /tmp/geoip/data.zip http://geolite…link
unzip -p /tmp/geoip/data.zip > /tmp/geoip/data.csv
Is there a way to squash them into a single command?
Hi,
Is it possible to send 100 requests per second using wget tool? Just like pumping more requests continuously to the site.
Thanks in advance for reply.
Thanks! This is a very good article
How to use wget command to store in different location and also rename it. I know for rename we can use -O but if use like this it is not getting stored in /tmp directory , its getting stored in current directory
wget -O vertica-8.0.0-0.x86_64.RHEL6.rpm “http://s11p192.network.com:50070/webhdfs/v1/dist/vertica-8.0.0-0.x86_64.RHEL6.rpm?op=OPEN” -P /tmp/
Thanks! This is a very good article
How to use wget command to store in different location and also rename it. I know for rename we can use -O but if use like this it is not getting stored in /tmp directory , its getting stored in current directory
I want to use -O -P together . Please help
wget -O vertica-8.0.0-0.x86_64.RHEL6.rpm “http://s11p192.network.com:50070/webhdfs/v1/dist/vertica-8.0.0-0.x86_64.RHEL6.rpm?op=OPEN” -P /tmp/