Programming Tips For Versatile Coders

• Tweet

• Tweet

I remember that back in the days that i started to learn more about the different linux/unix tools, i was fascinated by the great wget http downloader. Believe me, you will end up using this one quite a lot, even just for means of running a cron to update a website or for some more interesting stuff, like downloading a whole website.

You will have certainly been in the position where you just wanted to download a website because of its great information wealth. In this short post, i will tell you how to do that easily without having to use tools like Teleport Pro in Windows, but just using the awesome wget. Notice that while you can download a whole website that has static pages, downloading websites made in dynamic languages like php is not feasible. This happens because php is by nature a language that depends on user interaction. Therefore, wget cannot just go about getting php pages, it can be done but not this way.

However, most websites that people want to download are those that have lots of static content like videos, images and stuff. These websites can be downloaded by wget. The command to download a whole website is pretty easy:

wget -mk www.website.com

This creates a mirror(-m) for the website that we specify and convert its links to local links(-k) to make our offline browsing easy. Sometimes, this implementation may have a problem because some websites check the user agent field of our connection to specify whether we use wget to download the whole website, thus not allowing us to do so. The -m switch usese “wget” as a user agent specification. Therefore, if we want to specify another user agent, like Mozilla, we should use wget to download the whole website as follows:

wget -r -k -U Mozilla www.website.com

We specify -r here to download a website recursively (that is what -m actually does also). Moreover, using the -U switch, we specify the user agent as being “Mozilla” instead of “wget”.

Bonus Tip – Continue Download In Case of Interruption

For some reason, wget might be interrupted while you are downloading a website. This can be very frustrating, especially if you are downloading big chunks of data like videos. The good thing is that you won’t have to redownload the same files again using wget. Just by using the switch -c, you can just as simply continue the download right at the same spot where it stopped and not lose any progress. Doing that is easy, if you sometime get an interruption on your downloading, just use the same command you used at the first place only including the -c switch after wget, like:

wget -c -r -k -U Mozilla www.website.com

Random Posts

• The Strategy Design Pattern in Ruby
• Ubuntu Releases History – A Journey to The Past
• SSHFS – Remote Filesystem in Unix
• How to Hack Wireless WEP Connections in 10 Minutes or Less
• Top 10 PHP Techniques That Will Save You Time and Effort
• Rails Associations In Plain English : What I Wish I Had Known Before I Started
• Let’s Do Some Python Programming Together !
• Use Assert to Catch Logic Programming Errors
• 12 Tips That Will Make You a Better Programmer
• Understand Linux File Permissions Using Chmod And Chown Commands

This entry was posted on Friday, November 13th, 2009 at 1:01 am and is filed under Linux Tips. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.