Download Files from Web Using wget or curl

Automate downloads via URL.

Downloading files directly from the web is a routine task for developers, system administrators, and power users alike. Whether it’s images, software, documents, or dataset files, automating downloads saves time and effort. Two of the most popular command-line tools for this job are wget and curl. Both are powerful, flexible, and available on most Unix-like systems, including Linux and macOS (and Windows, with some tweaks).

In this article, we’ll explore how to use wget and curl to reliably download files with ease, covering practical examples and tips for automation.

Why Use wget or curl?

Both tools are designed for web interaction:

  • wget is specialized for downloading files and supporting recursive downloads, making it perfect for bulk downloads or fetching entire directories.
  • curl is more of a Swiss army knife for network transfers, supporting many protocols and making it great for both simple downloads and complex interactions like API requests.

Installing wget and curl

Before starting, make sure you have these tools available.

On Debian/Ubuntu:

sudo apt-get install wget curl

On macOS (with Homebrew):

brew install wget curl

On Windows:

  • curl is built into recent Windows 10+ releases as curl.exe.
  • wget can be used via WSL, or by downloading from eternallybored.org.

Downloading Files with wget

Basic Download

To download a file from a URL:

wget https://example.com/sample.pdf

The file will be saved with its original name in the current directory.

Save with a Different Name

wget -O newname.pdf https://example.com/sample.pdf

Resume Interrupted Download

wget -c https://example.com/largefile.zip

Download Multiple Files

Create a text file (e.g., urls.txt) with each URL on a new line, then run:

wget -i urls.txt

Download Entire Website Section

Recursively download directories or sites (be careful!):

wget -r https://example.com/directory/

Downloading Files with curl

Basic Download

Use the -O (uppercase o) flag to save with the original filename:

curl -O https://example.com/sample.pdf

Use the -o (lowercase o) flag to specify a filename:

curl -o newname.pdf https://example.com/sample.pdf

Resume Interrupted Download

curl -C - -O https://example.com/largefile.zip

Download Multiple Files

With a loop:

cat urls.txt | xargs -n 1 curl -O

Or, with a simple for loop:

for url in $(cat urls.txt); do
  curl -O "$url"
done

Download with Authentication

For sites requiring a username and password (use with caution):

curl -u username:password -O https://example.com/securefile.pdf

Tips and Best Practices

  • Check the License: Always ensure you have permission to download content.
  • Respect Servers: Don’t aggressively scrape websites or download large numbers of files without consent.
  • Automate with Scripts: Incorporate wget or curl in shell scripts for scheduling via cron or other job runners.
  • Handle Failures: For robust automation, check wget/curl exit codes and handle errors gracefully.

Summary Table

Feature wget curl
Default in Linux Distros Yes Yes
Recursive Download Yes No
Resume Downloads Yes (-c) Yes (-C -)
Save As Different Filename Yes (-O) Yes (-o)
Download Multiple URLs Yes (-i file.txt) Scripted Loop
HTTP Authentication Yes Yes (-u)
Protocols Supported HTTP, FTP, … HTTP, FTP, SFTP, …

Conclusion

Automating file downloads from the Web is simple and efficient with wget and curl. Both tools are invaluable for scripting, automation, or just convenience on the command line. Choose the one that best fits your workflow—or master both for maximum flexibility.

Next time you need to fetch files from a URL, try these commands and make downloading just a little bit easier!

Back to All Articles