Copy Your Site for Backup Purposes
---------------------------------------------------------------------------
wget -mirror -w 3 -p -P /home/me/wget_files/example2 ftp://username:password@ftp.yourdomain.com
Download a File
---------------------------------------------------------------------------
Wget -P /home/me/wget_files/example3 http://ftp.gnu.org/gnu/wget/wget-1.9.tar.gz
Download a File
---------------------------------------------------------------------------
wget --spider -r -nd -nv -w 1 -o check-links.txt https://mysite.com
# Then grep -B 1 'dead link' check-links.txt
Download PDFs listed in a page
---------------------------------------------------------------------------
wget -r -l 1 -nH -nd -np --ignore-case -A '*.pdf' https://matematicas.unex.es/~navarro/res/
Mirror a Website for Offline Browsing
mkdir wget_files
cd wget_files
wget \
--mirror \
--wait 1 \
--page-requisites \
# --adjust-extension \
--convert-links \
--cut-dirs=1 \
--relative \
--no-parent \
# --no-host-directories \
http://stat-www.berkeley.edu/~stark/SticiGui/Text/toc.htm
--mirror (m) : Specifies to mirror the site. Wget will recursively follow all links on the site and download all necessary files. It will also only get files that have changed since the last mirror, which is handy in that it saves download time.
--wait (-w) : Tells wget to "wait" or pause between requests, in this case for 2 seconds. This is not necessary, but is the considerate thing to do. It reduces the frequency of requests to the server, thus keeping the load down. If you are in a hurry to get the mirror done, you may eliminate this option.
--page-requisites (-p) : Causes wget to get all required elements for the page to load correctly. Apparently, the mirror option does not always guarantee that all images and peripheral files will be downloaded, so I add this for good measure.
--adjust-extension: All files with a non-HTML extension will be converted to have an HTML extension. This will convert any CGI, ASP or PHP generated files to HTML extensions for consistency.
--convert-links: All links are converted so they will work when you browse locally. Otherwise, relative (or absolute) links would not necessarily load the right pages, and style sheets could break as well.
-P (prefix folder): The resulting tree will be placed in this folder. This is handy for keeping different copies of the same site, or keeping a "browsable" copy separate from a mirrored copy.
--relative (-L) : Follow relative links only. Useful for retrieving a specific home page without any distractions, not even those from the same hosts.
--no-parent (-np) : Do not ever ascend to the parent directory when retrieving recursively. It guarantees that only the files below a certain hierarchy will be downloaded.
--no-host-directories (-nH) : Disable generation of host-prefixed directories.
Interesting options
-A acclist
--accept acclist
-R rejlist
--reject rejlist
Specify comma-separated lists of file name suffixes or patterns to accept or reject. Note that if any of the wildcard characters,
*, ?, [ or ], appear in an element of acclist or rejlist, it will be treated as a pattern, rather than a suffix. In this case, you
have to enclose the pattern into quotes to prevent your shell from expanding it, like in -A "*.mp3" or -A '*.mp3'.
--accept-regex urlregex
--reject-regex urlregex
Specify a regular expression to accept or reject the complete URL.
-nd
--no-directories
Do not create a hierarchy of directories when retrieving recursively. With this option turned on, all files will get saved to the current directory, without clobbering (if a name shows up more than once, the filenames
will get extensions .n).
--regex-type regextype
Specify the regular expression type. Possible types are posix or pcre. Note that to be able to use pcre type, wget has to be
compiled with libpcre support.
-D domain-list
--domains=domain-list
Set domains to be followed. domain-list is a comma-separated list of domains. Note that it does not turn on -H.
--exclude-domains domain-list
Specify the domains that are not to be followed.
-i file
--input-file=file
Read URLs from a local or external file. If - is specified as file, URLs are read from the standard input. (Use ./- to read from
a file literally named -.)
rsync over https
remote_dirs=( "https://example.com/" "…") # Add your remote HTTP directories here
local_dirs=( "~/tmp/example.com-2026-06-17T01:05:41" "…")
for (( i = 0 ; i < ${#local_dirs[@]} ; i++ )) do
cd "${local_dirs[$i]}"
wget -r -l1 --no-parent -A "*.pdf" -nd -nc ${remote_dirs[$i]}
done
# Explanation:
# -r to download recursively
# -l1 to include only one directory depth
# --no-parent to exclude parent directories
# -A "*.pdf" to accept only .pdf files
# -nd to prevent wget to create directories for everything
# -N to make wget to download only new files