Web Content Discovery (Directory Fuzzing)

A web server only shows what someone chose to link. The admin panel, the forgotten backup.zip, the .git/ deployed by mistake or the leftover db.php are still there and still answering, but no link leads to them. Not being linked is not the same as not existing, so once you have looked at the landing page the next step is to ask for paths blindly: thousands of requests with names from a wordlist, watching which ones answer differently from the ones that do not exist.

# gobuster: fast, with extensions matching the detected technology
gobuster dir -u http://10.10.10.10 -w /usr/share/seclists/Discovery/Web-Content/directory-list-2.3-medium.txt \
  -x php,txt,bak,zip -t 50

# dirsearch: similar, filtering out the status codes that add nothing
dirsearch -u http://10.10.10.10 -e php,html -x 403,404

# ffuf / wfuzz: the FUZZ keyword marks where the wordlist goes, so they work
# for paths, parameters, headers or POST fields
ffuf -u http://10.10.10.10/FUZZ -w raft-medium-directories.txt -fs 4242
wfuzz -u 'http://10.10.10.10/index.php?FUZZ=id' -w burp-parameter-names.txt --hh=275

What each flag does (and why).

• -w (the wordlist) is by far what shapes the result the most. No tool "discovers" anything: it only tries the words you hand it. common.txt is quick for a first pass; directory-list-2.3-medium.txt or SecLists' raft-* are the standard when that first pass comes back empty. If the target is a specific product (WordPress, Jenkins), a wordlist tailored to it pays off far more than a generic one.
• -x php,txt,bak (-e in dirsearch) tries each word with those extensions too. It is necessary because the server executes the file: asking for /admin may return 404 while /admin.php returns 200. The extensions are not picked at random, they come from the technology already identified by the version scan or by whatweb (port scanning): PHP → php, IIS/.NET → asp,aspx, and it is always worth adding the sloppiness ones (bak, old, zip, txt, ~).
• -t (threads) raises the number of parallel requests. More threads is faster, but also noisier and more likely to knock over a fragile application or to trip a WAF; 40-50 is a reasonable spot.
• Filtering the baseline response is the heart of fuzzing. The intuitive move is to drop the 404s (-b 404, --hc=404), but plenty of applications answer 200 with a "not found" page, and then everything looks like it exists. The fix is to first request something that certainly does not exist (/asdfasdf123), note the size of that response and filter it out: --hh=<bytes> in wfuzz (--hl by lines, --hw by words) or -fs <bytes> in ffuf (-fl, -fw). Whatever survives the filter is what genuinely answers differently, which is to say what exists.
• Recursion (-r in feroxbuster, --recursion in gobuster) re-fuzzes inside every directory found. It helps when content hangs several levels deep (/dev/api/v1/...), at the cost of multiplying requests.
• -k accepts self-signed certificates (common on lab HTTPS) and -H adds headers, which you need to fuzz while already authenticated (a session Cookie:) or to vary the Host:, the particular case covered by virtual host fuzzing.

Reading the output. The status code says more than "found or not": a 200 is content being served; a 301/302 is usually a directory redirecting to its trailing-slash version, or a protected resource bouncing you to the login; a 401/403 is just as valuable a finding, because it means it exists but I am not allowed in, and that marks where it is worth pushing (default credentials, bypassing the restriction, another path that reaches the same place).

Why it matters. Almost every chain that starts on the web goes through here: the login panel that yields credentials, the upload directory where the webshell later lands, the .git/ leaking the source code or the config file with the database password all show up at this step and not at another. When a web target "has no vector", usually it does have one and the wordlist or the extensions were simply the wrong ones.

How to detect it. Fuzzing is among the loudest things in the access log: hundreds or thousands of requests from a single IP in a few seconds, most of them 404 (or the size of the "not found" page), walking dictionary names no user would type. The gobuster/ffuf/dirsearch User-Agent and requests to sensitive paths (/.git/, .bak, /admin) confirm it; a WAF or a per-source rate-limit alerts on and throttles it. A spike of 403/401 also gives away that protected paths are being probed.

Defence. Disable directory listing, return a consistent 404 (same code and same size for everything that does not exist, so there is nothing to filter on), keep backups and repositories out of the docroot, and protect panels with authentication rather than with a hard-to-guess path.