Directory and subdomain enumeration with Gobuster

Overview

Gobuster is a tool used to brute-force: URIs (directories and files) in web sites, DNS subdomains (with wildcard support), Virtual Host names on target web servers, Open Amazon S3 buckets, Open Google Cloud buckets and TFTP servers.

Functionality

Gobuster is useful for pentesters, ethical hackers and forensics experts. It also can be used for security tests. It can be used for:

Searching for directories and files not indexed by search engines or invisible to regular users.
Discovering subdomains of a domain that are accessible but not publicly available.
Discovering hidden DNS records or subdomains using wordlists.
Discovering poorly protected S3 buckets in AWS, which may contain sensitive data.

Installation

The first step is to install Gobuster on your laptop.

sudo apt install gobuster

┌──(ardityo㉿corenux)-[~]
└─$ gobuster --help                                     
NAME:
   gobuster - the tool you love

USAGE:
   gobuster command [command options]

VERSION:
   3.8

AUTHORS:
   Christian Mehlmauer (@firefart)
   OJ Reeves (@TheColonial)

COMMANDS:
   dir      Uses directory/file enumeration mode
   vhost    Uses VHOST enumeration mode (you most probably want to use the IP address as the URL parameter)
   dns      Uses DNS subdomain enumeration mode
   fuzz     Uses fuzzing mode. Replaces the keyword FUZZ in the URL, Headers and the request body
   tftp     Uses TFTP enumeration mode
   s3       Uses aws bucket enumeration mode
   gcs      Uses gcs bucket enumeration mode
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

Recommended Tester’s Companion

Since Gobuster is a tool that is used in brute-force techniques, therefore, we need a wordlist that is used as a resource by Gobuster. SecLists is the security tester’s companion. It’s a collection of multiple types of lists used during security assessments, collected in one place. List types include usernames, passwords, URLs, sensitive data patterns, fuzzing payloads, web shells, and many more.

sudo apt install seclists

Seclists is located in the /usr/share/seclists

┌──(ardityo㉿corenux)-[~]
└─$ ls /usr/share/seclists
Discovery  Fuzzing  Miscellaneous  Passwords  Pattern-Matching  Payloads  README.md  Usernames  Web-Shells

Use Cases

Scan Hidden Directories and Files

One of the ability of the Gobuster is to scanning the hidden directories and files in a domain. Use dir as subcommand:

┌──(ardityo㉿corenux)-[~]
└─$ gobuster dir --help
NAME:
   gobuster dir - Uses directory/file enumeration mode

USAGE:
   gobuster dir [command options] [arguments...]

OPTIONS:
   --url value, -u value                                    The target URL
   --cookies value, -c value                                Cookies to use for the requests
   --username value, -U value                               Username for Basic Auth
   --password value, -P value                               Password for Basic Auth
   --follow-redirect, -r                                    Follow redirects (default: false)
   --headers value, -H value [ --headers value, -H value ]  Specify HTTP headers, -H 'Header1: val1' -H 'Header2: val2'
   --no-canonicalize-headers, --nch                         Do not canonicalize HTTP header names. If set header names are sent as is (default: false)
   --method value, -m value                                 the password to the p12 file (default: "GET")
   --useragent value, -a value                              Set the User-Agent string (default: "gobuster/3.8")
   --random-agent, --rua                                    Use a random User-Agent string (default: false)
   --proxy value                                            Proxy to use for requests [http(s)://host:port] or [socks5://host:port]
   --timeout value, --to value                              HTTP Timeout (default: 10s)
   --no-tls-validation, -k                                  Skip TLS certificate verification (default: false)
   --retry                                                  Should retry on request timeout (default: false)
   --retry-attempts value, --ra value                       Times to retry on request timeout (default: 3)
   --client-cert-pem value, --ccp value                     public key in PEM format for optional TLS client certificates]
   --client-cert-pem-key value, --ccpk value                private key in PEM format for optional TLS client certificates (this key needs to have no password)
   --client-cert-p12 value, --ccp12 value                   a p12 file to use for options TLS client certificates
   --client-cert-p12-password value, --ccp12p value         the password to the p12 file
   --tls-renegotiation                                      Enable TLS renegotiation (default: false)
   --interface value, --iface value                         specify network interface to use. Can't be used with local-ip
   --local-ip value                                         specify local ip of network interface to use. Can't be used with interface
   --wordlist value, -w value                               Path to the wordlist. Set to - to use STDIN.
   --delay value, -d value                                  Time each thread waits between requests (e.g. 1500ms) (default: 0s)
   --threads value, -t value                                Number of concurrent threads (default: 10)
   --wordlist-offset value, --wo value                      Resume from a given position in the wordlist (default: 0)
   --output value, -o value                                 Output file to write results to (defaults to stdout)
   --quiet, -q                                              Don't print the banner and other noise (default: false)
   --no-progress, --np                                      Don't display progress (default: false)
   --no-error, --ne                                         Don't display errors (default: false)
   --pattern value, -p value                                File containing replacement patterns
   --discover-pattern value, --pd value                     File containing replacement patterns applied to successful guesses
   --no-color, --nc                                         Disable color output (default: false)
   --debug                                                  enable debug output (default: false)
   --status-codes value, -s value                           Positive status codes (will be overwritten with status-codes-blacklist if set). Can also handle ranges like 200,300-400,404
   --status-codes-blacklist value, -b value                 Negative status codes (will override status-codes if set). Can also handle ranges like 200,300-400,404. (default: "404")
   --extensions value, -x value                             File extension(s) to search for
   --extensions-file value, -X value                        Read file extension(s) to search from the file
   --expanded, -e                                           Expanded mode, print full URLs (default: false)
   --no-status, -n                                          Don't print status codes (default: false)
   --hide-length, --hl                                      Hide the length of the body in the output (default: false)
   --add-slash, -f                                          Append / to each request (default: false)
   --discover-backup, --db                                  Upon finding a file search for backup files by appending multiple backup extensions (default: false)
   --exclude-length value, --xl value                       exclude the following content lengths (completely ignores the status). You can separate multiple lengths by comma and it also supports ranges like 203-206
   --force                                                  Continue even if the prechecks fail. Please only use this if you know what you are doing, it can lead to unexpected results. (default: false)
   --help, -h                                               show help

┌──(ardityo㉿corenux)-[~]
└─$ gobuster dir -u https://example.com -w /usr/share/seclists/Discovery/Web-Content/common.txt   

===============================================================
Gobuster v3.8
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     https://example.com
[+] Method:                  GET
[+] Threads:                 10
[+] Wordlist:                /usr/share/seclists/Discovery/Web-Content/common.txt
[+] Negative Status codes:   404
[+] User Agent:              gobuster/3.8
[+] Timeout:                 10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================
/index.html           (Status: 200) [Size: 1256]
/payments             (Status: 403) [Size: 369]
/pbc_download         (Status: 403) [Size: 377]
Progress: 4746 / 4746 (100.00%)
===============================================================
Finished
===============================================================

Gobuster Threads

Gobuster is fast, with hundreds of requests being sent using the default 10 threads. This will cause a system having a problem if your machine can’t handle it. Therefore, you could adjust the thread numbers lower (e.g., -t 4). Additionally it can be helpful to use the flag --delay (Time each thread waits between requests (e.g. 1500ms)). For example --delay 1s in other words, if --threads is set to 4 and --delay to 1s, this will send 4 requests per second.

┌──(ardityo㉿corenux)-[~]
└─$ gobuster dir -u https://example.com -w /usr/share/seclists/Discovery/Web-Content/common.txt -t 4 -d 1s -o results.txt
===============================================================
Gobuster v3.8
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     https://example.com
[+] Method:                  GET
[+] Threads:                 4
[+] Delay:                   1s
[+] Wordlist:                /usr/share/seclists/Discovery/Web-Content/common.txt
[+] Negative Status codes:   404
[+] User Agent:              gobuster/3.8
[+] Timeout:                 10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================
/index.html           (Status: 200) [Size: 1256]
/payments             (Status: 403) [Size: 369]
/pbc_download         (Status: 403) [Size: 377]
Progress: 4746 / 4746 (100.00%)
===============================================================
Finished
===============================================================

Check the results.txt to get the result

┌──(ardityo㉿corenux)-[~]
└─$ cat results.txt
/index.html           (Status: 200) [Size: 1256]
/payments             (Status: 403) [Size: 369]
/pbc_download         (Status: 403) [Size: 377]

Continue Enumerating

Since we got the /payments dir, we could continue to scan and look at what is behind this dir.

┌──(ardityo㉿corenux)-[~]
└─$ gobuster dir -u https://example.com/payments -w /usr/share/seclists/Discovery/Web-Content/common.txt -o results-payments.txt
===============================================================
Gobuster v3.8
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     https://example.com/payments
[+] Method:                  GET
[+] Threads:                 10
[+] Wordlist:                /usr/share/seclists/Discovery/Web-Content/common.txt
[+] Negative Status codes:   404
[+] User Agent:              gobuster/3.8
[+] Timeout:                 10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================
/.cache               (Status: 403) [Size: 384]
/.forward             (Status: 403) [Size: 386]
/.git-rewrite         (Status: 403) [Size: 394]
/rails                (Status: 403) [Size: 379]
/rar                  (Status: 403) [Size: 377]
Progress: 4746 / 4746 (100.00%)
===============================================================
Finished
===============================================================

Check result file for /payments

┌──(ardityo㉿corenux)-[~]
└─$ cat results-payments.txt 
/.cache               (Status: 403) [Size: 384]
/.forward             (Status: 403) [Size: 386]
/.git-rewrite         (Status: 403) [Size: 394]
/rails                (Status: 403) [Size: 379]
/rar                  (Status: 403) [Size: 377]

Scan Specific File Extension

An example of another flag to use is the -x File extension(s) to search for. This is for the times when a search for specific file extension or extensions is specified. Such as, -x .php or other only is required.

┌──(ardityo㉿corenux)-[~]
└─$ gobuster dir -u https://example.com/ -w /usr/share/seclists/Discovery/Web-Content/big.txt -x .php -o results-php.txt
===============================================================
Gobuster v3.8
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     https://example.com/
[+] Method:                  GET
[+] Threads:                 10
[+] Wordlist:                /usr/share/seclists/Discovery/Web-Content/big.txt
[+] Negative Status codes:   404
[+] User Agent:              gobuster/3.8
[+] Extensions:              php
[+] Timeout:                 10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================
/ServerAdministrator  (Status: 403) [Size: 380]
/absolutenm           (Status: 403) [Size: 371]
/absolutepm           (Status: 403) [Size: 371]
/america              (Status: 403) [Size: 368]
/anwender             (Status: 403) [Size: 369]
/csv.php              (Status: 403) [Size: 372]
/database_administration (Status: 403) [Size: 388]
/email_disclaimer.php (Status: 403) [Size: 389]
/old_stuff.php        (Status: 403) [Size: 382]
/oldstats             (Status: 403) [Size: 369]
Progress: 40956 / 40956 (100.00%)
===============================================================
Finished
===============================================================

DNS Enumeration

Use the DNS command to discover subdomains.

┌──(ardityo㉿corenux)-[~]
└─$ gobuster dns -q -r 8.8.8.8 -d example.com -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt

VHost Enumeration

The vhost command discovers Virtual host names on target web servers. Virtual hosting is a technique for hosting multiple domain names on a single server. Exposing hostnames on a server may reveal supplementary web content belonging to the target. Vhost checks if the subdomains exist by visiting the formed URL and cross-checking the IP address. To brute-force virtual hosts, use the same wordlists as for DNS brute-forcing subdomains. Similar to brute forcing subdomains eg. url = example.com, vhost looks for dev.example.com or beta.example.com etc.

┌──(ardityo㉿corenux)-[~]
└─$ gobuster vhost -u https://example.com -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt -q -o results-vhost.txt
===============================================================
Gobuster v3.8
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                       https://example.com
[+] Method:                    GET
[+] Threads:                   10
[+] Wordlist:                  /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt
[+] User Agent:                gobuster/3.8
[+] Timeout:                   10s
[+] Append Domain:             false
[+] Exclude Hostname Length:   false
===============================================================
Starting gobuster in VHOST enumeration mode
===============================================================
Finished
===============================================================

Get the result in results-vhost.txt

┌──(ardityo㉿corenux)-[~]
└─$ cat results-vhost.txt 
webmail Status: 400 [Size: 310]
webdisk Status: 400 [Size: 310]
mail Status: 400 [Size: 312]
pop Status: 400 [Size: 310]
ftp Status: 400 [Size: 310]
localhost Status: 400 [Size: 310]
www Status: 400 [Size: 312]
whm Status: 400 [Size: 310]

After we got something from vhost, we could:

Verify if the service is active: check if the virtual host responds, meaning there’s something running behind it.
Discover hidden areas: access pages like admin panels, staging sites, or backups not meant for public view.
Uncover sensitive info: the vhost may lead to exposed configuration files or forgotten content.

Conclusion

With the use cases above, we can find out some of Gobuster’s capabilities, including:

Description	Command
Web interesting files	gobuster dir -u mytarget.com -w path/to/my/awesome/wordlist.txt -e -t 100 -x php,txt,html,htm
Find images	gobuster dir -u mytarget.com -w path/to/my/awesome/wordlist.txt -e -t 100 -x jpg,jpeg,png,gif,ico
Skip SSL verification	gobuster dir -u mytarget.com -w path/to/my/awesome/wordlist.txt -k
Bypass Basic Auth	gobuster dir -u mytarget.com -w path/to/my/awesome/wordlist.txt -U BasicAuthUser -P BasicAuthPass
Custom DNS server	gobuster dns -d mytarget.com -w path/to/my/awesome/wordlist.txt -r 10.10.10.10 -i -v

Defensive Security

Mobile app analysis

Docs

Corenux

Title here

Directory and subdomain enumeration with Gobuster

Overview

Functionality

Installation

Recommended Tester’s Companion

Use Cases

Scan Hidden Directories and Files

Continue Enumerating

Scan Specific File Extension

DNS Enumeration

VHost Enumeration

Conclusion

Directory and subdomain enumeration with Gobuster

Overview

Functionality

Installation

Recommended Tester’s Companion

Use Cases

Scan Hidden Directories and Files#

Continue Enumerating#

Scan Specific File Extension#

DNS Enumeration#

VHost Enumeration#

Conclusion#

Scan Hidden Directories and Files

Continue Enumerating

Scan Specific File Extension

DNS Enumeration

VHost Enumeration

Conclusion