Add support for filtering results with a unique size for each status code

[itsbriany]

When attempting to brute force web servers, they may return similar responses for different URLs.

For example, a web server may redirect us to its homepage, giving us the same final result each time for multiple URLs. If we pipeline gobuster's output into other tools, there may be cases where we only care about URLs that point us to unique content.

For example, brute-forcing a web server with the following command:

./gobuster -m dir -q -k -v -l -w wordlist.txt -u https://example.com

And wordlist:

.
./lib
./bar
./car
../lib
../foo
../../bar

Will yield the following:

Missed: /../../bar (Status: 400) [Size: 357]
Missed: /../foo (Status: 400) [Size: 357]
Missed: /./car (Status: 404) [Size: 332]
Found: /./lib (Status: 301) [Size: 370]
Missed: /./bar (Status: 404) [Size: 332]
Missed: /../lib (Status: 400) [Size: 357]
Found: /. (Status: 200) [Size: 132]

Notice that the paths ../../bar and ../foo (likely) result in the same response since they have content of the same size.

There may be cases where I would like to have the following output instead:

Found: /./lib (Status: 301) [Size: 370]
Missed: /../foo (Status: 400) [Size: 357]
Missed: /./car (Status: 404) [Size: 332]
Found: /. (Status: 200) [Size: 132]

Since I only care about URLs that will give me a unique response.

I have implemented a patch that allows gobuster to filter results with a unique size for each status code to addresses the issue above.

If there is anything I have missed, I will be happy to address it in subsequent commits.

[itsbriany]

I thought about this again and I think it actually might be better to keep the existing gobuster functionality, but instead write the output to JSON or XML and let another tool parse that output. I am investigating #75 to see if I can get gobuster to output JSON instead.

If any of you think this is still a feature worth adding, let me know since my current solution in its current state will not function properly due to race conditions between threads. Additionally, synchronization checks are missing in my first commit. I also think that with my current solution, the amount of mutex locking/unlocking to achieve synchronization will also impact performance and hence should be designed differently.

[IvoPereira]

@itsbriany I find a lot of use for this feature and I wouldn't see a reason why it wouldn't benefit the whole package itself instead of having a separate tool for it.