Module Todo: Document Metadata Extraction

[TheTechromancer]

It would be useful to have a collection of modules that download documents (.pdf, .docx, etc.) and extract useful metadata such as usernames and internal domain names. Thanks to @pjhartlieb and @Sw3d1shPh1sh for requesting.

Also, per @nicpenning:

"FOCA is a tool used mainly to find metadata and hidden information in the documents it scans. These documents may be on web pages, and can be downloaded and analysed with FOCA."

Link to the project: https://github.com/ElevenPaths/FOCA/tree/master

Hard to say what the best approach is to implement so it's just an idea :) and maybe some of these features are taken care of in other modules.

Would require:

• A module that listens for UNVERIFIED_URLs matching PDF, DOCX, etc., downloads them, and re-raises them as binary blobs
• A module that consumes the binary blobs and extracts metadata

EDIT: Possible sources of metadata-extraction logic:

• https://github.com/datacoon/metawarc (supports MS Office old + new, PDF, Images)

[TheTechromancer]

Note: it would also be nice to emit the text from these documents as a generic event consumable by excavate, secretsdb, etc -- e.g. RAW_TEXT or something similar to Spiderfoot's RAW_RIR_DATA event type. This would enable them to feed the recursive discovery process with more subdomains, emails, URLs, etc.

[TheTechromancer]

As a prerunner for this, I have written a proof-of-concept filedownload module that watches for interesting filetypes and downloads them into the scan's output folder. Dev is happening on the filedownload-module branch.

@nicpenning here is the module:
https://github.com/blacklanternsecurity/bbot/blob/filedownload-module/bbot/modules/filedownload.py

You can use it like this:

bbot -t evilcorp.com -f subdomain-enum -m filedownload

Pairing it with the web spider can also be very effective:

bbot -t evilcorp.com -f subdomain-enum -m filedownload -c web_spider_depth=2 web_spider_distance=2

[domwhewell-sage]

This is probably relevant to this discussion #907 (comment).

Now there are FILESYSTEM events the downloads can probably be raised as that.

As mentioned in the linked discussion that is a ML model to detect human passwords in several file formats.

Perhaps more interesting though is it uses Apache Tika to extract the strings from

extracts metadata and text from over a thousand different file types (such as PPT, XLS, and PDF)

which we could then raise as RAW_DATA events for ingestion by other modules

[TheTechromancer]

Issue created: #1421. I likely won't have time to work on it for a bit, since I'm busy with BBOT 2.0. Anyone is welcomed to work on it in the meantime.

[nicpenning]

Where does the FILESYSTEM event type exist? I am trying to get a better understanding how these event producers/consumers work via https://www.blacklanternsecurity.com/bbot/dev/event/. I looked at some modules that use it but it doesn't clearly show me how/what FILESYSTEM is doing.

I see most of the other event types here: https://github.com/blacklanternsecurity/bbot/blob/stable/bbot/core/event/base.py

But FILESYSTEM doesn't exist and WEBSCREENSHOT is very limited

bbot/bbot/core/event/base.py

Lines 1180 to 1182 in eeae1cb

	class WEBSCREENSHOT(DictHostEvent):
	_always_emit = True
	_quick_emit = True

.

Could you please point me in the right direction where Event code lives?

[TheTechromancer]

The reason there isn't any unique code for those events is because we haven't needed it. In BBOT, an event is made up of a type (e.g. FILESYSTEM) and data (e.g. {"path": "~/.bbot/scans/suspicious_dobby/git_clone/bbot"}). You can specify whatever type and whatever data you want. There aren't any schemas or rules except for a few certain predefined event types.

If you want to add custom validation / functionality to an event type, you can create the class and override whichever methods you need, just like we've done with some of the other ones. The code would live core/event/base.py, so you're already in the right place.

[nicpenning]

That makes more sense then! Got it. Is that likely what will be needed for when we try to implement Base64 blobs? Or is that a different layer. I was going to take a crack at what that would look like but I am still learning BBOT dev.

[TheTechromancer]

Yep, I think it would end up looking something like this:

class FILESYSTEM(DictEvent):
    def sanitize_data(self, data):
        new_data = dict(data)
        if self.scan is not None:
            include_base64 = self.scan.config.get("include_base64_blob", False)
            
            if include_base64:
                data_path = Path(data["path"])
                if data_path.is_file():
                    # blob = <read file into blob>
                elif data_path.is_dir():
                    # blob = <tar up directory into blob>
                new_data["blob"] = blob

        return new_data