Parses Slack export archives into an ElasticSearch index for exploring with Kibana
Makes use of Haskell's Conduit streaming libraries along with Aeson and Bloodhound to unzip, parse and transform Slack export(s) efficiently into an index using the ElasticSearch bulk API.
Tested with the Elastic v5-alpha products (ElasticSearch, Kibana, Marvel) and ships with a fully functioning Docker deployment of these tools. Also includes a Slack admin scraper and working cron-based process for scheduled automated exports from Slack into your index (TODO
).
TODO
: Performance benchmarking of Conduit vector sizing with ElasticSearch Marvel
Pre-requisite: Stack
Attention: always check the source before piping from curl to your shell!
curl -sSL https://get.haskellstack.org/ | sh
stack setup
stack build
stack install
~/.local/bin/slack-export-indexer
TODO
: Rancher & Docker compose. Crond and Slack admin scraping
Usage: slack-export-indexer [-e|--es-url URL] [-b|--bulk-size SIZE]
[-s|--shards COUNT] [-r|--replicas COUNT]
[-i|--index NAME] [-m|--mapping NAME]
[EXPORT_ZIP_FILE(s)...]
Parses Slack export archives into ElasticSearch for exploring with Kibana
Available options:
-h,--help Show this help text
-e,--es-url URL Target ElasticSearch
URL. (default: "http://localhost:9200")
-b,--bulk-size SIZE Size of the vector buffered to ElasticSearch via the
bulk API. (default: 1000)
-s,--shards COUNT Number of primary shards. (default: 5)
-r,--replicas COUNT Number of replica shards. (default: 1)
-i,--index NAME Name of Slack index. (default: "slack")
-m,--mapping NAME Name of Slack index mapping. (default: "events")