Elasticsearch logger

[DestinyMKas]

Hello,

I made simple logger which output log record to elasticsearch.

Requirements

It is tested on phalcon 5.0.0alpha6 version and definitely doesn't works on phalcon <5 versions.
I use elasticsearch 7.15.2. Not sure does it works on other versions.
Also I use elasticsearch library:

$ composer info
elasticsearch/elasticsearch        v7.15.0            PHP Client for Elasticsearch

Code

Elasticsearch logger adapter

<?php

namespace App\Common\Library\Logger;

use Elasticsearch\Client as ElasticClient;
use Phalcon\Logger\Adapter\AbstractAdapter;
use Phalcon\Logger\Item;

class ElasticAdapter extends AbstractAdapter
{
    private $index;

    public function __construct(ElasticClient $client, string $index)
    {
        $this->client = $client;
        $this->index = $index;
    }

    public function close(): bool
    {
        return true;
    }

    public function process(Item $item): void
    {
        $message = $this->getFormattedItem($item);

        $params = [
            'index' => $this->index,
            'type' => '_doc',
            'body' => $message,
        ];

        $this->client->index($params);
    }
}

Elasticsearch logger formatter

<?php

namespace App\Common\Library\Logger;

use Phalcon\Helper\Json as JsonHelper;
use Phalcon\Logger\Formatter\AbstractFormatter;
use Phalcon\Logger\Item;

class ElasticFormatter extends AbstractFormatter
{
    public function format(Item $item): string
    {
        return JsonHelper::encode(
            [
                'level' => $item->getLevelName(), // severity
                'message' => $item->getMessage(),
                'context' => $item->getContext(), // all context params will be recorded here
                'created' => $item->getTime()->format('c')
            ]
        );
    }
}

You can choose any path to save it, just don't forget to change namespaces & uses

Dependency injection config, should be inserted into services.php

$di->setShared('elastic.client', function () {
    return \Elasticsearch\ClientBuilder::create()
        ->setHosts(['localhost']) // use your elastic host/ip
        ->build();
});

$di->setShared('logger', function () use ($di) {
    $formatter = new App\Common\Library\Logger\ElasticFormatter();

    $adapter = new App\Common\Library\Logger\ElasticAdapter($di->getShared('elastic.client'), 'app_' . date('YW')); // I use pattern to change log index every week, ex: app_202142
    $adapter->setFormatter($formatter);

    $logger  = new Phalcon\Logger(
        'messages',
        [
            'main' => $adapter,
        ]
    );

    return $logger;
});

Basically that's it what needs to be done for basic use of logger.

ES template:

Setup elasticsearch index template, documentation can be found here
I use following patterns for variable types:

• sName for string, ex: 'test'
• iCount for integers, ex: 123
• fPrice for floats, ex: 123.45
• oUserData, ex: {'ip': '127.0.0.1'}
• bIsNewUser, ex: true

PUT _index_template/app
{
  "index_patterns": [
    "app_*"
  ],
  "template": {
    "settings": {
      "index": {
        "number_of_shards": "4"
      }
    },
    "mappings": {
      "_source": {
        "enabled": true
      },
      "dynamic_templates": [
        {
          "strings": {
            "mapping": {
              "type": "keyword"
            },
            "match": "s*"
          }
        },
        {
          "ints": {
            "mapping": {
              "type": "long"
            },
            "match": "i*"
          }
        },
        {
          "floats": {
            "mapping": {
              "type": "float"
            },
            "match": "f*"
          }
        },
        {
          "objects": {
            "mapping": {
              "type": "object"
            },
            "match": "o*"
          }
        },
        {
          "boolean": {
            "mapping": {
              "type": "boolean"
            },
            "match": "b*"
          }
        }
      ],
      "properties": {
        "level": {
          "type": "keyword"
        },
        "created": {
          "type": "date"
        },
        "context": {
          "dynamic": true,
          "type": "object"
        },
        "message": {
          "type": "keyword"
        }
      }
    }
  },
  "composed_of": []
}

Non blocking logging

If you want to log events during web request, most likely you don't want to block response. To do it, I use transaction to write all logs to memory and on terminate script push records to elasticsearch. The example works on php-fpm, not sure about apache mod/fastcgi.
Don't use on cli long running scripts! Transaction keeps all records in memory, so it can consume all your free memory.

I added transaction begin just after autoloader:

/**
* Include Autoloader
*/
include APP_PATH . '/config/loader.php';

$this->di->getShared('logger')->getAdapter('main')->begin();

And transaction commit on terminate:

fastcgi_finish_request();
$this->di->getShared('logger')->getAdapter('main')->commit();

Example

$this->logger->info('pageOpened', [
    'sPageType' => 'category',
    'sTitle' => $category->name,
    'iCommentCount' => count($category->comments),
    'sReferer' => $this->request->getHTTPReferer(),
    'sSlug' => $category->slug,
    'sUserId' => $this->getUserId(),
    'sUserAgent' => $this->request->getUserAgent(),
    'bIsBot' => $this->isBot(),
]);

How the record looks like:

{
  "_index" : "app_202142",
  "_type" : "_doc",
  "_id" : "215hBn0BAC84nDsjrq1I",
  "_score" : 1.0,
  "_source" : {
    "level" : "INFO",
    "message" : "pageOpened",
    "context" : {
      "sPageType" : "category",
      "sTitle" : "Example blog",
      "iCommentCount" : 14,
      "sReferer" : "https://www.google.com",
      "sSlug" : "example-blog",
      "sUserId": "1234-5678-9012-3456",
      "sUserAgent": "Mozilla/5.0",
      "bIsBot": true
    },
    "created" : "2021-10-01T10:26:44+00:00"
  }
}

[niden]

Fantastic!!

Would you consider adding this in the incubator repo?

[DestinyMKas]

Who will maintain if I add it to incubator?

[niden]

All of us I guess. If you want to take the lead to maintain that one by all means.

[DestinyMKas]

I'll do it. Thanks for suggestion :)

[Jeckerson]

@DestinyMKas please reach me in Discord to talk about the details.