Vector Nightly (2022-05) version LUA is not calling libraries

[mindhack03d]

Dear Community

I tested the Nightly version (2022-05) related to encrypt function and it is working fine.
I added two "transforms" with "type = "remap"". and I have the following warning. I don't have more information about the warning.

2022-05-02T16:37:11.831091Z  WARN vector::transforms::remap: VRL compilation warning. warnings=
2022-05-02T16:37:11.831151Z  WARN vector::transforms::remap: VRL compilation warning. warnings=

However, when I tried to call LUA Libraries like httpclient, the result is:

2022-05-02T16:38:11.442467Z ERROR transform{component_kind="transform" component_id=t_lua component_type=lua
 component_name=t_lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess 
{ source: RuntimeError("[string \"src/transforms/lua/v2/mod.rs:162:17\"]:17: attempt to index a nil value (global 'http_request')
\nstack traceback:\n\t[C]: in metamethod 'index'\n\t[string \"src/transforms/lua/v2/mod.rs:162:17\"]:17: in function 'process'") }
 error_type="script_failed" error_code="runtime_error_hook_process" stage="processing" internal_log_rate_secs=30

, but it is not the same result on the Vector 0.21.1.

In order to reproduce the scenario, you can install the following packages:

BASIC CONFIGURATION OF THE VM

1. Install CentOs7 (CentOS-7-x86_64-Minimal-1810)
2. Install the following libraries in CentOs7

$ yum install -y epel-release
$ yum check-update
$ yum search epel
$ yum install -y epel-rpm-macros
$ yum check-update
$ yum install -y git gcc g++ make readline-devel unzip
$ yum install -y wget cmake yum-utils vim
$ yum install -y pcre2 pcre2-devel

3. Install LUA 5.4, as part of the OS.

$ curl -R -O http://www.lua.org/ftp/lua-5.4.3.tar.gz
$ tar -zxf lua-5.4.3.tar.gz
$ cd lua-5.4.3
$ make linux test
$ sudo make install
$ cd /usr/bin/
$ mv lua lua5.1
$ ln -s /usr/local/bin/lua

4. Install LUAROCK and configure it to user version 5.4

$ wget https://luarocks.org/releases/luarocks-3.8.0.tar.gz --no-check-certificate
$ tar zxpf luarocks-3.8.0.tar.gz
$ cd luarocks-3.8.0
$ ./configure --lua-version=5.4
$ make
$ make install
$ cd /usr/share/lua
$ ln -s /usr/local/share/lua/5.4/

5. Install LUA libraries, and fix it.

$ luarocks install lunajson
$ luarocks install luasocket
$ luarocks install inspect
$ luarocks install httpclient
$ sed -i -e 's,receive(),receive("*l"),g' /usr/local/share/lua/5.4/socket/tp.lua
$ sed -i -e 's,receive(),receive("*l"),g' /usr/local/share/lua/5.4/socket/http.lua

INSTALL AND TEST ON VECTOR 0.21.1

1. Install Vector 0.21.1

$ curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.rpm.sh' | sudo -E bash
$ sudo yum install vector

2. Test the following configuration

[sources.input_file]
type = "file"
include = ["/root/log"]
read_from = "end"

[transforms.t_json]
type = "remap"
inputs = [ "input_file" ]
source = """
. = parse_json!(.message)
"""

[transforms.t_lua]
type = "lua"
inputs = [ "t_json" ]
version = "2"
source = '''
-- LIBRARIES
ljson = require("lunajson")
key = "1a2b3c4d5e6f7g8h"
iv  = "1234ABCD1234ABCD"

function process(event, emit)
        -- HTTP Request
        json = require("lunajson")
        http_request = require("httpclient").new()
        MY_URL='http://www.google.com'
        KEY_PATH="/"
        headers={["User-Agent"]="Lua"}
        http_response = http_request:get(MY_URL .. KEY_PATH, headers)
        event.log.key = key
        event.log.iv = iv
        event.log.response = http_response
        emit(event)
end
'''
hooks.process = "process"

[transforms.t_encrypt]
type = "remap"
inputs = ["t_lua"]
source = '''
. = .
'''

[sinks.serverout]
type = "console"
inputs = ["t_encrypt"]
target = "stdout"
encoding.codec = "json"

3. Validating the configuration

$ vector validate vector.toml
√ Loaded ["vector.toml"]
√ Component configuration
√ Health check "serverout"
--------------------------
                 Validated

4. Test file, content of the test file.

{"context":{"correlationId":"00MYSECRETPASSWORD99"}}

5. Send the request

$ cd /root
$ cat test >> log

6. Result on Vector.

$ vector --config vector.toml
2022-05-02T17:03:39.779859Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-05-02T17:03:39.779916Z  INFO vector::app: Loading configs. paths=["vector.toml"]
2022-05-02T17:03:39.783051Z  INFO vector::topology::running: Running healthchecks.
2022-05-02T17:03:39.783177Z  INFO vector: Vector has started. debug="false" version="0.21.1" arch="x86_64" build_id="18787c0 2022-04-22"
2022-05-02T17:03:39.783194Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2022-05-02T17:03:39.783301Z  INFO vector::topology::builder: Healthcheck: Passed.
2022-05-02T17:03:39.783367Z  INFO source{component_kind="source" component_id=input_file component_type=file component_name=input_file}: vector::sources::file: Starting file server. include=["/root/log"] exclude=[]
2022-05-02T17:03:39.783909Z  INFO source{component_kind="source" component_id=input_file component_type=file component_name=input_file}:file_server: file_source::checkpointer: Loaded checkpoint data.
2022-05-02T17:03:39.784163Z  INFO source{component_kind="source" component_id=input_file component_type=file component_name=input_file}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/root/log file_position=583
{"context":{"correlationId":"00MYSECRETPASSWORD99"},"iv":"1234ABCD1234ABCD","key":"1a2b3c4d5e6f7g8h","response":{"body":"<!doctype html>... "status_line":"HTTP/1.1 200 OK"}}

7. The request worked as expected.

INSTALL AND TEST ON VECTOR NIGHTLY (2022-05)

1. Install Vector Nightly

$ cd /root/
$ mkdir -p vector && curl -sSfL --proto '=https' --tlsv1.2 https://packages.timber.io/vector/nightly/latest/vector-nightly-x86_64-unknown-linux-musl.tar.gz | tar xzf - -C vector --strip-components=2

Add the vector path to PATH. Edit the file ~/.bash_profile and

$ vim ~/.bash_profile
PATH=$PATH:$HOME/bin:/root/vector/bin
$ source ~/.bash_profile
# Add new Folder
$ cd /root/vector
$ mkdir lib

2. Test the following configuration

data_dir = "/root/vector/lib"

[sources.input_file]
type = "file"
include = ["/root/log"]
read_from = "end"

[transforms.t_json]
type = "remap"
inputs = [ "input_file" ]
source = """
. = parse_json!(.message)
"""

[transforms.t_lua]
type = "lua"
inputs = [ "t_json" ]
version = "2"
source = '''
-- LIBRARIES
ljson = require("lunajson")
key = "1a2b3c4d5e6f7g8h"
iv  = "1234ABCD1234ABCD"

function process(event, emit)
        -- HTTP Request
        json = require("lunajson")
        http_request = require("httpclient").new()
        MY_URL='http://www.google.com'
        KEY_PATH="/"
        headers={["User-Agent"]="Lua"}
        http_response = http_request:get(MY_URL .. KEY_PATH, headers)
        event.log.key = key
        event.log.iv = iv
        event.log.response = http_response
        emit(event)
end
'''
hooks.process = "process"

[transforms.t_encrypt]
type = "remap"
inputs = ["t_lua"]
source = '''
. = .
'''

[sinks.serverout]
type = "console"
inputs = ["t_encrypt"]
target = "stdout"
encoding.codec = "json"

3. Validating the configuration

$ vector validate vector.toml
√ Loaded ["vector.toml"]
2022-05-02T17:07:55.922143Z  WARN vector::transforms::remap: VRL compilation warning. warnings=

2022-05-02T17:07:55.922215Z  WARN vector::transforms::remap: VRL compilation warning. warnings=

√ Component configuration
√ Health check "serverout"
--------------------------
                 Validated

4. Test file, content of the test file.

{"context":{"correlationId":"00MYSECRETPASSWORD99"}}

5. Send the request

$ cd /root
$ cat test >> log

6. Result on Vector.

$ vector --config vector.toml
2022-05-02T17:09:05.691292Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-05-02T17:09:05.691349Z  INFO vector::app: Loading configs. paths=["vector.toml"]
2022-05-02T17:09:05.693939Z  WARN vector::transforms::remap: VRL compilation warning. warnings=

2022-05-02T17:09:05.694016Z  WARN vector::transforms::remap: VRL compilation warning. warnings=

2022-05-02T17:09:05.694090Z  INFO vector::topology::running: Running healthchecks.
2022-05-02T17:09:05.694124Z  INFO vector::topology::builder: Healthcheck: Passed.
2022-05-02T17:09:05.694196Z  INFO vector: Vector has started. debug="false" version="0.22.0" arch="x86_64" build_id="5099040 2022-04-29"
2022-05-02T17:09:05.694190Z  INFO source{component_kind="source" component_id=input_file component_type=file component_name=input_file}: vector::sources::file: Starting file server. include=["/root/log"] exclude=[]
2022-05-02T17:09:05.694213Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2022-05-02T17:09:05.694837Z  INFO source{component_kind="source" component_id=input_file component_type=file component_name=input_file}:file_server: file_source::checkpointer: Loaded checkpoint data.
2022-05-02T17:09:05.695114Z  INFO source{component_kind="source" component_id=input_file component_type=file component_name=input_file}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/root/log file_position=477
2022-05-02T17:09:15.982165Z ERROR transform{component_kind="transform" component_id=t_lua component_type=lua component_name=t_lua}: vector::internal_events::lua: Error in lua script; discarding event. error=RuntimeErrorHooksProcess { source: RuntimeError("[string \"src/transforms/lua/v2/mod.rs:162:17\"]:13: attempt to index a nil value (global 'http_request')\nstack traceback:\n\t[C]: in metamethod 'index'\n\t[string \"src/transforms/lua/v2/mod.rs:162:17\"]:13: in function 'process'") } error_type="script_failed" error_code="runtime_error_hook_process" stage="processing" internal_log_rate_secs=30

7. LUA is not working on Vector Nightly, but in Vector 0.21.1 is working fine.

You could help us to validate, why LUA is working on Vector 0.21.1 but it is not the same case on Vector Nightly (2022-05) or maybe I have a bad configuration in Vector Nightly.

Best Regards.

[jszwedko]

Hi @mindhack03d ! Thanks for opening this. I'm not aware of anything that changed with the Lua transform since 0.21, but I opened this issue to investigate: #12522