Skip to content
This repository has been archived by the owner on Dec 15, 2020. It is now read-only.

Kolide Fleet, incoming host indefinitely #2297

Open
TriflesT opened this issue Sep 6, 2020 · 13 comments
Open

Kolide Fleet, incoming host indefinitely #2297

TriflesT opened this issue Sep 6, 2020 · 13 comments
Labels
Needs: Update

Comments

@TriflesT
Copy link

TriflesT commented Sep 6, 2020 •
edited

What version of fleet are you using (fleet version --full)?

image

What operating system are you using?

Ubuntu 16.04 for kolide, Windows Server 2012/Windows 10 hosts

What did you do?

I ran the following command to add the windows host with the parameters filled in:

./linux/launcher --enroll_secret=[secret from kolide] --hostname=[ip]:8080 --root_directory=[directory]/osq --insecure

My windows hosts have osquery version 4.4.0 installed.

What did you expect to see?

I expected to see the host added to kolide fleet

What did you see instead?

image
@zwass
Copy link
Contributor

zwass commented Sep 17, 2020

Can you run Launcher with the --debug flag and see if you can see any errors? Please paste the logs here.

@TriflesT
Copy link
Author

TriflesT commented Sep 20, 2020 •
edited

Here is the launcher ran with the --debug flag.

image

@zwass
Copy link
Contributor

zwass commented Sep 21, 2020

I don't see anything unusual in the logs there. Is 192.168.1.61 (that osquery is connecting to) the same server as localhost (in your browser with the Fleet UI)? Can you connect to the DB and select * from hosts?

@TriflesT
Copy link
Author

TriflesT commented Sep 21, 2020 •
edited

Yes 192.168.1.61 is the localhost server:
image

Here is the result of the query:

image

@zwass
Copy link
Contributor

zwass commented Sep 24, 2020

Can you actually run that query against your MySQL database that Fleet is connected to? I want to see what Fleet has stored about the hosts.

Also, please paste the actual text rather than screenshots. Thanks!

@TriflesT
Copy link
Author

mysql> SELECT * FROM hosts
-> ;
+----+--------------------------------------+---------------------+---------------------+------------+---------+---------------------+----------------------------------+-----------+--------------------------------------+----------+-----------------+---------------+-------+---------------+-----------+----------------+-----------------+----------+-------------+------------------------------------------+--------------------+-------------------+-----------------+----------------+------------------+-----------------+---------------+---------------+---------------------+----------------------+-------------------+--------------------+------------+-------------------+---------------------+------------+--------------------+
| id | osquery_host_id | created_at | updated_at | deleted_at | deleted | detail_update_time | node_key | host_name | uuid | platform | osquery_version | os_version | build | platform_like | code_name | uptime | physical_memory | cpu_type | cpu_subtype | cpu_brand | cpu_physical_cores | cpu_logical_cores | hardware_vendor | hardware_model | hardware_version | hardware_serial | computer_name | primary_ip_id | seen_time | distributed_interval | logger_tls_period | config_tls_refresh | primary_ip | primary_mac | label_update_time | additional | enroll_secret_name |
+----+--------------------------------------+---------------------+---------------------+------------+---------+---------------------+----------------------------------+-----------+--------------------------------------+----------+-----------------+---------------+-------+---------------+-----------+----------------+-----------------+----------+-------------+------------------------------------------+--------------------+-------------------+-----------------+----------------+------------------+-----------------+---------------+---------------+---------------------+----------------------+-------------------+--------------------+------------+-------------------+---------------------+------------+--------------------+
| 1 | f7ec49d1-1ee4-428e-a5e8-3ac36f2072b1 | 2020-08-04 13:16:41 | 2020-09-02 11:15:43 | NULL | 0 | 2020-08-04 15:16:53 | ccHJ5ph6XRd2TX0H8ZLEep1X5gKH18QL | ubuntu | 559225b7-8519-48e2-a029-29e50e666029 | ubuntu | 4.4.0 | Ubuntu 16.4.0 | | debian | | 15053000000000 | 4143108096 | x86_64 | 158 | Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz | 4 | 4 | | | | | ubuntu | NULL | 2020-09-02 11:15:44 | 10 | 10 | 300 | 10.0.2.15 | 08:00:27:26:69:1d | 2020-08-04 15:16:53 | {} | default |
| 3 | 76973b88-a440-4286-9655-7e0537fe7635 | 2020-09-02 11:12:31 | 2020-09-02 11:15:45 | NULL | 0 | 1970-01-02 07:30:00 | 8Ij41jl4bE5vdmBFS2gpJ3xunuPzxV27 | | | | | | | | | 0 | 0 | | | | 0 | 0 | | | | | | NULL | 2020-09-02 11:15:45 | 10 | 10 | 0 | | | 1970-01-02 07:30:00 | NULL | default |
+----+--------------------------------------+---------------------+---------------------+------------+---------+---------------------+----------------------------------+-----------+--------------------------------------+----------+-----------------+---------------+-------+---------------+-----------+----------------+-----------------+----------+-------------+------------------------------------------+--------------------+-------------------+-----------------+----------------+------------------+-----------------+---------------+---------------+---------------------+----------------------+-------------------+--------------------+------------+-------------------+---------------------+------------+--------------------+
2 rows in set (0.00 sec)

@zwass
Copy link
Contributor

zwass commented Oct 1, 2020

We can see that host (id 3) in the database, and the detail_update_time is old enough that it should receive the detail queries. Your log screenshot indicates that it is not receiving those.

Does that same host work if you connect via plain osquery rather than Launcher?

@kevensen
Copy link

kevensen commented Oct 4, 2020 •
edited

I am actually seeing similar behavior with all hosts enrolled.

fleet - version 3.1.0
  branch: 	HEAD
  revision: 	c6ce648fef3bb39b6e604333ec47cff0e625ff8e
  build date: 	
  build user: 	root
  go version: 	go1.11.6

My osquery host is Ubuntu and running version osqueryd version 4.4.0

I did not use the launcher.

@zwass
Copy link
Contributor

zwass commented Oct 5, 2020

@kevensen can you please run osqueryd with the --verbose --tls_dump flags and paste the output here?

Also, please feel free to join the #kolide channel in osquery Slack where we can have a quicker back-and-forth.

@kevensen
Copy link

kevensen commented Oct 5, 2020 

Oct 05 10:41:17 pop-os.home.fakedomain.com osqueryd[3252]:   "error": "failed to ingest result: ingesting query kolide_detail_query_system_info: strconv.Atoi: parsing \"67314212864\": value out of range"
Oct 05 10:41:17 pop-os.home.fakedomain.com osqueryd[3252]: }
Oct 05 10:41:19 pop-os.home.fakedomain.com osqueryd[3252]: {"queries":{"kolide_detail_query_osquery_flags":[{"name":"config_refresh","value":"10"},{"name":"distributed_interval","value":"10"},{"name":"logger_tls_period","value":"10"}],"kolide_detail_query_osquery_info":[{"pid":"3252","uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","instance_id":"9a60d2e0-2d19-4012-928e-a8b34143e4f5","version":"4.4.0","config_hash":"b01efbf375ac6767f259ae98751154fef727ce35","config_valid":"1","extensions":"active","build_platform":"1","build_distro":"centos7","start_time":"1601919469","watcher":"3250","platform_mask":"9"}],"kolide_detail_query_system_info":[{"hostname":"pop-os.home.fakedomain.com","uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","cpu_type":"x86_64","cpu_subtype":"165","cpu_brand":"Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz","cpu_physical_cores":"8","cpu_logical_cores":"16","cpu_microcode":"0xc8","physical_memory":"67314212864","hardware_vendor":"System76","hardware_model":"Oryx Pro","hardware_version":"oryp6","hardware_serial":"123456789","board_vendor":"System76","board_model":"Oryx Pro","board_version":"oryp6","board_serial":"123456789","computer_name":"pop-os.home.fakedomain.com","local_hostname":"pop-os.home.fakedomain.com"}],"kolide_detail_query_uptime":[{"days":"0","hours":"0","minutes":"4","seconds":"43","total_seconds":"283"}],"kolide_label_query_6":[{"1":"1"}],"kolide_detail_query_network_interface":[{"address":"192.168.0.193","mac":"80:fa:5b:7f:f8:dd"},{"address":"fe80::54ca:d36:452c:d7b2%enp41s0","mac":"80:fa:5b:7f:f8:dd"},{"address":"192.168.86.122","mac":"c8:58:c0:24:fb:f3"},{"address":"fe80::eee8:69f4:a59:2460%wlp0s20f3","mac":"c8:58:c0:24:fb:f3"},{"address":"127.0.0.1","mac":"00:00:00:00:00:00"},{"address":"::1","mac":"00:00:00:00:00:00"},{"address":"172.16.23.1","mac":"00:50:56:c0:00:01"},{"address":"fe80::250:56ff:fec0:1%vmnet1","mac":"00:50:56:c0:00:01"},{"address":"172.16.194.1","mac":"00:50:56:c0:00:08"},{"address":"fe80::250:56ff:fec0:8%vmnet8","mac":"00:50:56:c0:00:08"}],"kolide_detail_query_os_version":[{"name":"Pop!_OS","version":"20.04 LTS","major":"20","minor":"4","patch":"0","build":"","platform":"pop","platform_like":"ubuntu debian","codename":"focal","arch":"x86_64I1005 10:41:19.127517  3261 config.cpp:1213] Refreshing configuration state
Oct 05 10:41:19 pop-os.home.fakedomain.com osqueryd[3252]: I1005 10:41:19.127760  3261 tls.cpp:253] TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/config
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: I1005 10:41:21.651185  3263 tls.cpp:253] TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/log
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: "}],"kolide_label_query_9":[]},"statuses":{"kolide_detail_query_osquery_flags":0,"kolide_detail_query_osquery_info":0,"kolide_detail_query_system_info":0,"kolide_detail_query_uptime":0,"kolide_label_query_6":0,"kolide_detail_query_network_interface":0,"kolide_detail_query_os_version":0,"kolide_label_query_9":0},"messages":{"kolide_detail_query_osquery_flags":"","kolide_detail_query_osquery_info":"","kolide_detail_query_system_info":"","kolide_detail_query_uptime":"","kolide_label_query_6":"","kolide_detail_query_network_interface":"","kolide_detail_query_os_version":"","kolide_label_query_9":""},"node_key":"v1v7cCrw2NgSKW9QteZovCHY98fwJd5A"}
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: {
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   "error": "failed to ingest result: ingesting query kolide_detail_query_system_info: strconv.Atoi: parsing \"67314212864\": value out of range"
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: }
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: {"node_key":"v1v7cCrw2NgSKW9QteZovCHY98fwJd5A"}
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: {
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   "decorators": {
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "load": [
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:       "SELECT uuid AS host_uuid FROM system_info;",
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:       "SELECT hostname AS hostname FROM system_info;"
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     ]
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   },
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   "options": {
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "disable_distributed": false,
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "distributed_interval": 10,
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "distributed_plugin": "tls",
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "distributed_tls_max_attempts": 3,
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "logger_plugin": "tls",
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "logger_tls_endpoint": "/api/v1/osquery/log",
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "logger_tls_period": 10,
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:     "pack_delimiter": "/"
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]:   }
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: }
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: {"data":[{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:11 2020 UTC","unixTime":"1601919671","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/log","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/distributed/read","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_network_interface: select address, mac\n                        from interface_details id join interface_addresses ia\n                               on ia.interface = id.interface where length(mac) > 0\n                               order by (ibytes + obytes) desc","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_os_version: select * from os_version limit 1","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_osquery_flags: select name, value from osquery_flags where name in (\"distributed_interval\", \"config_tls_refresh\", \"config_refresh\", \"logger_tls_period\")","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_osquery_info: select * from osquery_info limit 1","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece7738I1005 10:41:21.740239  3266 tls.cpp:253] TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/distributed/write
Oct 05 10:41:21 pop-os.home.fakedomain.com osqueryd[3252]: 9-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_system_info: select * from system_info limit 1","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"smbios_tables.cpp","line":"104","message":"Reading SMBIOS from sysfs DMI node","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_detail_query_uptime: select * from uptime limit 1","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_label_query_6: select 1;","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"distributed.cpp","line":"120","message":"Executing distributed query: kolide_label_query_9: select 1 from os_version where platform = 'centos' or name like '%centos%'","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:15 2020 UTC","unixTime":"1601919675","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/distributed/write","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:17 2020 UTC","unixTime":"1601919677","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/distributed/write","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:19 2020 UTC","unixTime":"1601919679","severity":"0","filename":"config.cpp","line":"1213","message":"Refreshing configuration state","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}},{"hostIdentifier":"ece77389-68a0-4351-8f7f-7f3ad20f2911","calendarTime":"Mon Oct  5 17:41:19 2020 UTC","unixTime":"1601919679","severity":"0","filename":"tls.cpp","line":"253","message":"TLS/HTTPS POST request to URI: https://kolide.home.fakedomain.com/api/v1/osquery/config","version":"4.4.0","decorations":{"host_uuid":"ece77389-68a0-4351-8f7f-7f3ad20f2911","hostname":"pop-os.home.fakedomain.com"}}],"log_type":"status","node_key":"v1v7cCrw2NgSKW9QteZovCHY98fwJd5A"}

@zwass
Copy link
Contributor

zwass commented Oct 5, 2020

@kevensen Did you by chance make a custom build of Fleet for a 32 bit architecture? Looks like your host's memory value is overflowing a 32 bit int. We can certainly fix that by explicitly specifying 64 bit integers but I am wondering why/how you ended up in this position.

@zwass zwass mentioned this issue Oct 5, 2020
Closed
@kevensen
Copy link

kevensen commented Oct 5, 2020

I was actually thinking that as well. In my home lab I am attempting to run Fleet on a Raspberry Pi 3b+. Obviously not a production environment but an intellectual curiosity. So yeah, 32-bit.

@zwass
Copy link
Contributor

zwass commented Oct 5, 2020 •
edited

@kevensen let's follow up in #2314 as I think this is unrelated to @TriflesT's issue (they seem to be using the official build).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Needs: Update
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zwass @kevensen @TriflesT