Forseti Issues, Not able to create module. Inventory is also showing zero count. Forseti Broken

[agupta-1]

Hi team,

We are facing issues after deploying Forseti server.

##Forseti Version:
"""Google Cloud Forseti."""

version = '2.25.0'
package_name = 'forseti-security'

--We are having an issue with Inventory and Model Module. Also scanner/violation is not working. Inventory is not getting collected. Showing as zero. Also not able to create Model. Even when i list it says no model found. When i try to create a module manually, it shows as broken.

--We also applied a policy however it is not being a being scanned and not showing any violations in security command center

forseti service

--It is running

--We deployed Forseti through terraform code.
--Server is deployed successfully however when we login into server and running inventory command it is not working. CSCC id was provided with correct service credentials.

Logs

--attached is the logs:
a) forseti.log
syslog file it is not allowing to upload as very big. please provide ftp

Please assist asap
[forseti.log](https://github.com/forseti-security/forseti-security/files/48463

forseti.log

[auto-comment]

Thank you for opening an issue. Our team's interrupts engineer will review your issue shortly.

Issue Resolution:

• [Interrupts Engineer] Triage / apply categorization labels
• [Interrupts Engineer] Verify / Reproduce the reported issue
• [Forseti Engineer] Perform root cause analysis
• Forseti Engineer] Add tasks and next steps to resolve this issue.

[gkowalski-google]

Hi @agupta-1, thanks for creating an issue. I recommend to upgrade to the latest Forset patch v2.25.1. There was an issue fixed with the version you are on; it might not resolve the issue, but don't want you to run into another issue. You can upgrade with Terraform by setting the version in the Forseti configuration to ~> 5.2.0.

I don't see any errors in the log that you sent that indicate why the inventory didn't get created. It doesn't look like you have configured the Service Account to use Domain wide delegation, which is required for scanning G Suite resources. If you do not want to scan those resources, then I recommend you also set the following variables in the Terraform config:

admin_disable_polling    = true
group_enabled                 = false
groups_settings_enabled = false

Please try those changes and then let me know if you are still running into issues. Instead of sharing the log file, if there are issues, you can search Stackdriver Logging for any errors coming from the VM during inventory, and include those error messages; just in case you accidentally share sensitive information.

[agupta-1]

Currently, reinstalling one more time and will check. Will upload the logs as requested

[agupta-1]

After re-installation same issue. Inventory is not getting collected even after 14 hours.

@forseti-server-vm-4e3a596a:~# forseti inventory list
{
"id": "1593694846257574",
"startTimestamp": "2020-07-02T13:00:46Z",
"schemaVersion": 1,
"status": "CREATED",
"warnings": "Your inventory contains warning message(s), please run command forseti inventory get 1593694846257574 for more information.",
"countObjects": 0,
"errors": ""
}

--Forseti model list is not listing any model
--When try to run inventory getting below status however it is always in hung state. not going beyond this screen.

• echo 'Running Forseti inventory.'
  Running Forseti inventory.
• forseti inventory create --import_as 20200702T130639

Forseti.log

---

2020-07-02 13:06:39,767 INFO [forseti-security][2.25.0] google.cloud.forseti.services.inventory.inventory(purge): retention_days is:
2020-07-02 13:06:39,767 INFO [forseti-security][2.25.0] google.cloud.forseti.services.inventory.inventory(purge): retention_days is not specified. Will use configuration default.
2020-07-02 13:06:39,767 INFO [forseti-security][2.25.0] google.cloud.forseti.services.inventory.inventory(purge): Purge is disabled. Nothing will be purged.
2020-07-02 13:06:40,760 INFO [forseti-security][2.25.0] google.cloud.forseti.services.cli(get_default_endpoint): Unable to read environment variable: FORSETI_CLIENT_CONFIG, will use the default endpoint instead, endpoint: localhost:50051
2020-07-03 01:47:54,150 WARNING [forseti-security][2.25.0] google.cloud.forseti.services.cli(load): IOError - trying to open configuration file located at /root/.forseti
2020-07-03 01:47:54,151 INFO [forseti-security][2.25.0] google.cloud.forseti.services.cli(get_default_endpoint): Unable to read environment variable: FORSETI_CLIENT_CONFIG, will use the default endpoint instead, endpoint: localhost:50051
2020-07-03 01:48:06,586 WARNING [forseti-security][2.25.0] google.cloud.forseti.services.cli(load): IOError - trying to open configuration file located at /root/.forseti
2020-07-03 01:48:06,587 INFO [forseti-security][2.25.0] google.cloud.forseti.services.cli(get_default_endpoint): Unable to read environment variable: FORSETI_CLIENT_CONFIG, will use the default endpoint instead, endpoint: localhost:50051

[agupta-1]

Hi Team,

Can you please check and confirm. The same version which i deployed was somehow working however after re-deployment again it is not collecting inventory and is not able to create module. I am already on latest version (2.25).

[agupta-1]

I did installed 2.25.1 i.e. the latest version however still facing the same issue. It is still not creating inventory and unable to create model. Please assist.

[gkowalski-google]

Hi @agupta-1, I am looking into this issue. It appears that others are having the same issue and I have reproduced it as well. Will provide an update as soon as I can track down what is going on. Similar ticket: #3770

[gkowalski-google]

Hi @agupta-1, I believe I have found an issue that might be the cause of Forseti not creating a model in your instance. More info here: #3774.

[agupta-1]

Can you give more clarity on "Update model creation to ignore the case of the permission.". How to achieve this task?

[gkowalski-google]

I will be providing a fix. I might have something available today from the main branch, if not then next week; and we will provide a patch release as well.

[agupta-1]

@gkowalski-google Waiting for your update. Thanks

[gkowalski-google]

Hi @agupta-1, the fix has been merged into the master branch. You can deploy this version with Terraform by setting forseti_version = "master"'. It is recommended to use the master branch version of the Forseti Terraform module as well, which can be done by using the module sourced from GitHub source = "git::https://github.com/forseti-security/terraform-google-forseti.git"`.

I am working on a patch release that will be out next week.

[agupta-1]

@gkowalski-google Hi as i can see it was merged in master branch, i did went for clean installation again. And guess what - Issue remains the same. Not sure how to resolve this now. Its almost a month.

[gkowalski-google]

@agupta-1 Can you provide your terraform configuration? And can you also provide the error that you are seeing?

[agupta-1]

@gkowalski-google Do you want me to provide main.tf file?
it is same error which i pasted earlier in case

[gkowalski-google]

@agupta-1 Yes, the main.tf would be helpful (please remove any sensitive info). To be sure, can you please filter the Stackdriver VM logs for errors? Some errors can be ignored, I am interested to know if you see anything like that contains sqlalchemy.exc.IntegrityError or Duplicate entry.

[agupta-1]

@gkowalski-google

I tried with both 5.2.0 and 5.2.1 version

Here is main.tf file:

module "forseti" {
source = "terraform-google-modules/forseti/google"
version = "5.2.1"
gsuite_admin_email = var.gsuite_admin_email
domain = var.domain
project_id = var.project_id
org_id = var.org_id
enable_write = "false"

Configure the Cloud Security Command Centre

cscc_source_id = var.cscc_source_id
cscc_violations_enabled = true

enabled config validator

config_validator_enabled = var.config_validator_enabled
config_validator_violations_should_notify = var.config_validator_violations_should_notify

enable policy library

commented out as tcp/22 is not available via nat gateway

policy_library_sync_enabled = var.policy_library_sync_enabled

policy_library_repository_url = var.policy_library_repository_url

Enter the type of instance that you want to create. Defaults to n1-standard-2 if left empty

server_type = var.server_type
client_type = var.client_type
cloudsql_type = var.cloudsql_type

Enter the locations where resources are to be created

storage_bucket_location = "#####"
bucket_cai_location = "#########"
server_region = "######"
client_region = "######"
cloudsql_region = "####"

Tell forseti to run every 30 mins

forseti_run_frequency = "0 */1 * * *"

This stops the client and server instance from getting a public ip

client_private = "true"
server_private = "true"

Enter the Network details if you want the Forseti cluster to be created in a specific location. If not Forseti will create its own network environment

network = var.network
subnetwork = var.subnetwork
network_project = var.network_project

Enter the tags for the instances created.

server_tags = ["removed"]
client_tags = ["removed"]

Enter an IP range in CIDR notation if you want to allow ssh access to the box. OPTIONAL

server_ssh_allow_ranges = []
client_ssh_allow_ranges = []

manage_rules_enabled = false
}

module "rules" {
source = "./modules/rules"

bucket_id = module.forseti.forseti-cai-storage-bucket
org_id = var.org_id
domain = var.domain
}

--Also as per stack driver logs:

Error 1:

[forseti-security][2.25.1] google.cloud.forseti.services.inventory.crawler(visit): (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) Traceback (most recent call last): File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect return fn() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 425, in connect return _ConnectionFairy._checkout(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 822, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 559, in checkout rec._checkin_failed(err) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 67, in exit compat.reraise(exc_type, exc_value, exc_tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 277, in reraise raise value File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 556, in checkout dbapi_connection = rec.get_connection() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 683, in get_connection self.__connect() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 701, in __connect connection = pool._invoke_creator(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 437, in connect return self.dbapi.connect(*cargs, **cparams) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/init.py", line 94, in Connect return Connection(*args, **kwargs) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 325, in init self.connect() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 599, in connect self._request_authentication() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 861, in _request_authentication auth_packet = self._read_packet() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 657, in _read_packet packet_header = self._read_bytes(4) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 707, in _read_bytes CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/services/inventory/crawler.py", line 126, in visit self.write(resource) File "/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/services/inventory/crawler.py", line 148, in write self.config.storage.write(resource) File "/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/services/inventory/storage.py", line 995, in write result = self.engine.execute(Inventory.table.insert(), resource_row) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2143, in execute connection = self.contextual_connect(close_with_result=True) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2192, in contextual_connect self._wrap_pool_connect(self.pool.connect, None), File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2232, in _wrap_pool_connect e, dialect, self File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1528, in _handle_dbapi_exception_noconnection util.raise_from_cause(sqlalchemy_exception, exc_info) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 296, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 276, in reraise raise value.with_traceback(tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 2228, in _wrap_pool_connect return fn() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 425, in connect return _ConnectionFairy._checkout(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 822, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 559, in checkout rec._checkin_failed(err) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 67, in exit compat.reraise(exc_type, exc_value, exc_tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 277, in reraise raise value File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 556, in checkout dbapi_connection = rec.get_connection() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 683, in get_connection self.__connect() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 701, in __connect connection = pool._invoke_creator(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/strategies.py", line 114, in connect return dialect.connect(*cargs, **cparams) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 437, in connect return self.dbapi.connect(*cargs, **cparams) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/init.py", line 94, in Connect return Connection(*args, **kwargs) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 325, in init self.connect() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 599, in connect self._request_authentication() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 861, in _request_authentication auth_packet = self._read_packet() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 657, in _read_packet packet_header = self._read_bytes(4) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 707, in _read_bytes CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)

Error2:
[forseti-security][2.25.1] google.cloud.forseti.services.model.importer.importer(_flush_session): Unexpected SQLAlchemyError occurred during model creation. Traceback (most recent call last): File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1236, in _execute_context cursor, statement, parameters, context File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 536, in do_execute cursor.execute(statement, parameters) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/cursors.py", line 170, in execute result = self._query(query) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/cursors.py", line 328, in _query conn.query(q) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 517, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 732, in _read_query_result result.read() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 1075, in read first_packet = self.connection._read_packet() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 684, in _read_packet packet.check_error() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/protocol.py", line 220, in check_error err.raise_mysql_exception(self._data) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/err.py", line 109, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (forseti_security.c984caa3da4033d08ef35c3c25afa37d_bindings, CONSTRAINT c984caa3da4033d08ef35c3c25afa37d_bindings_ibfk_2 FOREIGN KEY (role_name) REFERENCES c984caa3da4033d08ef35c3c25af)') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/forseti_security-2.25.1-py3.6.egg/google/cloud/forseti/services/model/importer/importer.py", line 230, in _flush_session self.session.flush() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2446, in flush self._flush(objects) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2584, in _flush transaction.rollback(_capture_exception=True) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 67, in __exit__ compat.reraise(exc_type, exc_value, exc_tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 277, in reraise raise value File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 2544, in _flush flush_context.execute() File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 416, in execute rec.execute(self) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 583, in execute uow, File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 245, in save_obj insert, File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 1116, in _emit_insert_statements statement, params File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 980, in execute return meth(self, multiparams, params) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement distilled_params, File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1240, in _execute_context e, statement, parameters, cursor, context File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception util.raise_from_cause(sqlalchemy_exception, exc_info) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 296, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 276, in reraise raise value.with_traceback(tb) File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1236, in _execute_context cursor, statement, parameters, context File "/home/ubuntu/forseti-security/.eggs/SQLAlchemy-1.2.18-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 536, in do_execute cursor.execute(statement, parameters) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/cursors.py", line 170, in execute result = self._query(query) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/cursors.py", line 328, in _query conn.query(q) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 517, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 732, in _read_query_result result.read() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 1075, in read first_packet = self.connection._read_packet() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/connections.py", line 684, in _read_packet packet.check_error() File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/protocol.py", line 220, in check_error err.raise_mysql_exception(self._data) File "/home/ubuntu/forseti-security/.eggs/PyMySQL-0.9.3-py3.6.egg/pymysql/err.py", line 109, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.IntegrityError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (forseti_security.c984caa3da4033d08ef35c3c25afa37d_bindings, CONSTRAINT c984caa3da4033d08ef35c3c25afa37d_bindings_ibfk_2 FOREIGN KEY (role_name) REFERENCES c984caa3da4033d08ef35c3c25af)') [SQL: 'INSERT INTO c984caa3da4033d08ef35c3c25afa37d_bindings (resource_type_name, role_name) VALUES (%(resource_type_name)s, %(role_name)s)'] [parameters: {'resource_type_name': 'project/sys-68073348388346939442958192', 'role_name': 'roles/owner'}] (Background on this error at: http://sqlalche.me/e/gkpj)

[gkowalski-google]

@agupta-1 The forseti_version var is missing, please set this to master. I am going to be releasing the patch release for this version today, so it would probably make sense to wait for that. Once that is published, then you won't need to set forseti_version and you would just need to update version from "5.2.1" to "5.2.2".

[agupta-1]

@gkowalski-google I am fine to re-deploy. Hence will wait for your patch.
Also to re-confirm are you saying i need to enter this variable in main.tf file for now as below

module "forseti" {
source = "terraform-google-modules/forseti/google"
version = "5.2.1"
forseti_version = "master"
gsuite_admin_email = var.gsuite_admin_email
domain = var.domain
project_id = var.project_id
org_id = var.org_id
enable_write = "false"

As yoy can see version is already set to 5.2.1 Do i need to remove this one while entering forseti_version = "master"

[gkowalski-google]

@agupta-1 Sorry for the confusion. Once patch is released then you can just set version = "5.2.2" and remove the forseti_master var from the config.

[gkowalski-google]

@agupta-1 The latest Forseti release is out to fix the model creation issue. Please upgrade at your convenience and let me know if the your issue is resolved. You can upgrade with Terraform by setting the Forseti module version to 5.2.2 or ~> 5.2.0.

[agupta-1]

@gkowalski-google Did updated however same issue. Apologies however i am sure that something is wrong in the code itself which is not allowing to create the model.

If you want i can re-deploy complete forseti once again. But i have a strong feeling it will produce the same results.

[gkowalski-google]

Hi @agupta-1, can you please send any errors that you are seeing in the Stackdriver logs? Can you also please schedule a meeting with me so that we can troubleshoot? My email is gkowalski at google.com. I am available now if you are.

[agupta-1]

@gkowalski-google Hi, Sure let me do a fresh install one more time with version 5.2.2 (by changing the version in main.tf) so that it will be easier for you to troubleshoot. Let me drop you a direct email to connect with the schedule.
Thanks

[agupta-1]

@gkowalski-google
Hi Gregg,

Please find the update as of now:

1. I checked inventory is showing as created,
   ~$ forseti inventory list
   {
   "id": "1596085245764606",
   "startTimestamp": "2020-07-30T05:00:46Z",
   "completeTimestamp": "2020-07-31T07:43:48Z",
   "schemaVersion": 1,
   "countObjects": 362184,
   "status": "SUCCESS",
   "warnings": "Your inventory contains warning message(s), please run command forseti inventory get 1596085245764606 for more information.",
   "errors": ""

2. I checked model is showing as created as partial success:

$ forseti model list
{
"handle": "3389c80630e665466c8e4ac8abfb2219",
"name": "20200730T050044",
"status": "PARTIAL_SUCCESS",
"description": "{"gsuite_enabled": false, "pristine": true, "source": "inventory", "source_info": {"inventory_index_id": 1596085245764606}, "source_root": "organization/xxxxxxxxxx"}",
"message": "228588",
"createdAt": "31 July 2020 - 07:43:48"

I am nor sure if the above model is completed or not. Because Still we are not receiving any notification in GCP Security center CSCC even though it is enabled.

Note: I increased forseti security resources to n1-standard-8 as you mentioned earlier.

Let me know.

[gkowalski-google]

Hi @agupta-1, sorry I missed your update. Can you run forseti model get 20200730T050044 and see what the warnings/errors are? You can also filter the server VM logs for any errors during inventory/model creation. Let me know if you can provide any of those details.

[gkowalski-google]

@agupta-1 Were you able to see what the model issues were? A partial success for model creation is fairly typical. There are some warnings that get reported when Forseti is trying to process the roles, and these do not break anything. If you are still not seeing any CSCC findings, can you please check the server VM logs for any errors and confirm you followed all steps to set up CSCC integration?