[Bug]: Encoding issue with Dashboard on Prowler 4.1.0 - Aces High

[irboi746]

Steps to Reproduce

After running the scan, there is an encoding error when trying to open the dashboard using python prowler.py dashboard

The error is as below:

Traceback (most recent call last):
  File "C:\Users\Test\Desktop\prowler-4.1.0\prowler.py", line 7, in <module>
    sys.exit(prowler())
  File "C:\Users\Test\Desktop\prowler-4.1.0\prowler\__main__.py", line 64, in prowler
    from dashboard.__main__ import dashboard
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\__main__.py", line 28, in <module>
    dashboard = dash.Dash(
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 545, in __init__
    self.init_app()
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 623, in init_app
    self.enable_pages()
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 2130, in enable_pages
    _import_layouts_from_pages(self.config.pages_folder)
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\_pages.py", line 435, in _import_layouts_from_pages
    spec.loader.exec_module(page_module)
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\pages\compliance.py", line 43, in <module>
    num_rows = sum(1 for row in reader)
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\pages\compliance.py", line 43, in <genexpr>
    num_rows = sum(1 for row in reader)
  File "C:\Python310\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 5171: invalid start byte

When attempting to change encoding_format in \prowler-4.1.0\dashboard\config.py from utf-8 to windows-1252 or ISO-88659-1 we will get a new error as follows:

Traceback (most recent call last):
  File "C:\Users\Test\Desktop\prowler-4.1.0\prowler.py", line 7, in <module>
    sys.exit(prowler())
  File "C:\Users\Test\Desktop\prowler-4.1.0\prowler\__main__.py", line 64, in prowler
    from dashboard.__main__ import dashboard
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\__main__.py", line 28, in <module>
    dashboard = dash.Dash(
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 545, in __init__
    self.init_app()
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 623, in init_app
    self.enable_pages()
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 2130, in enable_pages
    _import_layouts_from_pages(self.config.pages_folder)
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\_pages.py", line 435, in _import_layouts_from_pages
    spec.loader.exec_module(page_module)
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\pages\overview.py", line 92, in <module>
    data = load_csv_files(csv_files)
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\pages\overview.py", line 80, in load_csv_files
    df = pd.read_csv(file, sep=";", on_bad_lines="skip")
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\pandas\io\parsers\readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\pandas\io\parsers\readers.py", line 626, in _read
    return parser.read(nrows)
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\pandas\io\parsers\readers.py", line 1923, in read
    ) = self._engine.read(  # type: ignore[attr-defined]
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 234, in read
    chunks = self._reader.read_low_memory(nrows)
  File "parsers.pyx", line 838, in pandas._libs.parsers.TextReader.read_low_memory
  File "parsers.pyx", line 905, in pandas._libs.parsers.TextReader._read_rows
  File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows
  File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status
  File "parsers.pyx", line 2053, in pandas._libs.parsers.raise_parser_error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position 175279: invalid start byte

Expected behavior

Dashboard should be running

Actual Result with Screenshots or Logs

First Error

Traceback (most recent call last):
  File "C:\Users\Test\Desktop\prowler-4.1.0\prowler.py", line 7, in <module>
    sys.exit(prowler())
  File "C:\Users\Test\Desktop\prowler-4.1.0\prowler\__main__.py", line 64, in prowler
    from dashboard.__main__ import dashboard
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\__main__.py", line 28, in <module>
    dashboard = dash.Dash(
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 545, in __init__
    self.init_app()
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 623, in init_app
    self.enable_pages()
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 2130, in enable_pages
    _import_layouts_from_pages(self.config.pages_folder)
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\_pages.py", line 435, in _import_layouts_from_pages
    spec.loader.exec_module(page_module)
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\pages\compliance.py", line 43, in <module>
    num_rows = sum(1 for row in reader)
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\pages\compliance.py", line 43, in <genexpr>
    num_rows = sum(1 for row in reader)
  File "C:\Python310\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 5171: invalid start byte

Second Error

Traceback (most recent call last):
  File "C:\Users\Test\Desktop\prowler-4.1.0\prowler.py", line 7, in <module>
    sys.exit(prowler())
  File "C:\Users\Test\Desktop\prowler-4.1.0\prowler\__main__.py", line 64, in prowler
    from dashboard.__main__ import dashboard
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\__main__.py", line 28, in <module>
    dashboard = dash.Dash(
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 545, in __init__
    self.init_app()
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 623, in init_app
    self.enable_pages()
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\dash.py", line 2130, in enable_pages
    _import_layouts_from_pages(self.config.pages_folder)
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\dash\_pages.py", line 435, in _import_layouts_from_pages
    spec.loader.exec_module(page_module)
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\pages\overview.py", line 92, in <module>
    data = load_csv_files(csv_files)
  File "C:\Users\Test\Desktop\prowler-4.1.0\dashboard\pages\overview.py", line 80, in load_csv_files
    df = pd.read_csv(file, sep=";", on_bad_lines="skip")
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\pandas\io\parsers\readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\pandas\io\parsers\readers.py", line 626, in _read
    return parser.read(nrows)
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\pandas\io\parsers\readers.py", line 1923, in read
    ) = self._engine.read(  # type: ignore[attr-defined]
  File "C:\Users\Test\AppData\Local\pypoetry\Cache\virtualenvs\prowler-SRE5BNE--py3.10\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 234, in read
    chunks = self._reader.read_low_memory(nrows)
  File "parsers.pyx", line 838, in pandas._libs.parsers.TextReader.read_low_memory
  File "parsers.pyx", line 905, in pandas._libs.parsers.TextReader._read_rows
  File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows
  File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status
  File "parsers.pyx", line 2053, in pandas._libs.parsers.raise_parser_error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position 175279: invalid start byte

How did you install Prowler?

Cloning the repository from github.com (git clone)

Environment Resource

Running on workstation.

OS used

Windows 10
Kali Linux (attempting to parse output from windows 10 in Kali Linux gives the same error.)

Prowler version

4.1.0

Pip version

pip 24.0

Context

In an attempt for a quick fix, I have changed the following areas with some success.

Firstly, change encoding_format in \prowler-4.1.0\dashboard\config.py from utf-8 to windows-1252 or ISO-88659-1

Initial:

# Encoding
encoding_format = "utf-8"

Quick Fixed:

# Encoding
encoding_format = "windows-1252"

Next add encoding='unicode_escape' to df = pd.read_csv() line in function def load_csv_files(csv_files) in \prowler-4.1.0\dashboard\pages\overview.py

Initial :

def load_csv_files(csv_files):
    """Load CSV files into a single pandas DataFrame."""
    dfs = []
    for file in csv_files:
        df = pd.read_csv(file, sep=";", on_bad_lines="skip")

Quick Fixed:

def load_csv_files(csv_files):
    """Load CSV files into a single pandas DataFrame."""
    dfs = []
    for file in csv_files:
        df = pd.read_csv(file, sep=";", on_bad_lines="skip",encoding='unicode_escape')
        if "CHECK_ID" in df.columns:

[pedrooot]

Hi @irboi746 thanks for the report. We found a solution for this issue but it's still in review because we are evaluating the best approach. In this pr you can find the current solution. Tell me if it works for you please!!
Thanks for using Prowler! 🚀

[sh3llcmdr]

I had the same issue and the pr you linked to has fixed the issue for me. Thanks

[mishradhananjai117]

in My case still I am stuck after the suggested change.

First Changes

encoding_format = "windows-1252"

Second Changes

def load_csv_files(csv_files):
"""Load CSV files into a single pandas DataFrame."""
dfs = []
for file in csv_files:
df = pd.read_csv(file, sep=";", on_bad_lines="skip",encoding='unicode_escape')
if "CHECK_ID" in df.columns:
if "TIMESTAMP" in df.columns or df["PROVIDER"].unique() == "aws":
dfs.append(df.astype(str))

Result
Loading all CSV files from the folder C:\Users\Dhananjai Mishra/output ...

Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Scripts\prowler.exe_main.py", line 7, in
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\prowler_main.py", line 64, in prowler
from dashboard.main import dashboard
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\dashboard_main_.py", line 28, in
dashboard = dash.Dash(
^^^^^^^^^^
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\dash\dash.py", line 545, in init
self.init_app()
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\dash\dash.py", line 623, in init_app
self.enable_pages()
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\dash\dash.py", line 2130, in enable_pages
_import_layouts_from_pages(self.config.pages_folder)
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\dash_pages.py", line 435, in _import_layouts_from_pages
spec.loader.exec_module(page_module)
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\dashboard\pages\compliance.py", line 96, in
data, results = load_csv_files(csv_files)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\dashboard\pages\compliance.py", line 53, in load_csv_files
df = pd.read_csv(file, sep=";", on_bad_lines="skip")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py", line 1026, in read_csv
return _read(filepath_or_buffer, kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py", line 620, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py", line 1620, in init
self._engine = self._make_engine(f, self.engine)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\readers.py", line 1898, in _make_engine
return mapping[engine](f, **self.options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Dhananjai Mishra\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 93, in init
self._reader = parsers.TextReader(src, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "parsers.pyx", line 574, in pandas._libs.parsers.TextReader.cinit
File "parsers.pyx", line 663, in pandas._libs.parsers.TextReader._get_header
File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows
File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status
File "parsers.pyx", line 2053, in pandas._libs.parsers.raise_parser_error
File "", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 5171: invalid start byte

[pedrooot]

Hey! @mishradhananjai117 this can be happpening because you are still using an older version or the output folder has old files with incorrect format. To ensure everything is ok you can make these comprobations:

• Clone the last version of Prowler from github, the dashboard encoding fix is not released in pip at the moment (will be released soon)
• Ensure that the output folder hasn't old outputs with bad enconding. I would delete this folder, generate the new files with the latest version of Prowler and run the dashboard.

If after all the steps you are still getting errors, we will investigate furthermore. Thanks for using Prowler!! 🚀