[Bug] SSL Error when trying to .list_indexes()

[oddrationale]

Is this a new bug in the Pinecone Python client?

• I believe this is a new bug in the Pinecone Python Client
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When trying to .list_indexes(), I'm getting a SSL error (see logs below). I do have certifi installed in the Python environment.

Expected Behavior

Successfully able to list existing indexes.

Steps To Reproduce

1. Import pinecone.
2. Initialize pinecone with and api_key and environment.
3. Run pinecone.list_indexes().

Relevant log output

Python 3.11.3 | packaged by conda-forge | (main, Apr  6 2023, 08:50:54) [MSC v.1934 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pinecone

In [2]: pinecone.init(api_key="<redacted>", environment="us-east1-gcp")

In [3]: pinecone.list_indexes()
---------------------------------------------------------------------------
SSLError                                  Traceback (most recent call last)
File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\util\ssl_.py:402, in ssl_wrap_socket(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir, key_password, ca_cert_data, tls_in_tls)
    401 try:
--> 402     context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)
    403 except (IOError, OSError) as e:

SSLError: [X509] PEM lib (_ssl.c:4149)

During handling of the above exception, another exception occurred:

SSLError                                  Traceback (most recent call last)
File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\connectionpool.py:703, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    702 # Make the request on the httplib connection object.
--> 703 httplib_response = self._make_request(
    704     conn,
    705     method,
    706     url,
    707     timeout=timeout_obj,
    708     body=body,
    709     headers=headers,
    710     chunked=chunked,
    711 )
    713 # If we're going to release the connection in ``finally:``, then
    714 # the response doesn't need to know about the connection. Otherwise
    715 # it will also try to release it and we'll have a double-release
    716 # mess.

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\connectionpool.py:386, in HTTPConnectionPool._make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    385 try:
--> 386     self._validate_conn(conn)
    387 except (SocketTimeout, BaseSSLError) as e:
    388     # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\connectionpool.py:1042, in HTTPSConnectionPool._validate_conn(self, conn)
   1041 if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
-> 1042     conn.connect()
   1044 if not conn.is_verified:

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\connection.py:419, in HTTPSConnection.connect(self)
    417     context.load_default_certs()
--> 419 self.sock = ssl_wrap_socket(
    420     sock=conn,
    421     keyfile=self.key_file,
    422     certfile=self.cert_file,
    423     key_password=self.key_password,
    424     ca_certs=self.ca_certs,
    425     ca_cert_dir=self.ca_cert_dir,
    426     ca_cert_data=self.ca_cert_data,
    427     server_hostname=server_hostname,
    428     ssl_context=context,
    429     tls_in_tls=tls_in_tls,
    430 )
    432 # If we're using all defaults and the connection
    433 # is TLSv1 or TLSv1.1 we throw a DeprecationWarning
    434 # for the host.

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\util\ssl_.py:404, in ssl_wrap_socket(sock, keyfile, certfile, cert_reqs, ca_certs, server_hostname, ssl_version, ciphers, ssl_context, ca_cert_dir, key_password, ca_cert_data, tls_in_tls)
    403     except (IOError, OSError) as e:
--> 404         raise SSLError(e)
    406 elif ssl_context is None and hasattr(context, "load_default_certs"):
    407     # try to load OS default certs; works well on Windows (require Python3.4+)

SSLError: [X509] PEM lib (_ssl.c:4149)

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
Cell In[3], line 1
----> 1 pinecone.list_indexes()

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\pinecone\manage.py:185, in list_indexes()
    183 """Lists all indexes."""
    184 api_instance = _get_api_instance()
--> 185 response = api_instance.list_indexes()
    186 return response

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\pinecone\core\client\api_client.py:776, in Endpoint.__call__(self, *args, **kwargs)
    765 def __call__(self, *args, **kwargs):
    766     """ This method is invoked when endpoints are called
    767     Example:
    768
   (...)
    774
    775     """
--> 776     return self.callable(self, *args, **kwargs)

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\pinecone\core\client\api\index_operations_api.py:1132, in IndexOperationsApi.__init__.<locals>.__list_indexes(self, **kwargs)
   1128 kwargs['_check_return_type'] = kwargs.get(
   1129     '_check_return_type', True
   1130 )
   1131 kwargs['_host_index'] = kwargs.get('_host_index')
-> 1132 return self.call_with_http_info(**kwargs)

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\pinecone\core\client\api_client.py:838, in Endpoint.call_with_http_info(self, **kwargs)
    834     header_list = self.api_client.select_header_content_type(
    835         content_type_headers_list)
    836     params['header']['Content-Type'] = header_list
--> 838 return self.api_client.call_api(
    839     self.settings['endpoint_path'], self.settings['http_method'],
    840     params['path'],
    841     params['query'],
    842     params['header'],
    843     body=params['body'],
    844     post_params=params['form'],
    845     files=params['file'],
    846     response_type=self.settings['response_type'],
    847     auth_settings=self.settings['auth'],
    848     async_req=kwargs['async_req'],
    849     _check_type=kwargs['_check_return_type'],
    850     _return_http_data_only=kwargs['_return_http_data_only'],
    851     _preload_content=kwargs['_preload_content'],
    852     _request_timeout=kwargs['_request_timeout'],
    853     _host=_host,
    854     collection_formats=params['collection_format'])

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\pinecone\core\client\api_client.py:413, in ApiClient.call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, async_req, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host, _check_type)
    359 """Makes the HTTP request (synchronous) and returns deserialized data.
    360
    361 To make an async_req request, set the async_req parameter.
   (...)
    410     then the method will return the response directly.
    411 """
    412 if not async_req:
--> 413     return self.__call_api(resource_path, method,
    414                            path_params, query_params, header_params,
    415                            body, post_params, files,
    416                            response_type, auth_settings,
    417                            _return_http_data_only, collection_formats,
    418                            _preload_content, _request_timeout, _host,
    419                            _check_type)
    421 return self.pool.apply_async(self.__call_api, (resource_path,
    422                                                method, path_params,
    423                                                query_params,
   (...)
    431                                                _request_timeout,
    432                                                _host, _check_type))

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\pinecone\core\client\api_client.py:200, in ApiClient.__call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout, _host, _check_type)
    196     url = _host + resource_path
    198 try:
    199     # perform request and return response
--> 200     response_data = self.request(
    201         method, url, query_params=query_params, headers=header_params,
    202         post_params=post_params, body=body,
    203         _preload_content=_preload_content,
    204         _request_timeout=_request_timeout)
    205 except ApiException as e:
    206     e.body = e.body.decode('utf-8')

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\pinecone\core\client\api_client.py:439, in ApiClient.request(self, method, url, query_params, headers, post_params, body, _preload_content, _request_timeout)
    437 """Makes the HTTP request using RESTClient."""
    438 if method == "GET":
--> 439     return self.rest_client.GET(url,
    440                                 query_params=query_params,
    441                                 _preload_content=_preload_content,
    442                                 _request_timeout=_request_timeout,
    443                                 headers=headers)
    444 elif method == "HEAD":
    445     return self.rest_client.HEAD(url,
    446                                  query_params=query_params,
    447                                  _preload_content=_preload_content,
    448                                  _request_timeout=_request_timeout,
    449                                  headers=headers)

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\pinecone\core\client\rest.py:236, in RESTClientObject.GET(self, url, headers, query_params, _preload_content, _request_timeout)
    234 def GET(self, url, headers=None, query_params=None, _preload_content=True,
    235         _request_timeout=None):
--> 236     return self.request("GET", url,
    237                         headers=headers,
    238                         _preload_content=_preload_content,
    239                         _request_timeout=_request_timeout,
    240                         query_params=query_params)

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\pinecone\core\client\rest.py:202, in RESTClientObject.request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
    199             raise ApiException(status=0, reason=msg)
    200     # For `GET`, `HEAD`
    201     else:
--> 202         r = self.pool_manager.request(method, url,
    203                                       fields=query_params,
    204                                       preload_content=_preload_content,
    205                                       timeout=timeout,
    206                                       headers=headers)
    207 except urllib3.exceptions.SSLError as e:
    208     msg = "{0}\n{1}".format(type(e).__name__, str(e))

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\request.py:74, in RequestMethods.request(self, method, url, fields, headers, **urlopen_kw)
     71 urlopen_kw["request_url"] = url
     73 if method in self._encode_url_methods:
---> 74     return self.request_encode_url(
     75         method, url, fields=fields, headers=headers, **urlopen_kw
     76     )
     77 else:
     78     return self.request_encode_body(
     79         method, url, fields=fields, headers=headers, **urlopen_kw
     80     )

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\request.py:96, in RequestMethods.request_encode_url(self, method, url, fields, headers, **urlopen_kw)
     93 if fields:
     94     url += "?" + urlencode(fields)
---> 96 return self.urlopen(method, url, **extra_kw)

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\poolmanager.py:376, in PoolManager.urlopen(self, method, url, redirect, **kw)
    374     response = conn.urlopen(method, url, **kw)
    375 else:
--> 376     response = conn.urlopen(method, u.request_uri, **kw)
    378 redirect_location = redirect and response.get_redirect_location()
    379 if not redirect_location:

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\connectionpool.py:815, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    810 if not conn:
    811     # Try again
    812     log.warning(
    813         "Retrying (%r) after connection broken by '%r': %s", retries, err, url
    814     )
--> 815     return self.urlopen(
    816         method,
    817         url,
    818         body,
    819         headers,
    820         retries,
    821         redirect,
    822         assert_same_host,
    823         timeout=timeout,
    824         pool_timeout=pool_timeout,
    825         release_conn=release_conn,
    826         chunked=chunked,
    827         body_pos=body_pos,
    828         **response_kw
    829     )
    831 # Handle redirect?
    832 redirect_location = redirect and response.get_redirect_location()

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\connectionpool.py:815, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    810 if not conn:
    811     # Try again
    812     log.warning(
    813         "Retrying (%r) after connection broken by '%r': %s", retries, err, url
    814     )
--> 815     return self.urlopen(
    816         method,
    817         url,
    818         body,
    819         headers,
    820         retries,
    821         redirect,
    822         assert_same_host,
    823         timeout=timeout,
    824         pool_timeout=pool_timeout,
    825         release_conn=release_conn,
    826         chunked=chunked,
    827         body_pos=body_pos,
    828         **response_kw
    829     )
    831 # Handle redirect?
    832 redirect_location = redirect and response.get_redirect_location()

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\connectionpool.py:815, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    810 if not conn:
    811     # Try again
    812     log.warning(
    813         "Retrying (%r) after connection broken by '%r': %s", retries, err, url
    814     )
--> 815     return self.urlopen(
    816         method,
    817         url,
    818         body,
    819         headers,
    820         retries,
    821         redirect,
    822         assert_same_host,
    823         timeout=timeout,
    824         pool_timeout=pool_timeout,
    825         release_conn=release_conn,
    826         chunked=chunked,
    827         body_pos=body_pos,
    828         **response_kw
    829     )
    831 # Handle redirect?
    832 redirect_location = redirect and response.get_redirect_location()

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\connectionpool.py:787, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    784 elif isinstance(e, (SocketError, HTTPException)):
    785     e = ProtocolError("Connection aborted.", e)
--> 787 retries = retries.increment(
    788     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    789 )
    790 retries.sleep()
    792 # Keep track of the error for the retry warning.

File ~\scoop\apps\mambaforge\current\envs\llm\Lib\site-packages\urllib3\util\retry.py:592, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
    581 new_retry = self.new(
    582     total=total,
    583     connect=connect,
   (...)
    588     history=history,
    589 )
    591 if new_retry.is_exhausted():
--> 592     raise MaxRetryError(_pool, url, error or ResponseError(cause))
    594 log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)
    596 return new_retry

MaxRetryError: HTTPSConnectionPool(host='controller.us-east1-gcp.pinecone.io', port=443): Max retries exceeded with url: /databases (Caused by SSLError(SSLError(524297, '[X509] PEM lib (_ssl.c:4149)')))

In [4]:

Environment

- OS: Windows 10
- Python: Python 3.11.3
- pinecone: 2.2.1

Additional Context

I'm able to successfully able to ping the hostname (controller.us-east1-gcp.pinecone.io). So network connectivity appears to be working.

I haven't found a way to disable SSL verification.

[lparisi]

Hi, can you double check your Environment name, I was having the same error, but my env name had a typo.

[oddrationale]

Hello!

So, you were correct that I had the wrong environment. I checked my pinecone account and I am on us-east4-gcp not us-east1-gcp.

However, I tried again with the correct environment:

pinecone.init(api_key="<redacted>", environment="us-east4-gcp")

But I still get the same error when trying to .list_indexes():

MaxRetryError: HTTPSConnectionPool(host='controller.us-east4-gcp.pinecone.io', port=443): Max retries exceeded with url: /databases (Caused by SSLError(SSLError(524297, '[X509] PEM lib (_ssl.c:4149)')))

[philipk19238]

+1 getting same error

[SimonB97]

+1 same issue for me

[jrinder42]

I had the same issue and I believe it is due to the requests package and how it handles ssl. If you take the Python code with requests and swap it with something like urllib3, it will potentially work (as it did for me). I took the curl commands, put them in a curl-to-Python converter, and then swapped requests with urllib3.

The real issue is that you have to link to an ssl cert for verification. You can use requests if you know how to use the verify parameter (i.e. linking to a trusted CA). You can read more about it here

[liwalayouni]

I resolved a similar issue by setting server proxy.
check solution here: https://community.pinecone.io/t/proxy-servers-with-pinecone/287/2

[it176131]

I had the same issue and believe it was due to my corporate firewall blocking the api call.

If requests were being used I'd navigate around the problem with https://pypi.org/project/pip-system-certs/

pip install pip-system-certs

I noticed that pinecone is using urllib3 (at the time of writing) and the pip-system-certs package doesn't patch it. So I had to concatenate the certadmin.pem file from https://www.pinecone.io/ to my environments cacert.pem file.

cat ./Downloads/certadmin.pem >> /c/path/to/certifi/package/cacert.pem

Worked after that.

---

Additional resource:

• https://stackoverflow.com/questions/69505218/how-to-download-server-ssl-certificate-with-urllib3