Getting "KeyError: 'd'" when reading XML OData

[aviasd]

I'm trying to read from this site:
http://knesset.gov.il/Odata/ParliamentInfo.svc/

The site is in XML and every table is saved in XML too.
For example:
http://knesset.gov.il/Odata/ParliamentInfo.svc/KNS_Agenda

When reading with python-pyodata, there is a key error when trying to get entities.

Code:

import requests
import pyodata

SERVICE_URL = 'http://knesset.gov.il/Odata/ParliamentInfo.svc/'

# Create instance of OData client
client = pyodata.Client(SERVICE_URL, requests.Session())

# This is the problematic line
a = client.entity_sets.KNS_Agenda.get_entities().execute()

The exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pyodata/v2/service.py", line 305, in execute
    return self._handler(response)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/pyodata/v2/service.py", line 1102, in get_entities_handler
    entities = content['d']['results']
KeyError: 'd'

Is there a way to fix this?

Thanks in advance,

[filak-sap]

@aviasd Thank you for taking the time to report this issue. Unfortunately, I do believe the service is OData V3 service which is not supported yet.

I have created an experimental fix which is available on the branch:
https://github.com/SAP/python-pyodata/tree/experimental_v3

I was able to run your example without issues with pyodata from that branch. However, I didn't try to fetch any additional entity sets, so there might be more incompatibilities.

If you have time and you want to help us make pyodata better, please, try to use pyodata from the experimental_v3 branch and report results here.

[aviasd]

@filak-sap Thank you for the quick answer and solution.
The solution is working perfectly well almost all of the time.
Today I ran into a problem for the first time:

The code:

import pyodata
import requests

SERVICE_URL = 'http://knesset.gov.il/Odata/ParliamentInfo.svc/'
client = pyodata.Client(SERVICE_URL, requests.Session())

# count is 44535
count = client.entity_sets.KNS_Bill.get_entities().count().execute()

laws = []
for i in range(34300, count, 100):
    laws += [client.entity_sets.KNS_Bill.get_entities().skip(i).top(100).execute()]

This is the error:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\model.py", line 409, in from_literal
    value = datetime.datetime.strptime(value, '%Y-%m-%dT%H:%M:%S.%f')
  File "C:\ProgramData\Anaconda3\envs\BD\lib\_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "C:\ProgramData\Anaconda3\envs\BD\lib\_strptime.py", line 362, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: 7

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\model.py", line 412, in from_literal
    value = datetime.datetime.strptime(value, '%Y-%m-%dT%H:%M:%S')
  File "C:\ProgramData\Anaconda3\envs\BD\lib\_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "C:\ProgramData\Anaconda3\envs\BD\lib\_strptime.py", line 362, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: .3766667

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\model.py", line 415, in from_literal
    value = datetime.datetime.strptime(value, '%Y-%m-%dT%H:%M')
  File "C:\ProgramData\Anaconda3\envs\BD\lib\_strptime.py", line 577, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "C:\ProgramData\Anaconda3\envs\BD\lib\_strptime.py", line 362, in _strptime
    data_string[found.end():])
ValueError: unconverted data remains: :50.3766667

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Aviad/PycharmProjects/BigData/util_classes.py", line 44, in <module>
    laws += [client.entity_sets.KNS_Bill.get_entities().skip(i).top(100).execute()]
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\service.py", line 305, in execute
    return self._handler(response)
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\service.py", line 1109, in get_entities_handler
    entity = EntityProxy(self._service, self._entity_set, self._entity_set.entity_type, props)
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\service.py", line 723, in __init__
    self._cache[type_proprty.name] = type_proprty.typ.traits.from_json(proprties[type_proprty.name])
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\model.py", line 401, in from_json
    return self.from_literal(value)
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\model.py", line 417, in from_literal
    raise PyODataModelError('Cannot decode datetime from value {}.'.format(value))
pyodata.exceptions.PyODataModelError: Cannot decode datetime from value 2019-04-28T19:40:50.3766667.

Process finished with exit code 1

Thank you,

[aviasd]

Hey @filak-sap , I found another bug:

The code:

import pyodata
import requests

SERVICE_URL = 'http://knesset.gov.il/Odata/ParliamentInfo.svc/'
client = pyodata.Client(SERVICE_URL, requests.Session())
pers = client.entity_sets.KNS_Person.get_entity(PersonID=48).execute()

The error:

Traceback (most recent call last):
  File "C:/Users/Aviad/PycharmProjects/BigData/try.py", line 6, in <module>
    pers = client.entity_sets.KNS_Person.get_entity(PersonID=48).execute()
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\service.py", line 305, in execute
    return self._handler(response)
  File "C:\ProgramData\Anaconda3\envs\BD\lib\site-packages\pyodata\v2\service.py", line 1074, in get_entity_handler
    entity = response.json()['d']
KeyError: 'd'

I think it has something to do with the first bug you already solved.
The first one you solved was with get_entities() function, but the error was the same.
Thank you,

[filak-sap]

Interestingly enough, Edm.DateTime allows 7 digits for microseconds while Python allows only 6 digits (which is sane because the 7th digit is hundreds of nanoseconds). For the time being, I will ignore the last 7th digit.

I posted a comment to a blog post about SAP Gateway Date Time:
https://blogs.sap.com/2017/01/05/date-and-time-in-sap-gateway-foundation/comment-page-1/#comment-497276

[filak-sap]

@aviasd I pushed commits fixing the recent problems you found to the branch experimental_v3. I can run all your requests without exceptions.

Please, keep in mind that the branch is not stable, not official and servers only the experimental purposes.

[aviasd]

Hey,
I am using your branch experimental_v3,
It is working perfectly fine.
I have a question on the Usage:
I am trying to pickle <class 'pyodata.v2.service.EntityProxy'>, it is not working.. I get:

Traceback (most recent call last):
  File "C:/PycharmProjects/BigData/kafka_try.py", line 36, in <module>
    ab = pickle.dumps(a)
_pickle.PicklingError: Can't pickle <enum 'Kinds'>: attribute lookup Kinds on pyodata.v2.model failed

The purpose is to pickle or serialize it somehow so I can send it via kafka Producer.
Is there a way to do that?

Thank you very much,

[filak-sap]

Hi @aviasd, EntityProxy is a thin wrapper holding a cache of properties. These instances also have a reference to HTTP session. I don't think you want to pickle EntityProxy. It would be better to pickle only the downloaded properties converted to Python types.

I am starting thinking about a customizable OData response de-serializer because Panda users also needs it.

How do you want to de-serialize and use it?

[elisim]

any progress making this solution official?

[filak-sap]

@elisim unfortunately, the experimental branch will remain experimental. We are currently working on V4 #39 which would make it simple to add support for V3.

Is there anything I can backport to the experimental branch?

[TELangelaar]

Hi,
Ran in the same problem as aviasd, experimental branch works for me. Thank you guys for the work on this.

[truebeck-datawh]

Experimental_v3 branch worked for me to run this code:
vendors = getattr(client.entity_sets, 'DW_Vendor').get_entities().execute() and I am able to retrieve all the data from that feed in one line of code.

When using the production v2 code I was getting the KeyError['d'] error.

However using _v3 I still see this error when I try to perform a count() operation on the oData feed:
vendors = getattr(client.entity_sets, 'DW_Vendor').get_entities().count().execute()

Not sure if this is related to the OData feed itself. It shows a 404 error, but as I mentioned above, dropping .count() successfully returned rows.