Raw bytes as query parameter

[doctorperceptron]

I'm using FastAPI to send pickled objects (using dill) to a remote server, but am having trouble sending/receiving my pickled object as raw bytes.

I'm using a POST endpoint and requests for sending the data. The closest I've come to success is to specify the pickled object parameter as a query parameter of type bytes:

@endpoint.post("/{my_var}")
async def runtask(
    my_var: str,
    obj_param: bytes,
):
<unpickle obj_param>

For example, using numpy's subtract function and pickling with dill, I get a byte string like:

b'\x80\x03cnumpy\nsubtract\nq\x00.'

My POST looks like this (using requests.post and f strings to post the byte string object for obj_oparam):

http://127.0.0.1:8000/LpMetric?obj_param=b'%5Cx80%5Cx03cnumpy%5Cnsubtract%5Cnq%5Cx00.'

When I successfully receive this object at my endpoint, unpickling fails, because the byte string now looks like this:

b"b'\\x80\\x03cnumpy\\nsubtract\\nq\\x00.'"

The issue seems to be that FastAPI is assuming my obj_param is a string, and is therefore converting it to bytes, giving me a byte string of a byte string. I know that FastAPI treats raw bytes as strings "under the hood", so is this expected behaviour? Am I doing something wrong when sending my pickled object, or is there something else I should be doing on the receiving end?

Hope that's clear. I'm new to APIs so would really appreciate any help you can give. Thanks for your patience and thanks for your help!

[dmontagu]

There's a few issues to be aware of here:

1. Yes, this is expected behavior in the sense that you are actually sending "b'\\x80\\x03cnumpy\\nsubtract\\nq\\x00.'" to the server, so FastAPI will treat the received value as the result of converting that string into bytes, as you are seeing.

   If you want to receive the original bytes, you need to make the request to your server with the actual bytes in your query parameter (url-encoded), not the stringified version that you are currently sending. (I'm not 100% sure whether query parameters are allowed to have arbitrary (url-encoded) bytes, though.)

2. In general, you probably don't want to put arbitrary bytes in the query string; typically data like that would go in the body. This can be achieved when making a client request using requests by something like requests.post(endpoint, data=my_pickled_bytes).

3. Accepting pickled objects in a server endpoint is one of the most common and most serious security risks you'll typically encounter during API development -- if a malicious party gets access to the endpoint, they can immediately and easily execute arbitrary code on your server. You should absolutely never(!) expose such an endpoint to the open internet, or even to any untrusted user (basically, never in any kind of production environment where you aren't the sole user).

   If you know what you are doing then it might be okay, but considering you said you are new to APIs I figured it was worth calling this out explicitly just in case.

For what it's worth, if what you are trying to submit is something related to a numpy array, you might be better off using the tobytes method to convert the array to bytes, and the frombuffer function to load the bytes into an array on the server (you may also want to transmit the dtype as a separate field). This data could be sent with the request as part of a json payload, or as form data, and would prevent any remote code execution risk.

[tincumagic]

You should encode the raw bytes in base64 then send encoded data as any normal string parameter and decode it on server as base64

[dmontagu]

Yes, @tincumagic's suggestion is also a great choice for this situation. The same warnings about pickling would still apply though!

[doctorperceptron]

Thanks for the detailed response @dmontagu !

[tiangolo]

Thanks @dmontagu and @tincumagic ! Great explanation including the security risks @dmontagu 🙇‍♂️ 🍰

I think that solves your issue, right @doctorperceptron ? If so, you can close the issue.