compat.dump raises ValidationError when nullable Nested fields (with required subfields) without dump_only of a ResponseSchema are None

[fcr--]

We found this issue when observed a ValidationError being thrown when serializing the response of an endpoint. The curiosity of this issue is that it happens in the compat.dump when all of these are true:

• There is a Nested field in a ResponseSchema with allow_none=True,
• The None value is used for that field.
• That field has a schema with required subfields (in this case it was an id str subfield without a default).

Then by reading the compat.dump code and the called functions, we've seen that setting dump_only=True on the Nested field served as a workaround.

More info:

• https://jira.autodesk.com/browse/UL-7077
• https://app.bugsnag.com/plangrid/bridge/errors/6479c9bae0b8f60007816be6

This bug happens at least on version 3.2.0

[fcr--]

This is how to reproduce the bug:

from flask_rebar import compat, ResponseSchema
from marshmallow import Schema, fields


class N(Schema):
    n = fields.Integer(required=True)


class S(ResponseSchema):
    sub = fields.Nested(N(), required=False, allow_none=True)


obj = {"sub": None}

print("default dump:", S().dump(obj))
print("compat.dump:", compat.dump(S(), obj))

which when run:

(venv) francisco_castro@GLOBQW4HV7DF7V bridge % python3 repro.py
default dump: {'sub': None}
Traceback (most recent call last):
  File "/Users/francisco_castro/proj/bridge/repro.py", line 16, in <module>
    print("compat.dump:", compat.dump(S(), obj))
                          ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/francisco_castro/proj/bridge/venv/lib/python3.11/site-packages/flask_rebar/compat.py", line 51, in dump
    schema.load(filtered.loadable)  # trigger validation
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/francisco_castro/proj/bridge/venv/lib/python3.11/site-packages/marshmallow/schema.py", line 723, in load
    return self._do_load(
           ^^^^^^^^^^^^^^
  File "/Users/francisco_castro/proj/bridge/venv/lib/python3.11/site-packages/marshmallow/schema.py", line 910, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'sub': {'n': ['Missing data for required field.']}}

[fcr--]

So, this is where the empty dict came from:

flask-rebar/flask_rebar/validation.py

Lines 46 to 61 in 748b8a5

	loadable = dict()
	for k, v in non_dump_only.items():
	field = schema.fields[output_to_input_keys.get(k, k)]
	# see if we have a nested schema (using either Nested(many=True) or List(Nested())
	field_schema = None
	if isinstance(field, fields.Nested):
	field_schema = field.schema
	elif isinstance(field, fields.List) and isinstance(
	field.inner, fields.Nested
	):
	field_schema = field.inner.schema
	if field_schema is None:
	loadable[k] = v
	else:
	field_filtered = filter_dump_only(field_schema, v)
	loadable[k] = field_filtered.loadable

Which then reaches compat.dump's schema.load(filtered.loadable), resulting on the error shown here.