Nested Casts Don't Match 010 Behavior

[d0c-s4vage]

This is a tricky one that came up during the work on #120. I decided to make it a separate issue.

Describe the bug

Nested casts do not match gcc behavior - is the "correct" behavior here undefined in the C standard? (still tracking that down). I thought it seemed clear, but I need to verify.

Consider the 010 template code below:

local ubyte a_byte = 0xff;
local int final = (uint)((ushort)(a_byte << 4));
Printf("%08x", final);

The output of the template above when run in 010 Editor is 000000f0.

Now consider this small C program:

#include <stdio.h>

typedef unsigned char ubyte;
typedef unsigned short ushort;
typedef unsigned int uint;

int main(int argc, const char *argv[])
{
    unsigned char a_byte = 0xff;
    int final = (uint)((ushort)(a_byte << 4));
    printf("%08x", final);

    return 0;
}

When compiled and run with gcc, the output is 00000ff0, which is what pfp is currently mirroring.

Which is correct? Is it a bug in 010 Editor, or undefined behavior in the C standard?

If it's undefined behavior, we need to mirror 010 Editor. If it's a bug in 010 Editor, we should match the C standard.

To Reproduce

Run the provided template script and C programs and compare their output.

Expected Behavior

010 Editor, pfp, and gcc should have the same output, 00000ff0.

Implementation/Fix Notes/Thoughts

Here's a test that demonstrates the problem:

    def test_nested_casts(self):
        dom = self._test_parse_build(
            "",
            """
                local ubyte a_byte = 0xff;
                local int final = (uint)((ushort)(a_byte << 4));
                Printf("%08x", final);
            """,
            # stdout="00000ff0", # matches gcc
            stdout="000000f0", # matches 010 editor
        )