"illegal base64 data" in 3.7.3 encrypted file, rejected by 3.8.x

[PeterJCLaw]

I've recently observed a failure where 3.8.x was unable to decrypt a file which 3.7.3 had encrypted (and could still decrypt). The error message is as follows:

Expanding encrypted .env file /path/to/file.env
Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  projects/<project>/locations/<location>/keyRings/<keyring>/cryptoKeys/<key>: FAILED
    - | illegal base64 data at input byte 148

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

As the file itself was still readable by 3.7.3 though not by 3.8.x (tested with 3.8.0 and 3.8.1) it appears the issue was in the way that the file itself was encoded.

Some manual attempting to decode parts of the file (though this is very much guessing in the dark -- I don't know at all how the file format works) seems to suggest that this is due to missing padding on the sops_gcp_kms__list_0__map_enc key. I'm assuming that that key is base64 encoded¹, both by examples in other files and that adding a single = (none were present) allows the errant files to be decrypted by newer sops.

Re-encrypting the same values afresh (on Ubuntu, with any of these versions of sops) does not appear to reproduce the issue -- the new value for sops_gcp_kms__list_0__map_enc has trailing padding as would be expected.

The issue was first noticed in a debian based docker container, reproducible on Ubuntu linux, both x86-64. I have colleagues who use macOS, and it seems the file was likely created on macOS however it had been edited on both platforms (using 3.7.3) over its lifetime and without that key changing value (i.e: it was created with missing padding and remained that way).

Footnotes

1. and with some checking by decoding the value using Python's base64.b64decode, which reports about missing padding. ↩