Need NES 2.0 headers support

[nsx0r]

Some NES carts (aftermarket, pirate, multicart, etc.) cannot be dumped because the mapper number isn't supported.
I tried adding mapper 268 manually but it did not work, I don't understand the inner-workings enough.
Is there a way to implement all the missing headers? Or at least making it possible to add mapper numbers >255?
Reference: https://www.nesdev.org/wiki/Mapper#Plane_1

[Ancyker]

Just wanted to note here I'm completely redoing how NES.ino works, so please don't change it significantly XD

(Sorry to post this 3 times, just wanted to note it on all of the NES issues)

[PsychoFox11]

I just wanted to bump this - basically we need support for mapper numbers higher than 255. There are scripts, I can swap them out, like replace mapper 3 a different script for mapper 446, but Is it possible we could proceed on this? I THINK all that has to be done is to allow the mapper number to be more than 1 byte, and store mappers over 255.
Ideally it would include submappers, or offer a second menu to choose submapper when applicable, but just being able to assign mapper numbers higher than 255 would be a great start, and allow those of use using extra scripts to not need to reflash so often!

[Ancyker]

Yes, I had this working pretty well, but I was looking at moving both the mappers and the database files to a binary file format. Mappers to free up the PROGMEM and decrease the memory usage, and a binary file format to increase the speed at which the files can be parsed.

The original mapper rework I did in the previous reply above used a JSON library to store the mapper data. This was nice and I did estimate that switching everything over to it would decrease memory usage slightly, doing it for just the mappers increased it for the time being. JSON is nice for being more human-readable but the binary file format is more MCU-friendly.

So really I guess it depends on which we want to prioritize. Do we prefer to have more PROGMEM available and faster database searches at the expense of having to run the human-readable files through a program to convert them to a binary format or do we prefer a human-readable format at the expense of slower searched and more PROGMEM usage?

[Ancyker]

Here's an example of the binary file format, this is a database for the Mega Drive:

It's easy to make your own by uploading a No-Intro database file here: https://oscr.tools/db/megadrive/cart

The file format is fairly simple:

• There is a header that always starts with the hex value of 0xDA705C12 (an attempt at writing "DATOSCR" in hex). This tells the firmware it is indeed a "crdb" file.
• 2 bytes are for the version of the format, 0x01 in this case. It tells the firmware how the following is laid out.
• NUL byte
• 18 bytes are a NUL-padded string containing the file's purpose and core name ("DB/MEGADRIVE" here for a Mega Drive database).
• 4 bytes are the date but written in hex so it's readable in a hex editor.
• NUL byte.
• 4 bytes are the size of a record, NUL-padded (0x6C or 108 in this case).
• NUL byte
• 40 bytes representing how records are laid out. In this case it's saying the first 4 bytes are the CRC32, the following 4 bytes are the size, and the last 100 bytes are the NUL-padded name.
• NUL padding to the size specified as the record size (108 in this case).

The resulting file is significantly larger than the current format because of the NUL padding, but because of the fixed record size, it's significantly faster to search. The code jumps to the start of the first record, reads the first 4 bytes for the CRC32, and if it doesn't match it jumps 104 bytes ahead to the next CRC32 and checks it and so on. This means it doesn't need to read each byte looking for newlines or special characters. Each record is a fixed length, which makes searching the file much faster.

[PsychoFox11]

I'd definitely defer to @nsx0r to comment on the best direction for NES stuff. I'd just like to be able to use newer mappers without reflashing so much, I've never found the time to be a big issue on NES games for me, and I don't always use the lookup afterwards cause I'm often dumping stuff that hasn't been recorded yet, especially if I'm working with 2.0.

[nsx0r]

Does that mean the scripts would be stored in files on the SD card and not in the NES.ino anymore?
That would be cool regardless of the format: adding/updating scripts without flashing the whole thing and maybe have the menu scroll through the mappers by enumerating the script files in some folder on the SD card.
Next up: construct correct headers and get rid of the power-of-2 sizes.

[Ancyker]

Does that mean the scripts would be stored in files on the SD card and not in the NES.ino anymore? That would be cool regardless of the format: adding/updating scripts without flashing the whole thing and maybe have the menu scroll through the mappers by enumerating the script files in some folder on the SD card.

This would be somewhat hard to do but you can store values in the file. I had only planned to store the mapper table in a database file. I did consider using scripts for dumping carts but there isn't really a good pre-made scripting language for AVRs and making one would be a ton of work.

Next up: construct correct headers and get rid of the power-of-2 sizes.

That can easily be done. I can make the sizes in the db file a 4-byte integer. The cost of doing it in the db file is small, the cost of doing it in the flash-mapped variable was fairly expensive.

[Ancyker]

I'd definitely defer to @nsx0r to comment on the best direction for NES stuff. I'd just like to be able to use newer mappers without reflashing so much, I've never found the time to be a big issue on NES games for me, and I don't always use the lookup afterwards cause I'm often dumping stuff that hasn't been recorded yet, especially if I'm working with 2.0.

I am suggesting going with a binary format for all database files on the SD card.

[PsychoFox11]

I am suggesting going with a binary format for all database files on the SD card.

Though currently mapper scripts aren't on the SD, are you only meaning what's currently there, or moving more things like mapper scripts to the SD for convenience/ease of use?
As a side note, the MD.ino keeps growing as special cases have to be added to dump certain carts right (and the NES.ino WOULD grow if we could add more mappers as they come out), and it really seems that info could be on SD as well.

[Ancyker]

Though currently mapper scripts aren't on the SD, are you only meaning what's currently there, or moving more things like mapper scripts to the SD for convenience/ease of use?

I meant everything in the SD folder basically. All of the databases.

As a side note, the MD.ino keeps growing as special cases have to be added to dump certain carts right (and the NES.ino WOULD grow if we could add more mappers as they come out), and it really seems that info could be on SD as well.

I'm looking into it, it seems it might be possible to have some of the logic in external files. Further testing is needed but I don't have any of the MD/Genesis carts with exceptions, and I only have 1 NES game. We should probably work on this in a branch because it will almost certainly be broken for a bit, lol.

[Ancyker]

I'm considering this layout (folder/file structure) for this rework of the SD card files (using MD and NES as examples):

• / (root)
  • .oscr/
    • cores/
      • md/
        • eeprom.in
        • eeprom.crlu
        • sumfix.in
        • sumfix.crlu
      • nes/
        • mappers/
          • <mapper-id>.in: Mapper logic input file used to create the binary file
          • <mapper-id>.crmp: Mapper logic binary file. Replaces (most of) the large switch (mapper)... statement. The mapper-id in the filename is the ID of the mapper, might contain submapper logic or that might be in another file (if in another file, probably <mapper-id>-<submapper-id>.crmp).
        • mappers.in: Mapper index input file.
        • mappers.crlu: Mapper index binary file. Replaces the mapsize array.
      • <system-name>/
        • ...
    • databases/
      • nes.in: The NES's database input file.
      • nes.crdb: The NES's binary database.
      • md.in: The Mega Drive's database input file.
      • md.crdb: The Mega Drive's binary database.
      • <system-name>.in: Input file used to generate a binary database, likely just No-Intro's XML format
      • <system-name>.crdb: Compiled binary database file, made by running the .in file through the converter

databases/
The .in files are just text files, the extension is so it's clear they are inputs for the related file. We probably don't need to include the ones for the database in the repo since they are just exports from No-Intro. The .crdb files are the binary version of the files containing the info that the .txt files do.

cores/
For .crlu these will look very similar to .crdb files but won't be standardized so that arbitrary data can be stored in them. Any cores that use them will use more PROGMEM (flash) and RAM. This may be dropped in favor of just using the .crdb format for them.

The .crmp files are a bit more complex. The .in files are also text files, but they are not XML. They'd look similar to assembly, containing a keyword followed by some values. What the exact format of their input will look like is unknown at this time, but here's an example of a mapper's file that may or may not resemble what they end up looking like:

11.in

DEFINE $0 $banks
DEFINE $1 $prgsize
DEFINE $2 $addr
BEGIN
SET $banks POW 2 $prgsize
SET $banks DIV $banks 2
FOR 0 LT $banks 1
SET $addr ADD 0xFFB0 $i
WRITE $addr $i
DUMP 0 0x8000 $base
ROF
END

Something like this will be compiled into a binary file and read by the firmware to carry out instructions. The resulting file will likely only be a few bytes. Keeping them as multiple files instead of one big file will keep the code that reads them and executes the instructions simple.

The output will turn these keywords into magic numbers that the firmware can quickly read and handle. For instance, "WRITE" might be equal to a byte of 0x10 and $addr would be first some value indicating whether the parameter is a value or a variable, let's say 0x01 for value and 0x02 for parameter, so 0x02 in this case, followed by the "slot" the variable is in (the DEFINE statements are only used by the interpreter that makes the binary file to make them easier to read), in this case 0x02, and so on.

It sounds complicated (because it is), however, you don't need to fully understand how it works to use it. I looked around for any existing libs that do what we need and while there are some they are too large to reasonably run on a microcontroller.