After much trial and error, I have managed to re-program SMW's sprite tile allotter, and basically eliminated sprite tile limits (per sprite) and the need for "special allowances" defined in the sprite header setting. Instead of allotting a certain number of tiles to each sprite, each sprite's tile index in the OAM starts right after the end of the last sprite tiles in the OAM.
Since for some reason yoshi doesn't work right otherwise, he will always get the first two slots. This also fixes the annoying priority glitch when yoshi goes behind sprites and mario goes in front, such as with the arrow lift or other ridable sprites.
This will also allow more tiles for custom sprites, so custom sprites with more than 6 tiles will be possible without special allowances. Special allowances for certain sprites defined in the sprite header also limit the number of sprites of that type that can be on the screen at one time usually to 2 or 1.
I probably won't make this always in effect in the ROM, rather there will be code inserted with an xkas patch (same type of patch as the dynamic sprite patch), and then a generator that sets a flag that activates this.
Also, the blowhard and goonie now work without header setting 0E
Here are some examples of a few things that will glitch without this patch:
There is, however, a small chance that this will cause a major glitch somewhere else and can't be used at all, and can't be released, but it's most likely that it will be released within the next few weeks, while I test many different sprites with this, and make sure there aren't any noticeable glitches.
EDIT: and those screenshots were taken with normal sprite header setting 00
EDIT2: just so you know, this does not have anything to do with the number of enemies on-screen at one time, but rather the main purpose is to allow "bigger" custom sprites (LM) and provide a more efficient method of handling sprite (hardware) tiles.
- slightly slows down game if there are too many sprites
- doesn't load very first frame, this can be fixed by making the routine check the first block at the top left of the screen instead of a generator. I don't mean custom blocks, I mean copies of the block (usually #025) that would originally be there would have their map16 numbers entered into the patch when it's patched. The issue with generators setting a flag is that they seem to be processed after sprites are processed.
- dynamic sprites seem to glitch (probably because of the address I used for the flag)
EDIT4: I'm just going to make it use the (almost) unused sprite header setting 10, which appears to be an exact clone of 00, used only in bowser battles. The bowser battles seem to work perfectly fine with sprite header setting 00, though.
EDIT5: I have the patch ready for public test!
Download see next post edit
It overrides sprite header setting 10, a clone of 00 only used in bowser battles for some reason, so you must change the bowser battle sprite header setting to 00.
EDIT6: Here's the final patch code as of now: Link to post