Notes about SubOffscreen and other sprite subroutines (OAM index handling)

I thought of doing this because of some tidbits of information I found about the SubOffscreen routine. I may add things about other subroutines later on if I find anything, though.

So, SubOffscreen, if you don't know already, is a sprite subroutine that checks whether a sprite is offscreen. If it is, it erases the sprite, though it will be reloaded if the sprite status is 08 or greater and if $161A,x isn't set to #$FF. The interesting part about it, though, is that it actually has multiple variants. You might have noticed that custom sprites with copy-pasted subroutines often have more than just one SubOffscreen there (even though none of them that I know of actually use more than one...just another reason why copy-pasted subroutines are a waste *grumble grumble*). Many have SubOffscreenX0, X1, X2, and X3, while some even have X4-X7 as well. The only difference between the different variants is how far offscreen a sprite has to be before it will despawn.

Now, in the original SMW, there are three ROM banks that contain sprites: banks 01, 02, and 03. And since even SMW copy-pastes sprite subroutines, albeit only 3 times instead of up to 192, each bank has its own copy of SubOffscreen (as well as GetDrawInfo and the like). But the three are NOT exactly the same. SubOffscreenX0 is identical in all three banks, but the other three variants actually differ, making for a total of 8 different variants. I'll still call them SubOffscreenX0-X7, though from what I've seen, sprites that list all 8 actually are missing SubOffscreenX4 and have an extra copy of X0. (As, I believe, does at least one version of my shared subroutine patch...that thing could use an update.) They are distinguished by these tables:



The first table is a series of low bytes, and the second is the corresponding high bytes. Both of them come in pairs, with one pair corresponding to each variant of the routine, the first byte for the right side of the screen and the second for the left. Oddly, the routine actually alternates which side it checks depending on the lowest bit of $13.

So, what does this mean for sprite coding? Well, as it turns out, the subroutine is simpler than I had originally thought. It's designed kind of oddly, but the important part is that those bytes in the tables indicate the range, from the left boundary (the second byte in each pair) to the right boundary (the first byte in each pair), as follows:

- SubOffscreenX0, known in SMW as "SubOffscreen0Bnk1", "SubOffscreen0Bnk2", and "SubOffscreen0Bnk3", is used by many sprites in banks 1, 2, and 3. It goes from -$40 to +$30 ($0130,$FFC0).
- SubOffscreenX1, known in SMW as "SubOffscreen1Bnk1" and "SubOffscreen1Bnk2", is used by the horizontal red Para-Koopa, checkerboard and flying rock platforms, line-guided sprites, and Big Boo in bank 1 and by the dolphins, Hammer Brother, and his platform in bank 2. It goes from -$40 to +$A0 ($01A0,$FFC0).
- SubOffscreenX2, known in SMW as "SubOffscreen2Bnk1", is used by the brown chained platform (sprite 5F). It goes from -$10 to +$A0 ($01A0,$FFF0).
- SubOffscreenX3, known in SMW as "SubOffscreen3Bnk1", is used by the Eeries. It goes from -$70 to +$60 ($0160,$FF90).
- SubOffscreenX4, known in SMW as "SubOffscreen2Bnk2", is used by the scale platforms. It goes from -$90 to +$A0 ($01A0,$FF70)
- SubOffscreenX5, known in SMW as "SubOffscreen1Bnk3", is not used by any sprites. It goes from -$80 to +$A0 ($01A0,$FF80).
- SubOffscreenX6, known in SMW as "SubOffscreen2Bnk3", is not used by any sprites. It goes from $40 to +$A0 ($0040,$01A0).
- SubOffscreenX7, known in SMW as "SubOffscreen3Bnk2" and "SubOffscreen3Bnk3", is used by the wall-following sprites, Ball 'n' Chain, and rotating gray platform in bank 2 and by the Mega Mole in bank 3. It goes from -$50 to +$60 ($FFB0,$0160).

The non-X0 variants are actually used by surprisingly few sprites, especially X2, X3, and X4, which are used by only one sprite each (well, technically two in the case of X3). But the really interesting one is SubOffscreenX6 (or SubOffscreen2Bnk3 if you prefer). Notice the lack of a sign on that first value? SubOffscreenX6 is the only variant that does not use a negative number for its second table value; it uses $0040 instead, which means that a sprite using it will actually count as offscreen even if it is 64 pixels away from the left screen boundary. I suppose it's not surprising that this one was never actually used for any sprites in SMW. Though SubOffscreenX5 (or SubOffscreen1Bnk3) doesn't have that excuse; its range is fairly ordinary, but no sprites ever used it either.

It's also worth noting that it is indeed possible to add custom values to the range table if you wanted a SubOffscreenX8 or something, and values greater than $01FF or less than $FF00 do still work properly. In fact, I may tweak that routine to where you can specify the range values manually if you give it the right inputs.

----------------

I'm working on a hack! Check it out here. Progress: 64/95 levels.

Seems to me a lot of this could be simplified into one routine if we just required the distance offscreen to be loaded into a register before calling the routine.

Now here's something I typed up back in June but didn't think to post here until just recently: sprite OAM indexes. Plenty of people know about the No More Sprite Tile Limits patch, fewer people probably know that SMW normally hardcodes a sprite's OAM index depending on which slot it's in and what the sprite memory setting is, but there is a method to these hardcoded slots. As far as I can tell:

- There is a set of 7 tables starting at $02A773 related to allowed sprite indexes. When the sprite is loaded, it checks if the sprite number is a reserved one. (Each sprite memory setting can have up to 2 sprites reserved for certain slots, and the tables for which sprite numbers are reserved are at $02A7D2 and $02A7E4.) If not, it uses the default table for what the highest index is ($02A773) as well as the lowest ($02A7AC), but if so, it uses the tables for the reserved sprites ($02A786 and $02A799 for the high end and $02A7BF for the low; the low end for the second reserved sprite is always $FF).
- It then checks each sprite slot starting at the high boundary and ends the loop if it reaches the low one. (See $02A914-$02A921.) Once it finds a free sprite slot, it uses that as the index for the newly-loaded sprite.
- Now, back to the OAM slot thing...the table at $07F0B4 is used to index the table of possible OAM indexes at $07F000, and the sprite memory setting is in turn used to index the table at $07F0B4. The index to $07F000 is equal to the index from $07F0B4 plus the sprite index (which should normally be in X and/or $15E9). The value from the $07F000 table is then stored to the sprite OAM index at $15EA,x.

Effectively, all this stuff seems to determine what OAM index each sprite will start at and, as a result, how many tiles each sprite is allowed to use without making parts of other sprite disappear. The OAM index table looks like this:



That's not an exact replication of the table at $07F000, because a few of the values at $07F0B4 actually point to the same place in $07F000, so I just listed those separately anyway. I also made a table for how many tiles each sprite gets. The rows are sprite memory indexes, and the columns are sprite slot indexes.



I listed a sprite memory setting 13 even though Lunar Magic only shows up to 12 and the game doesn't seem designed for settings beyond 12, but the table at $07F0B4 does have a 0x14th index, so I figured...why not? A few of the tile count values I'm not sure on, though. Also, a couple of the sprite memory settings don't ever seem to be used, particularly the duplicate ones. Settings 06, 0F, 11, and 12 are duplicates. Setting 11 is used in Choco-Ghost House (or in hacks, anywhere you'd put a Fishin' Boo without the patch), but the reserved sprite for setting 06 is sprite 88, the winged cage, and settings 0F and 12 don't even have reserved sprites. Though 0F is used in the first room of Forest Fortress (or anywhere that has the background candles but no revolving net doors, it seems) and 12 is used for Iggy and Larry. I'm also pretty sure setting 03 isn't used either, given that its reserved sprite also never shows up in the original SMW (though at least that one is fully functional). Settings 05 and 08 don't seem to show up anywhere either. (05 seems designed for a room with more than one Big Boo, but 08...well, given that it's basically 00 with fewer slots and considering some of the other interactions between sprite memory and cluster sprites, maybe it was intended for use with the Swooper ceiling?)

On a side note, the No More Sprite Tile Limits patch honestly does not seem to use a very efficient algorithm...there has to be a better way to find an OAM index than checking literally every free one. I tried to make a version that used free RAM to keep track of the last used slot, but I couldn't get it to work.

----------------

I'm working on a hack! Check it out here. Progress: 64/95 levels.

Speaking of that, I notice that the Grey Falling Platform (from star road), has a "Respawn boundary" VERY CLOSE the left edge of the screen, so if you make it fall, and slowly scroll the screen to the right, you'll see that the platform "pops into existence" on he left edge of the screen. Speaking of despawn/Respawn boundary, when the sprite enters the screen, does the (Re)spawn boundaries (2 of them) are more closer to the center, and the despawn boundaries (2 of them again) are further away from the center of the screen? Here is a visual:

Or that the respawn and despawn boundaries are the exact same spot?

---

Give thanks to RPG hacker for working on Asar.