A bunch of miscellaneous fixes today:
After making the decision to remove my original alpha-rejection code as it was causing more bugs than it was fixing, I began the arduous process of tracking down the multitude of ways in which the Nintendo 64 can handle textures that need an alpha border to be cut out.
First, I ran across Quake 64, which uses the RDP’s Alpha Compare functionality to cut out its font. Alpha Compare is relatively simple - it takes place after the Color Combiner stage and before the Blender stage. If the alpha value of the color output by the Color Combiner is less than or equal to the alpha value specified in the Blend Color register, the RDP will not pass that pixel into the Blender. If the combined alpha is greater than the alpha value specified in the Blend Color register, the RDP will rasterize the pixel through the Blender.
Sure enough, the UI’s text is now fine:

Next, I finally came to the realization that there is no other form of automatic pixel rejection based on an alpha value, thanks to a tip from a fellow who goes by the nickname of ‘Happy’. He put forth the idea that the N64 only supports Alpha Compare and no other method for automatic pixel rejection, which caused me to look a bit closer at things like the karts in Mario Kart 64 and the trees in Super Mario 64.
Sure enough, it turns out that I had to emulate the effects of the “Coverage Times Alpha” bit in the Set Other Modes command. I was only emulating the “Use Coverage As Alpha” bit, and since the transparent pixels were fully covered, they were being rasterized due to their coverage being non-zero. However, the pixel alpha value was, in fact, zero, so multiplying the coverage value by zero resulted in a final blended alpha of zero! Brilliant! Mario Kart 64 and Super Mario 64 now look much better:



I then turned to Space Station: Silicon Valley, wherein the Take 2 logo was completely black. It turns out that if both alpha values of the second Blend cycle are zero, the pixel does not become zero - in fact, the cycle is then just ignored. A quick fix later and the logo sprang to life:


Lastly is a personally controversial fix in that it wasn’t until this blog post that I realized that the fix I came up with causes problems in other games. In Bust-A-Move 2, there is an animated ground plane that is drawn line-by-line down the screen. The topmost line of the plane actually has a T coordinate of 160.0, and as a result the Set Tile Size command sets the tile’s T base to be 160.0. However, the S and T mask bits were set to 7, which results in the coordinates mirroring from 0 to 127 and back. Since the texture coordinates were being adjusted by the values in the Set Tile Size command after mirroring, and the mirroring was affecting the coordinates, they were not remaining at the appropriate position.
Before my modification to the order of coordinate adjustment, the ground plane looked like this:

After my change, the ground plane looked like this:

Unfortunately, I discovered when writing this blog post that the same change seems to affect the text in some games, so it will need further examination. Still, I feel I’m on the right track.