Ok, here's my best guess at a bit-accurate Mode 7 formula. Please verify if you can.

First, M7HOFS and M7VOFS ($210d and $210e) are 13 bits signed, as are M7CX and M7CY ($211f and $2120). The matrix parameters are all 16 bits signed.

| | |

| **Code:**
#define CLIP(a) (((a)&0x3ff)|(((a)&0xf000)?-0x400:0))
X[0,0] = ((A*CLIP(HOFS-CX))&~63) + ((B*CLIP(VOFS-CY))&~63) + (CX<<8) Y[0,0] = ((C*CLIP(HOFS-CX))&~63) + ((D*CLIP(VOFS-CY))&~63) + (CY<<8)
X[0,y] = X[0,y-1] + B Y[0,y] = Y[0,y-1] + D
X[x,y] = X[x-1,y] + A Y[x,y] = Y[x-1,y] + C | |

| | |

X and Y here ar fixed point with 8 bits of fraction, so >>8 to properly use them. When M7SEL bit 7 ($211a) is clear, further restrict X and Y to the range 0-1023. When bits 6 and 7 are both set, use the low 3 bits of X and Y to select from character zero for X or Y <0 or >1023.