SPC7110 Decompression

Strictly for discussing ZSNES development and for submitting code. You can also join us on IRC at irc.libera.chat in #zsnes.
Please, no requests here.

Moderator: ZSNES Mods

Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

SPC7110 Decompression

Post by Nach »

Anyone want to lend me a hand?

Here's what I got so far from Far East of Eden Zero:

Hudson Logo:
Image
Scary doors with Logo in upper right:
Image
The parting waters:
Image
Image
Image
Birds flying by:
Image
Cat fighting warrior:
Image
Image
Crying guy in the rain:
Image
Image
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
byuu

Post by byuu »

All of the graphics are stored using compression modes 1 and 2, which haven't been cracked yet.

The mode 0 stuff appears to all be miscellaneous junk, hence why you aren't seeing anything remotely useful.
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

Ah.

Would that explain why the outline of stuff looks right somewhat? Since if I fill the decompression buffer with random junk instead, I don't see anything remotely recognizable if anything at all.

Edit:
If you look in the last two images, the font for example looks perfect.
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
byuu

Post by byuu »

Hard to say. It seems modes 1 and 2 also use an 8-bit QM-coder, but there's probably some additional decoding going on or somesuch that we don't have figured out yet.

You probably shouldn't have posted this here, it'll just get everyone's hopes up. We may not be able to crack modes 1 and 2; or it may take a really, really long time.

Myself, I'm battling with this damn S-RTC timer. For Daikaijuu Monogatari 2, it's simple enough to just fill in the struct with localtime() data. But for FEoEZ, it tries setting the time to 1999-12-31 23:59:59, then waiting a second, then verifying it wrapped to 2000-01-01 00:00:00 ...

I'll have to do it like SNES9x, using a skew against the local system time. I tried to shave off 90% of that code by using mktime(), but whenever I give mktime() 1999-12-31 23:59:59 (or technically 99-11-31 23:59:59 in its format), and convert that back with localtime(), I get 1999-12-31 22:59:59 ... must be some sort of bug.

Fuck.
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

byuu wrote:Hard to say. It seems modes 1 and 2 also use an 8-bit QM-coder, but there's probably some additional decoding going on or somesuch that we don't have figured out yet.
Speaking of which, do we need a method to rebuild the index file? I'm still using it.
byuu wrote: You probably shouldn't have posted this here, it'll just get everyone's hopes up. We may not be able to crack modes 1 and 2; or it may take a really, really long time.
Who really cares? The games are already playable. Anyone who is hoping probably has patience. I've waited for so long, I don't mind waiting another couple of years if that's what modes 1 and 2 need.
byuu wrote: Myself, I'm battling with this damn S-RTC timer. For Daikaijuu Monogatari 2, it's simple enough to just fill in the struct with localtime() data. But for FEoEZ, it tries setting the time to 1999-12-31 23:59:59, then waiting a second, then verifying it wrapped to 2000-01-01 00:00:00 ...
I hope you're using the doc I e-mailed you, and not the outdated doc that neviksti posted.
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
caitsith2
New Member
Posts: 8
Joined: Tue Aug 03, 2004 9:27 am

Post by caitsith2 »

Mode 1 has been cracked now.
neviksti
Lurker
Posts: 122
Joined: Thu Jul 29, 2004 6:15 am

Post by neviksti »

Nach wrote:I hope you're using the doc I e-mailed you, and not the outdated doc that neviksti posted.
I posted what I had, and requested "If anyone has more info please post."
Why didn't you just share a link so we can all see?
Or if it is "proprietary" at least note corrections to the "documents" (I noted the few I noticed so far).

I keep running into bizarre startup errors with my "on real hardware" decompressor. I can't figure out what causes it, but several resets usually does the trick. I'd love, for extra peace of mind, to iron out these little bugs when running tests. So if you have better information, please do share in that thread.

Are you "jolly_codger"? Who's jolly codger?
byuu

Post by byuu »

The document he sent was an update by The Dumper. It was definitely a huge improvement, having the corrected bit fields for the data port control register, but there were still far too many missing flags and too much wrong info.

For instance:

Code: Select all

00 - manual decompression, $4800 is used to read directly from the data rom
     ( really - disable offset modes x2/x4 (x8?) )

02 - hardware decompression, decompressed data is mapped to $50:0000, $4800 can be used to read sequentially from bank $50
     ( enable offset modes based on first byte in table index?  offset_mode/addr-h/addr-m/addr-l
       offset mode: offset start by $4805/6 times value
       00 = x1 offset
       01 = x2 offset ?
       02 = x4 offset ?
       03 = ??? )
MDH performs transfers with $480b = 0, and it fully expects it to be decompressed. It also makes the first byte of the table seem like only an index scalar. It's actually that, plus the decompression mode in one. It's really bizarre, actually. And I'm not sure how hardware can manage to start decompressing in the middle of data like that. Almost certainly, it's clearing $480c's ready flag, and slowly decoding and dumping data up to that point, so our emulation will likely be running too fast.

I was going to see about writing up some hardware tests to throw over to you guys if my copier-swap trick doesn't work. There's so much that really needs to be tested still.

I don't know if this doc is supposed to be secret (though why would it be?), so I'll PM it to you. If Nach gives the ok, we can post it here I guess.
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

neviksti wrote:
Nach wrote:I hope you're using the doc I e-mailed you, and not the outdated doc that neviksti posted.
I posted what I had, and requested "If anyone has more info please post."
I don't have an account on NESDev, nor do I plan to make one.
neviksti wrote: Why didn't you just share a link so we can all see?
Don't have a link to it. My host went down months ago. There's something called e-mail.
neviksti wrote: Or if it is "proprietary".
It was posted on the old Snes9x forum years ago.
neviksti wrote: Are you "jolly_codger"?
No.
neviksti wrote:Who's jolly codger?
That's a good question.

Anyways, since you're on this forum again (where we could've used you for a while now).

Code: Select all

SPC7110 Info, Reverse Engineered by & (c) Dark Force

New data added by The Dumper


SPC7110 Rom Map (far east of eden zero)
40mb total: 8mb program rom (000000-0FFFFF) + 32mb data rom (100000-4FFFFF)
program rom mapped to $C0:0000-$CF:FFFF/$80:0000-$8F:FFFF
data rom in 8mb chunks can be mapped to hirom areas using the bank mapping regs
default mapping: data rom 000000-2FFFFF mapped to $D0:0000-$FF:FFFF
data decompressed from data rom by spc7110 mapped to $50:0000-$50:FFFF

SPC7110 Sram Map
$006000 - $007FFF sram 8k (slow rom?)
$306000 - $307FFF mirrored sram from $006000 - $007FFF (fast rom?)


defval: = default power-on/reset register value

SPC7110 Registers, banks $00/$80
$4800   DECOMPRESSED DATA CONTINUOUS READ PORT: returns a decompressed value from bank $50 and decrements 16 bit counter value at $4809/A by 1
$4801   COMPRESSED DATA TABLE POINTER: ($0000FF) low offset, defval:00
$4802   COMPRESSED DATA TABLE POINTER: ($00FF00) high offset, defval:00
$4803   COMPRESSED DATA TABLE POINTER: ($FF0000) bank, defval:00
$4804   COMPRESSED DATA TABLE INDEX: index of 32 bit compressed data pointer (big-endian), defval:00
$4805   DECOMPRESSED DATA OFFSET: ($00FF) low byte of offset in bank $50, defval:00
$4806   DECOMPRESSED DATA OFFSET: ($FF00) high byte of offset in bank $50, defval:00
$4807   DMA CHANNEL FOR DECOMPRESSION, set to match snes dma channel used for compressed data, defval:00
$4808   C r/w option, unknown, defval:00
$4809   COMPRESSION LENGTH COUNTER: ($00FF) low byte, defval:00
            write: set start counter value low byte
            read: get counter value low byte
$480A   COMPRESSION LENGTH COUNTER: ($FF00) high byte, defval:00
            write: set start counter value high byte
            read: get counter value high byte
$480B   DECOMPRESSION COMMAND MODE: see decompression command modes, defval:00
$480C   DECOMPRESSION FINISHED STATUS: high bit set = done, high bit clear = processing, cleared after successful read,
            high bit is cleared after writing to $4806, $4809/A is set to compressed data length, defval:00


;----------

My testing leads me to believe that the direct ROM read section starts out
as inactive.

One of the ways to activate direct reads is to write a non-zero value to $4813.
No other action need be taken. You can write a non-zero value and immediately
write a zero to it and that's OK.  The order of writes to $4811/2/3 don't
seem to matter so long as $4813 has been written to once with a non-zero
value.  There may be a way to deactivate the direct reads again (maybe a
decompression cycle?) but I haven't tested that.

There appears to be another way to activate direct reads that is more complex
but I haven't quite pinned that down yet.

;----------



$4810   DATA ROM CONTINUOUS READ PORT: returns a byte from data rom at data rom pointer location, defval:00
$4811   DATA ROM POINTER: ($0000FF) r/w low offset, defval:00   ( $4811/2/3 may increment on a $4810 read depending on data rom command mode byte )
$4812   DATA ROM POINTER: ($00FF00) r/w high offset, defval:00
$4813   DATA ROM POINTER: ($FF0000) r/w bank offset, defval:00
            bank offset is zero based from start of data rom: banks $00-$3f data rom -> $10-$4f full rom


; The $4814/$4815 pair is sometimes incremented on $4810 reads (depending on data rom command mode byte)

$4814   DATA ROM POINTER ADJUST: ($00FF) low byte, defval:00   (Pointer is 16-bits - wraps at 16-bits)
$4815   DATA ROM POINTER ADJUST: ($FF00) high byte, defval:00  (Addition with $4811/2/3 is 24-bits)


$4816   DATA ROM POINTER INCREMENT: ($00FF) low byte, defval:00
$4817   DATA ROM POINTER INCREMENT: ($FF00) high byte, defval:00
$4818   DATA ROM COMMAND MODE: bit field control of data rom pointer (see data rom command mode byte), defval:00
            write: set command mode,
            read: performs action instead of returning value, unknown purpose,
            command mode is loaded to $4818 but only set after writing to both $4814 and $4815 in any order
$481A   DATA ROM READ AFTER ADJUST PORT: returns a byte from data rom at data rom pointer location + adjust value ($4814/5), defval:00
$4820   16 BIT MULTIPLICAND: ($00FF) low byte, defval:00
        32 BIT DIVIDEND: ($000000FF) low byte of low word, defval:00
$4821   16 BIT MULTIPLICAND: ($FF00) high byte, defval:00
        32 BIT DIVIDEND: ($0000FF00) high byte of low word, defval:00
$4822   32 BIT DIVIDEND: ($00FF0000) low byte of high word, defval:00
$4823   32 BIT DIVIDEND: ($FF000000) high byte of high word, defval:00
$4824   16 BIT MULTIPLIER: ($00FF) low byte, defval:00
$4825   16 BIT MULTIPLIER: ($FF00) high byte, defval:00
$4826   16 BIT DIVISOR: ($00FF), defval:00
$4827   16 BIT DIVISOR: ($FF00), defval:00
$4828   32 BIT PRODUCT: ($000000FF) low byte of low word, defval:00
        32 BIT QUOTIENT:($000000FF) low byte of low word, defval:00
$4829   32 BIT PRODUCT: ($0000FF00) high byte of low word, defval:00
        32 BIT QUOTIENT:($0000FF00) high byte of low word, defval:00
$482A   32 BIT PRODUCT: ($00FF0000) low byte of high word, defval:00
        32 BIT QUOTIENT:($00FF0000) low byte of high word, defval:00
$482B   32 BIT PRODUCT: ($FF000000) high byte of high word, defval:00
        32 BIT QUOTIENT:($FF000000) high byte of high word, defval:00
$482C   16 BIT REMAINDER: ($00FF) low byte, defval:00
$482D   16 BIT REMAINDER: ($FF00) high byte, defval:00
$482E   MUL/DIV RESET, write = reset $4820 to $482D, defval:00
$482F   MUL/DIV FINISHED STATUS: high bit, on = processing, off = finished,
            high bit is set after a write to multiplier or divisor regs $4825/$4827, defval:00
$4830   SRAM CHIP ENABLE/DISABLE: high bit, on = enable, off = disable, defval:00
$4831   ROM BANK MAPPING A $D0:0000-$DF:FFFF, see bank mapping modes, defval:00
$4832   ROM BANK MAPPING B $E0:0000-$EF:FFFF, see bank mapping modes, defval:01
$4833   ROM BANK MAPPING C $F0:0000-$FF:FFFF, see bank mapping modes, defval:02
$4834   SRAM BANK MAPPING?, workings unknown, defval:00
$4840   RTC CHIP ENABLE/DISABLE: low bit, on = enable, off = disable, defval:00
$4841   RTC INDEX/DATA PORT:
            first write after rtc enable: rtc command mode byte (see rtc command modes)
            subsequent writes: index of rtc register to read/write (00-0f)
            read: returns value of indexed rtc register
            auto-increment of register index occurs after each subsequent read/write
$4842   RTC READY STATUS: high bit set = ready, high bit clear = processing, tested before reading rtc data
            high bit cleared after successful read


Decompression Command Modes:
00 - manual decompression, $4800 is used to read directly from the data rom
     ( really - disable offset modes x2/x4 (x8?) )

02 - hardware decompression, decompressed data is mapped to $50:0000, $4800 can be used to read sequentially from bank $50
     ( enable offset modes based on first byte in table index?  offset_mode/addr-h/addr-m/addr-l
       offset mode: offset start by $4805/6 times value
       00 = x1 offset
       01 = x2 offset ?
       02 = x4 offset ?
       03 = ??? )

Note: decompression mode is activated after writing to $4806 and finishes after reading the high bit of $480C


Note: The $4814-$4815 pair can optionally be used as an offset to the base address in $4811/$4812/$4813
      The $4816-$4817 pair can optionally be used as an increment value

Data Rom Command Mode Byte:
X6543210
||||||||
||||||||__ : 0 - use 1 as the increment value, add immediately after reading $4810
|||||||    : 1 - use $4816 as increment value, add immediately after reading $4810
|||||||___ : 0 - disable $4814/5 offset addition
||||||     : 1 - enable $4814/5 addition as an offset to the base address in $4811/2/3
||||||____ : 0 - unsigned calculation for $4816
|||||        1 - signed calculation for $4816
|||||_____ : 0 - unsigned calculation for $4814
||||         1 - signed calculation for $4814
||||______ : 0 - increment value gets added to $4811/2/3
|||          1 - increment value gets added to $4814/5
| \_______ : 00 - disable exceptions to $4814/5 offset addition (listed below)
|          : 01 - 8 bit offset addition using $4814, immediately after writing to $4814/5
|          : 10 - 16 bit offset addition using $4814/5, immediately after writing to $4814/5
|          : 11 - 16 bit offset addition using $4814/5, only after reading $481A
|_________ : unused

Note: the data rom command mode is activated only after registers $4814 and $4815 have been written to, regardless of the order they were written to


Bank Mapping Modes:
00 - use data rom 000000-0FFFFF for mapping
01 - use data rom 100000-1FFFFF for mapping
02 - use data rom 200000-2FFFFF for mapping
03 - use data rom 300000-3FFFFF for mapping


RTC Command Modes:
03 - normal sequential read/write mode
0c - sequential read/write mode also
other commands unknown as of yet


RTC Registers (8 bit values):   [default values]
00 - seconds 1's digit                 00
01 - seconds 10's digit                00
02 - minutes 1's digit                 00
03 - minutes 10's digit                00
04 - hours 1's digit                   00
05 - hours 10's digit                  00
06 - day of month 1's digit            01
07 - day of month 10's digit           00
08 - month 1's digit                   01
09 - month 10's digit                  00
0a - year 1's digit                    00
0b - year 10's digit                   00
0c - day of week                       00
0d - control register                  01
0e - control register                  0F
0f - control register                  06


RTC Control Register Bits:
Register 0d
XXXXXX10
      ||_ : 0 - normal timer operation
      |     1 - pause rtc registers updating, time still increments normally
      |__ : 0 - disable rtc interrupts
            1 - enable rtc interrupts

Register 0e
XXXX3210
    \\\|_ : unknown

Register 0f
XXXXX210
     |||_ : 0 - normal
     ||     1 - stops timer and resets seconds to 00 (date set to 01/01/00 00:00:00 ?)
     ||__ : 0 - normal timer operation
     |      1 - stop timer
     |___ : 0 - 12 hour time, bit 2 of "hours 10's digit" register contains am/pm bit (am=0,pm=1)
            1 - 24 hour time
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
Nightcrawler
Romhacking God
Posts: 922
Joined: Wed Jul 28, 2004 11:27 pm
Contact:

Post by Nightcrawler »

Can I take that document and add it to RHDN? Or are you guys still working on a better one? It would be nice to archive it there for safekeeping and ease of locating for anyone interested.

By all means if you have any other special chip or other hardware docs not there, I'm sure it would be appreciated to add by many. I'm sure I've seen some buried material on the SNES9x and NESDev forums that can't be found by anyone not 'in the know'. ;)
[url=http://transcorp.romhacking.net]TransCorp[/url] - Home of the Dual Orb 2, Cho Mahou Tairyku Wozz, and Emerald Dragon SFC/SNES translations.
[url=http://www.romhacking.net]ROMhacking.net[/url] - The central hub of the ROM hacking community.
Deathlike2
ZSNES Developer
ZSNES Developer
Posts: 6747
Joined: Tue Dec 28, 2004 6:47 am

Post by Deathlike2 »

Nightcrawler wrote:Can I take that document and add it to RHDN? Or are you guys still working on a better one? It would be nice to archive it there for safekeeping and ease of locating for anyone interested.

By all means if you have any other special chip or other hardware docs not there, I'm sure it would be appreciated to add by many. I'm sure I've seen some buried material on the SNES9x and NESDev forums that can't be found by anyone not 'in the know'. ;)
I'd say it would happen.. but that's not my say though. :P

Just brace for idiots wondering when FEOZ has the "when it will get a translation" question when you need to judge that with the Star Ocean translation's time before release.. and what this sounds to me is that it will take much longer to do because of the massive amounts of data in this game..

So, the translation should be done after 2010! ;)
Continuing [url=http://slickproductions.org/forum/index.php?board=13.0]FF4[/url] Research...
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

Nightcrawler wrote:Can I take that document and add it to RHDN?
Go right ahead.
Nightcrawler wrote:Or are you guys still working on a better one?
Well, some of it I'm finding to be a bit inaccurate. Some of it is slightly ambiguous. And overall, it leaves several questions open.

A newer one would be a good idea, although I think I'll need someone to do some research for us, to verify a few things, and clear up some questions.
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

Okay, there's several issues involved with emulating the SPC7110. I'd appreciate if those who can can do the research necessary so we can get to the bottom of this.

When writing to $4806 I need to verify some of what happens.
Is decompression done to $50:0000, $50:($4805,6), or $50:($4805,6)<<mode?
At this point is ($4805,6)<<mode written back to ($4805,6)?
What happens if the previous decompression wasn't fully processed and this is written to again?

How does $480C work? When is the bit there set and cleared?
Is the bit set when there is a write to $4806, and then cleared when there is a read, or perhaps a full read from $4800? Reading from $480C sets ($4809,A) immediately after a write to $4806 or only after the decompression is finished?

$4800 does data ROM reads when $480B is 0, but where in the data ROM does it read from? Does it do anything besides reading direct data ROM or from the buffer at $50:0000-FFFF? Such as stream decompression? Is ($4805,6) incremented?

Another thing. Can someone go through the code in FEoEZ and MDH and see what exactly is the logic in write to ($4805,6)? I know it's based on math in conjunction with ($4809,A), but haven't been able to figure it out. If someone can pull it out from the games themselves, that'd be a great help.

neviksti:
I found this little nugget in your decompression code:

Code: Select all

		if((i+1)<len)
		{
			*dataout = (BIT(out,30)<<7) + (BIT(out,26)<<6) + (BIT(out,22)<<5) + (BIT(out,18)<<4)
				+ (BIT(out,14)<<3) + (BIT(out,10)<<2) + (BIT(out,6)<<1) + BIT(out,2);
			dataout++;
		}
Where did you come up with the logic for that? Did you find that was the only way to match a 32KB decompress read? If so, is your code correct when a decompress read <32KB is performed?

Due to issues labeled above, I notice byuu, and to some extent myself are cheating when asking for decompression, passing the wrong len, and in fact getting part of our buffer not decompressing as should be due to that logic changed based on the length.

Thanks for your help.
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
neviksti
Lurker
Posts: 122
Joined: Thu Jul 29, 2004 6:15 am

Post by neviksti »

Nach wrote:neviksti:
I found this little nugget in your decompression code:

Code: Select all

		if((i+1)<len)
		{
			*dataout = (BIT(out,30)<<7) + (BIT(out,26)<<6) + (BIT(out,22)<<5) + (BIT(out,18)<<4)
				+ (BIT(out,14)<<3) + (BIT(out,10)<<2) + (BIT(out,6)<<1) + BIT(out,2);
			dataout++;
		}
Where did you come up with the logic for that? Did you find that was the only way to match a 32KB decompress read? If so, is your code correct when a decompress read <32KB is performed?
The routine is passed a pointer to a buffer and a length parameter. I tried to write the code to respect that length parameter and not write beyond that in the buffer. That is all that test is doing... making sure it is okay to write the next byte to the buffer.

I'm sorry for the messy code.
I agree with byuu, it is probably best to change the code to allow calling a "setup" routine, and then a decompression routine that just returns the next byte. Byuu made a comment that he is waiting for the code to stabalize, but I'm fairly confident the decompressions are correct and no other updates to it are needed (maybe that's not what he meant?).

If you guys want I can change the code to be "setup" and "next byte" instead of an all at once thing. For all I know you guys are already in the middle of that though.
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

neviksti wrote: If you guys want I can change the code to be "setup" and "next byte" instead of an all at once thing. For all I know you guys are already in the middle of that though.
I was in the middle, but I'm trying to remove all dependence on the "i" variable, that's one of them that bothered me.
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
byuu

Post by byuu »

Byuu made a comment that he is waiting for the code to stabalize, but I'm fairly confident the decompressions are correct and no other updates to it are needed (maybe that's not what he meant?).

If you guys want I can change the code to be "setup" and "next byte" instead of an all at once thing. For all I know you guys are already in the middle of that though.
No, I was waiting until you gave your final say that you were done editing the code. The last thing I wanted to do was convert 99% of it, only to get an entirely new version and have to start over :)

But, if you're done with it now ... I suppose I can get to work on that. I really wanted to hook up my cothreads so that the only thing I'd have to change is add a yield() after each output write (and delete/recreate the thread each time a new decompression was triggered), but that'd incur some crazy performance penalties ... so I'll have to make it a class with lots of object-local variables, I suppose. Not too good for C89/C99 users ...

So then, any chance you'd be willing to write compressors for each mode, just to complete the whole thing? :)
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

byuu wrote: So then, any chance you'd be willing to write compressors for each mode, just to complete the whole thing? :)
We do need compressors, so we can have the translation groups put in new locale/language graphics where appropriate.
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
Nightcrawler
Romhacking God
Posts: 922
Joined: Wed Jul 28, 2004 11:27 pm
Contact:

Post by Nightcrawler »

Nach wrote:
byuu wrote: So then, any chance you'd be willing to write compressors for each mode, just to complete the whole thing? :)
We do need compressors, so we can have the translation groups put in new locale/language graphics where appropriate.
Yes, that would be very helpful. I previously looked into doing the translation outside of the compressed graphics (before the chip was cracked) and space was certainly at a premium. I thought it should be possible to do, but difficult to say the least and some corners might need to be cut. It should be much more feasible with a working compressor and being able to utilize much of the same space the compressed graphics use already.
[url=http://transcorp.romhacking.net]TransCorp[/url] - Home of the Dual Orb 2, Cho Mahou Tairyku Wozz, and Emerald Dragon SFC/SNES translations.
[url=http://www.romhacking.net]ROMhacking.net[/url] - The central hub of the ROM hacking community.
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

Technically, someone can append graphics to one of the dumps, and have the game set read directly from data ROM mode, if we can ever get that implemented properly. So no need to recompress your new graphics.
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
neviksti
Lurker
Posts: 122
Joined: Thu Jul 29, 2004 6:15 am

Post by neviksti »

Okay, here is utility stuff for SPC7110 mode 2. (meant for 4bitplane SNES graphics, for example sprites)
win32 exe's
http://neviksti.com/SPC7110/DecompMode2.exe
http://neviksti.com/SPC7110/CompMode2.exe
running without arguments should explain what to do hopefully

source files so rom hackers can make their own tools
http://neviksti.com/SPC7110/DecompMode2.c
http://neviksti.com/SPC7110/CompMode2.c


Note:
The decompression code has been tweaked slightly to change the return value (the decompressed data of course is still the same). This probably has no use for anyone except algorithm testers and rom hackers.
Before, the return value was the number of bytes from the input that were fed into the "val" variable (basically how many bytes had bits that ended up in calculations).

I have now changed the return value to reflect the number of bytes from the input needed to fix the output data. This is the more relevant quantity for algorithm testers and rom hackers. This required more "data following" than I previously wanted to bother with.


Caitsith,
If you are curious, you can go back and see how many previously reported underdumps just barely passed the mark between this byte cutoff and the previous one's (and so aren't underdumps). This shouldn't shift things more than a byte, so I'm not sure how many it will actually change. If your hunch that they overlapped the data end of one entry / data begin of another to save some bytes is true (which seems reasonable), there will still be plenty of underdumps.

Nightcrawler,
I'll finish up the other modes later. Please try these out to make sure they work as you'd expect. Let me know if you find anything wrong, or need something changed / explained, because I'd prefer to finish up any core code you need to make your tools while this is still fresh in my mind.

EDIT: I have no way of knowing ahead of time how much room I need for compressed or decompressed data. So I limit the uncompressed data to $8000 bytes by hand. You can always increase this of course, I just thought I'd point it out now.

EDIT2: Sometimes my compressor gives files shorter than the .compressed files in the test data pack. My code looks fine to me, and appending different values doesn't change the result... so it may be that I optimized my "compressed data end calculation" a bit beyond what the game manufacturer themselves did. Or my code could just be wrong. Let me know if you can find a situation in which it truncates the data too much.
caitsith2
New Member
Posts: 8
Joined: Tue Aug 03, 2004 9:27 am

Post by caitsith2 »

Tested one single raw entry, and the compressor seems to be matching byte for byte exactly to the original data. The specific is raw entry 000008/00, 32768 bytes uncompressed, 14269 bytes compressed. I may just automate this testing to see how bit-perfect the compressor is.
caitsith2
New Member
Posts: 8
Joined: Tue Aug 03, 2004 9:27 am

Post by caitsith2 »

Okay, I now totally automated the process, and ran it against the same raw gfx pack data as I ran the decompressor code against, and I declare the compressor to be absolutely BIT-PERFECT, for Mode 2 data. Now just mode 1 and mode 0 to be written for. (how long will that take?)
Nach
ZSNES Developer
ZSNES Developer
Posts: 3904
Joined: Tue Jul 27, 2004 10:54 pm
Location: Solar powered park bench
Contact:

Post by Nach »

neviksti wrote: The decompression code has been tweaked slightly to change the return value (the decompressed data of course is still the same). This probably has no use for anyone except algorithm testers and rom hackers.
Dark Force / The Dumper doc wrote: $480C DECOMPRESSION FINISHED STATUS: high bit set = done, high bit clear = processing, cleared after successful read,
high bit is cleared after writing to $4806, $4809/A is set to compressed data length, defval:00
Is the return value the correct values to write to ($4809/A) ?
May 9 2007 - NSRT 3.4, now with lots of hashing and even more accurate information! Go download it.
_____________
Insane Coding
neviksti
Lurker
Posts: 122
Joined: Thu Jul 29, 2004 6:15 am

Post by neviksti »

caitsith2 wrote:Okay, I now totally automated the process, and ran it against the same raw gfx pack data as I ran the decompressor code against, and I declare the compressor to be absolutely BIT-PERFECT, for Mode 2 data. Now just mode 1 and mode 0 to be written for. (how long will that take?)
Yeah! Thanks for helping out.
The other modes should be quick now that the code for the arithmetic encoder is written up (and tested). I was just wanted to make sure everything was okay before basing more things on that code.
Nach wrote:Is the return value the correct values to write to ($4809/A) ?
You don't need to write anything to those registers. I never bothered to and the decompressions always worked with the unmodified cart. I had some weirdness with the modified cart that required me to reset a couple times before everything would work, and I never fully tracked it down, but I really think it is just something strange with the FIFO in my mod.

EDIT: Wait, did you mean for the emulator? The hardware can't tell how long the compressed data is. It doesn't write any compressed data length to those registers.

I may still be misunderstanding your question though.
Post Reply