bsnes v0.033 released

[byuu]

This release adds SPC7110 emulation, without the need for graphics packs!!, and a rewritten S-RTC (real-time clock) emulator.

SPC7110 support means that Far East of Eden Zero, FEoEZ: Shounen Jump Edition, Momotarou Dentetsu Happy and Super Power League 4 are now all fully playable. I will warn you, the emulation is very slow in this version -- while most areas of each game will run at the same speed as other games, there are a few peak moments where speed will drop by up to ~50%. The reason for the slow-down is that I am currently uncertain how to determine the amount of data to decompress in advance, so I default to the maximum amount possible. The reason I am releasing now anyway, is because I beleive in the "release early, release often" paradigm. It will likely take me a few weeks to finish researching this chip, and I didn't want to keep the work I had private during that time. But rest assured, bsnes v034 should feature much faster SPC7110 emulation.

neviksti, Andreas Naive and jolly_codger worked non-stop on the SPC7110 decompression algorithm for the past two weeks. caitsith2 provided valuable data to the effort. I only wish that I could've been of some use, but alas, I had no role in this. In the end, it was neviksti who managed to crack all three(!!) compression modes of this chip, which turned out to be a customized 8-bit QM-coder with a prediction model. You can read more about this here. I would also like to thank Dark Force and John Weidman (aka The Dumper) for their research notes on the SPC7110 register interface.

For those who don't understand the hoopla about figuring out this compression algorithm when we already had graphics pack simulation, I should note that we have since found a few errors in these packs. Not to mention, you no longer need ~4-16MB packs for each game you wish to run. They work like any other game now. Better still, the chip can now be used to compress new graphics, eg for any future translation efforts on these titles.

The real-time clocks in both Far East of Eden Zero and Dai Kaijuu Monogatari 2 will now save a ".rtc" file in your save folder, which contains the clock as set by the video game, as well as a timestamp from your computer when the time was last updated. It uses the difference between the saved timestamp and current time to update the time. This allows you to specify any time you like, whereas previously bsnes would just use your computer's current time, ignoring the time you set in-game. It also allows the "round clock by 30 seconds" option in both games to work. I avoided this before because this method makes supporting daylight savings time and such impractical, although I should note that the original hardware did not support DST, either. This method was required to pass the SPC7110 tests, and is overall much more faithful to how the original chips worked.

Once again, I'd really like to personally thank neviksti for his tireless efforts. Eliminating graphics packs from SNES emulation was one of my primary reasons for getting involved in the SNES emulation scene. That neviksti managed to crack this algorithm means a lot to me. Thank you so much, neviksti. This release is dedicated to you, now go get some sleep ;)

[odditude]

Congratulations to all!

[neo_bahamut1985]

That must've been one heck of chip to crack!

[FitzRoy]

Damn, many thanks to neviksti and Andreas, and byuu for adding it so quickly to bsnes. I'm amazed at how quickly they managed to crack it. All that experience and knowledge got pooled, we're really lucky to still have them around.

It's funny, I pulled the bounty to fund decapping stuff instead around the time I started the "special chip" thread, totally unaware that byuu had a thread on nesdev trying to generate interest. I don't know if that was a carrot for them, but they're each entitled to a portion if they want.

The only real issue I've noticed so far is that this decompression is a heck of a lot more taxing than S-DD1. I get some pretty massive drops, 30fps on SPL4's title screen for example. Room for optimization or nature of the beast?

Again, awesome work.

[byuu]

Lots of room for optimization.

The chip has a length / decrement register, but it does no good since you can read forever from the decompression port. Many times, games do not set the register at all, so you can't use that.

neviksti's decompression routines take fixed sizes, eg "give me an input buffer, an output buffer, and the number of bytes you want decompressed."

Right now, I force it to decompress 32,768 bytes. Even if it was only trying to decompress ~1,000.

So, one of two things need to happen:

One, we need to figure out a way to determine the lengths needed for each table entry.

Two, we need to modify the decoder to a state machine that spits out one byte at a time, and can be reset. This will cause more overhead per byte, but there will be less unneeded decoding, so it should make it significantly faster overall.

I'm leaning toward the latter, but I want to wait until neviksti is 100% done tweaking his decompression code before I try converting it to a state machine.

Another optimization will be making it read directly from the ROM, rather than my memory::cartrom -> SPC7110Codec::buffer transfer, so that neviksti's routines can access the data in a uint8_t array.

Also, this chip has an index into the decompressed buffer that you can specify. The only way this makes sense is if the chip slowly decodes up to that point with the ready flag clear. Whereas again, special chips are treated like memory devices with no clock, so we have to be ready immediately. I don't have a way around that, but games very seldom use the decompression index register anyway. I've only seen it in SPL4 on a dozen or so graphics.

The S-DD1, by comparison, set the DMA transfer registers to the requested output length, so there was never any redundant decompression.

Sorry, I knew it'd be unplayable in some cases, especially SPL4. But I didn't want to sit on this. It always irked me in the '90s when I'd see new special chips being added with teaser pictures, but no release for another six months. I figured something is better than nothing.

[Dullaron]

Congratulations guys.

[I.S.T.]

WOW!

Congratulations on all of your hard work, neviksti, Andreas Naive, jolly_codger and caitsith2!

Congratulations to byuu for releasing it in a form non-coders can use!

[Dullaron]

How I get those games to run? I getting Japanese message screen.

[adventure_of_link]

maybe try forcing NTSC? Or "unplugging" peripherals (sp) from controller port 2?

That said, congrats all.

[FitzRoy]

How I get those games to run? I getting Japanese message screen.

You have to reset the game after each test. SPL4 tells you in English, but Eden doesn't.

[byuu]

To start a game, you have to press 'B', and all the tests will run. Then reset, then press 'A', and the save RAM test(s) will run. Then reset, and the game will start. You only have to do each test one time. Also, you can technically skip the first test on MDH and SPL4, but not on FEoEZ.

And give all thanks to neviksti, please :)

[neviksti]

WOW!

Congratulations on all of your hard work, neviksti, Andreas Naive, jolly_codger and caitsith2!

Congratulations to byuu for releasing it in a form non-coders can use!

Thanks everyone. It's nice to see all this outpouring of support.

Just as a note, pretty much the only reason I wrote the initial code with the last few bits figured out was because I happenned to have more free time than Andreas. That's one smart fella. I wouldn't have been able to do this at all without him, and wouldn't have been able to even keep up with him without the extra free time.

My main "contribution" to the team was more along the "do runs on real hardware" line. Although I should post a picture of the cart modified to have a FIFO instead of a DataROM... it is hideous (it was made to work and nothing else ).

Damn, many thanks to neviksti and Andreas, and byuu for adding it so quickly to bsnes. I'm amazed at how quickly they managed to crack it. All that experience and knowledge got pooled, we're really lucky to still have them around.

I still don't know how everyone managed to gather there. I hadn't checked the nesdev forum in months and when I did it just happenned to be at the right time. I wonder if it was the same for everyone else.

It's funny, I pulled the bounty to fund decapping stuff instead around the time I started the "special chip" thread, totally unaware that byuu had a thread on nesdev trying to generate interest. I don't know if that was a carrot for them, but they're each entitled to a portion if they want.

Nah, I just do this for fun (my roommate keeps joking that the amount of time I spend staring at hex code on the SNES vastly outweighs the time I spend playing games on it).

It would be awesome to see the DSP1b program rom finally extracted though. However I still remain skeptical that this can be done for several hundred dollars... every place I talked to kept quoting me much much more than that. By all means don't let my skepticism stop you or anyone else... just make sure it is more than decapping+pictures that you will be getting before you fork over money, as I've already decapped and taken pictures myself.

[Deathlike2]

Nice. Finally to rid of the graphics packs once and for all.

[I.S.T.]

My main "contribution" to the team was more along the "do runs on real hardware" line. Although I should post a picture of the cart modified to have a FIFO instead of a DataROM... it is hideous (it was made to work and nothing else

I'd like to see that.

[Dullaron]

Thanks for telling me how to run these games.

[grinvader]

Code: Select all

<Nach> Muahahaha!
<Nach> I RE'd a new register not covered properly in our existing docs
<Nach> and now I get the proper length to decompress :D

(...)

Code: Select all

<Nach> Okay, I got it up to 60/60 everywhere :D

[Turambar]

Congratulations! This is great news. This board had been inactive too long a time, it's fun to follow the progress again. I think I'll compile bsnes soon, it has been somewhat long since the last time.

[Verdauga Greeneyes]

Well done everyone! I've been quietly following the research thread for about two weeks, and it's been amazing to see the constant activity there

[Snark]

Wow...what happens when many talented people work together. Nice work to all involved.

[burning shadow]

Very good job everyone! Thanks a lot! You are awesome!

[Nightcrawler]

Cheers to all involved in cracking the chip. Very nice community effort. I've followed the developments, but had nothing helpful to contribute. We've got some intelligent beans amongst our community that I am happy are among us.

[bobthebuilder]

Code: Select all

<Nach> Muahahaha!
<Nach> I RE'd a new register not covered properly in our existing docs
<Nach> and now I get the proper length to decompress :D

(...)

Code: Select all

<Nach> Okay, I got it up to 60/60 everywhere :D

Cool, has the team working on the chip been told about the new register. I ask in case one of them wants write a doc fully explaining the chip.

[neviksti]

Cool, has the team working on the chip been told about the new register. I ask in case one of them wants write a doc fully explaining the chip.

Nach doesn't even like using a real snes, and last I heard doesn't even own one. So take claims of "reverse engineering" an unknown/new hardware register lightly.

What byuu said above is correct. What Nach said should probably be better interpretted as he "found some hacks to put in the emulation to speed up these games".

[Nach]

Cool, has the team working on the chip been told about the new register. I ask in case one of them wants write a doc fully explaining the chip.

Nach doesn't even like using a real snes, and last I heard doesn't even own one. So take claims of "reverse engineering" an unknown/new hardware register lightly.

What byuu said above is correct. What Nach said should probably be better interpretted as he "found some hacks to put in the emulation to speed up these games".

I'd appreciate if you don't make stuff up about me.

[neviksti]

I'd appreciate if you don't make stuff up about me.

Your "software" vs. "real hardware" approach came up multiple times in discussion after byuu released his "state of emulation" rant. It was there that you made such comments. If they have changed, so be it. I may have misconstrued your comments about owning a snes at the time, but that was my recollection from what you said. I am not just making stuff up.

And if you have discovered a "new" register, please do let us know. I have run tests on the real hardware and it appears that what byuu said is correct "The chip has a length / decrement register, but it does no good since you can read forever from the decompression port. Many times, games do not set the register at all, so you can't use that." The only other registers you need to write to in oder to start a decompression cannot be used to determine a hardware decompression length. So the only option that seems reasonable to me is that you have found some emulation hacks specific to these games (probably DMA related)... and not found a new register. Of course, I could be wrong. New registers are exciting finds... please do share.