What is JMA?

[evilant]

Forgive my ignorance. I am curious--what is this new JMA support Nach has been adding?

It sounds like a compressed file format, but what exactly is it good for?
Can it do things that you can't do with an ordinary .zip file? Is it used by other SNES tools or something?

[kode54]

It can be used to help ROM kiddies save space storing their xxxbox hueg collections of 100% legal ROM images.

[byuu]

Oh man, awesome! I was just recently facing a dilemma about this very issue!
Do I keep such awesome games around as: DynaMate, Lotto, Pong, Tic Tac Toe, and Zophar's Domain 2 Year Anniversary Demo, or do I free up some much needed hard drive space to make room for Duke Nukem Forever?
Now, I get to have my cake, and eat it too! WOW!

Seriously though, am I correct in assuming this is something like the arcade ROMs, where you basically store one version of a ROM, and then diff files for the 40 variants of it? Cowering loves to index anything and everything.

[grinvader]

Cowering has nothing to do with jma. And he won't have anything to do about it for a while if things keep on being what they are currently.

JMA features an incredible efficiency of compression for roms. For single OR solid archives, jma is better than zip, rar, and 7z.

[evilant]

JMA features an incredible efficiency of compression for roms. For single OR solid archives, jma is better than zip, rar, and 7z.

Sounds cool! Now if we could only get some sort of universal header onto SNES roms so that future emulators did not have to heuristically detect the correct memory mapping settings.

First, there needs to be agreement on the header format. Then, tools like NSRT can start putting the headers on roms, and emulators can start reading and maybe even using the header info If its a 512-byte header it should be compatible with non-header-aware emulators, as long as they already know about copier headers.

[grinvader]

Now if we could only get some sort of universal header onto SNES roms so that future emulators did not have to heuristically detect the correct memory mapping settings.

First, there needs to be agreement on the header format. Then, tools like NSRT can start putting the headers on roms, and emulators can start reading and maybe even using the header info If its a 512-byte header it should be compatible with non-header-aware emulators, as long as they already know about copier headers.

What are you saying ? Headers are always meaningless to emulators... they are only used by copiers. Well, nsrt can use some unused header bytes to add his own controller data for use in zsnes, but apart from that, headers aren't used at all for emulators.

You should read this: this is how you know how to map your rom correctly. And it's 100% perfectly working, it's not "heuristical" or anything.

[zidanax]

I'm curious: how can JMA achieve better compression with ROMS than other compression formats?

[grinvader]

Well, ask yourself some questions first.

Why (and how) does a compressor compress a file ? Why is ZIP often faster but RAR often better at compression, and why does 7z take all my RAM and is slow as heck while compressing but decompresses in a flash ?

Once you got that, you'll have your answer.

[Nach]

Since no one brought it up.

JMA also has an unlimitly expandable field which will allow emulator authors to call for special data to be stored in, so various data can be gotten immediatly without decompressing.

Possibilities:
Number the IPS patches, so they can be applied in order.
Store internal name, so Snes9x can display it which it can't do with zip files.
Mark the ROM as being good or not.

[Nightcrawler]

Well, ask yourself some questions first.

Why (and how) does a compressor compress a file ? Why is ZIP often faster but RAR often better at compression, and why does 7z take all my RAM and is slow as heck while compressing but decompresses in a flash ?

Once you got that, you'll have your answer.

No you won't. You need to know what techniqes Nach applied to JMA. Just because you know why and how data is compressed, which I do, and even know the answers to some of your other questions, that still won't tell you what JMA does. Most major lossless compression programs are some sort of variation/combination of LZxx, Huffman, RLE, and dictionary compression. Often they are more complex implementations of these basics, but that's the basis nonetheless.

How sir does that tell me why JMA is better than than ZIP or RAR?

[Nach]

How sir does that tell me why JMA is better than than ZIP or RAR?

As he said, RAR is more intense than ZIP, and 7z is more intense than RAR and JMA is more intense than 7z.

The container format is also designed especially for stuff like SNES ROMs so we can do things simply not possible with the container format of ZIP.
See my previous post.

[Nightcrawler]

How sir does that tell me why JMA is better than than ZIP or RAR?

As he said, RAR is more intense than ZIP, and 7z is more intense than RAR and JMA is more intense than 7z.

The container format is also designed especially for stuff like SNES ROMs so we can do things simply not possible with the container format of ZIP.
See my previous post.

Exactly my point. 'More intense' isn't really an answer.

[Nach]

Exactly my point. 'More intense' isn't really an answer.

It is *the* answer.

Why does 7-Zip compress zip files much better than PKZip? Only one reason, 7-Zip is more intense in it's deflate algorithm.

[Bahamut_ZERO_Clue]

Just to back up Nach... JMA doesn't really increase the binary's size either. It's just another compression reading libary that will have an actual platform function.

[grinvader]

JMA doesn't really increase the binary's size either.

Code: Select all

-rwx------  1 grin users 430421 Nov 23 17:55 /home/grin/zsnesl*

Indeed. Maybe a 10 kB increase.

[illegal eagle]

Exactly my point. 'More intense' isn't really an answer.

It is *the* answer.

#define "intense", then.

[Nach]

If you think about it, deflate is actually a representation in a smaller format.
I can take the data found here to turn it back into the full size.

The only question is how do we generate this?

While PKZip has it's way of generating this and multiple levels of it, 7-Zip will do more passes over the data looking for similarities, it could have a larger dictionary too.

So even though both of them can be inflated into the exact same data, since one was more intense in it's ellimination of matches, it created a smaller file, yet they remain compatible between each other.

#define INTENSE MORE_MATCH_FINDING

[byuu]

More match finding... so it's using a larger LZSS window / longer match strings?

Well, please don't implement the SNES emulator header everyone's been talking about into this, as I do not intend to add support for any compressed ROM formats whatsoever to anything I write (I realize you don't care :P , but others might feel the same way). I just can't relate to wanting to heavily compress files that average 2mb in size, regardless if you have 10 or 10,000 of them. I can't see why you'd want to anyway since that idea is for a 512-byte header on the ROM itself, but just in case.

What I see happening from this is that either a) people pass it up because they don't want to recompress their files, or b) people convert all of their files to JMA, and then won't have any easy way (WinRAR won't support it) to decompress them back to SMC, so only emulators that implement JMA will be able to play these games. Or perhaps you were wanting to make a nice Win32 interface for the archiver?

I guess I'll have to wait and see...

Also, about Cowering, I wasn't implying you were working with him on it, just wondering if you were working on top of his ... interesting ... ideas for how to archive ROMs (it is not respectful to associate translations with mass piracy of ROMs by including other peoples' works in your set, without even the original readmes). Thankfully, you are not.

[Nach]

More match finding... so it's using a larger LZSS window / longer match strings?

Yes.

Well, please don't implement the SNES emulator header everyone's been talking about into this, as I do not intend to add support for any compressed ROM formats whatsoever to anything I write (I realize you don't care , but others might feel the same way). I just can't relate to wanting to heavily compress files that average 2mb in size, regardless if you have 10 or 10,000 of them. I can't see why you'd want to anyway since that idea is for a 512-byte header on the ROM itself, but just in case.

I'm implementing what I'm implementing, it's up to you what you want to support. What will be in the special JMA header for SNES ROMs will change as I discuss it with the other developers.

Or perhaps you were wanting to make a nice Win32 interface for the archiver?

I'll make the archiver and GUI people can make all the frontends they want. I don't touch Win32 platform specific nonsense unless it's a must.

Also, about Cowering, I wasn't implying you were working with him on it, just wondering if you were working on top of his ... interesting ... ideas for how to archive ROMs (it is not respectful to associate translations with mass piracy of ROMs by including other peoples' works in your set, without even the original readmes). Thankfully, you are not.

I am completely anti all of Cowering's archiving and databasing ideas. You would know that if you used some of my software. Don't even think of suggesting I'm doing anything along the lines of what he is.

[evilant]

What are you saying ? Headers are always meaningless to emulators... they are only used by copiers. Well, nsrt can use some unused header bytes to add his own controller data for use in zsnes, but apart from that, headers aren't used at all for emulators.

I think by "headers are always meaningless to emulators", you are referring to the 512-byte headers that some copiers put on ROM dumps. Of course those are of no use to emulators. Those aren't what I meant.

All the current emulators have to look at the ROM and *guess* what address mappings it uses, using *heuristics* which are usually correct but not always. There are many reasons the emulator might guess wrong, including:
- hacked ROMs or translations, that contain incorrect bytes in the ROM information
- interleaved ROMs that actually look better if read as un-interleaved data or vice versa (fools the heuristic algorithm)
- phases of the moon.

You should read this: this is how you know how to map your rom correctly. And it's 100% perfectly working, it's not "heuristical" or anything.

I've read that before. It's not 100% reliable, in the sense that a real SNES does not use that information to decide how bus addresses get mapped to the ROM data. On a real SNES the address mapping is physically wired into the design of each cart, or programmed into an address decoder chip on the cart, etc.

Maybe you could guess the address mapping of most ROMs using nothing but that info--but if you look in e.g. the snes9x source code, you'll find a big wad of code that assigns several different "scores" to the ROM which quantify how likely it is to be HiROM, LoROM, interleaved, non-interleaved, DSP, superFX, seta, or whatever. This is a *heuristic* and the result it produces is an *educated guess*.

It would be nice if future emulators didn't have to peek in the ROM to figure out how it should be mapped, what hardware it should use, etc. A better way might be:

(1) A tool such as NSRT, positively identifies your ROM by its CRC or other methods.
(2) The tool puts a standard header on the front of your ROM, that contains the correct memory mapping information. (It might also correct existing headers if the tool version is newer than the header). This is a 512-byte header in a new format, that contains enough information to construct the address mapping (in some extensible format, so that emus don't have to be updated if some new mappings are discovered in the future).
(3) The header is the same size as a copier header, so all the current emulators will assume it is a copier header and it will be safely ignored.
(4) "New-header-aware" emulators, would be able to distinguish it from a copier header and CRC-check it, so they would know they could trust its contents.

[funkyass]

unless the memmap info for a ROM comes from the cart that originated it, its suspect.

[grinvader]

There are many reasons the emulator might guess wrong, including:
- hacked ROMs or translations, that contain incorrect bytes in the ROM information
(...)
A better way might be:
(2) The tool puts a standard header on the front of your ROM, that contains the correct memory mapping information. (It might also correct existing headers if the tool version is newer than the header). This is a 512-byte header in a new format, that contains enough information to construct the address mapping (in some extensible format, so that emus don't have to be updated if some new mappings are discovered in the future).
(3) The header is the same size as a copier header, so all the current emulators will assume it is a copier header and it will be safely ignored.
(4) "New-header-aware" emulators, would be able to distinguish it from a copier header and CRC-check it, so they would know they could trust its contents.

This new header would have the same trouble: hacks
Don't tell me you can't hack a CRC to fit your need... I remember Nach could make any ROM look like SO CRC-wise (3DBDFDBF) in mere minutes.

[Nach]

I remember Nach could make any ROM look like SO CRC-wise (3DBDFDBF) in mere minutes.

Not just SO, any CRC32 I desire. And it takes ~10 seconds.

[byuu]

I'm implementing what I'm implementing, it's up to you what you want to support. What will be in the special JMA header for SNES ROMs will change as I discuss it with the other developers.

Understood.

I am completely anti all of Cowering's archiving and databasing ideas. You would know that if you used some of my software. Don't even think of suggesting I'm doing anything along the lines of what he is.

Not at all, and I commend you for your position on this.

Not just SO, any CRC32 I desire. And it takes ~10 seconds.

So use MD5 / SHA-1 :P

Anyway, yes. People -can- fake the new header format, but why would they? If they want to hack their game so that it doesn't play on emulators, then that's their business, as far as I'm concerned.
The odds of generating an identical MD5 for an existing game and for a hack / translation of a game are infinitesimal. And there is practically zero chance of generating a ROM that matches the CRC32, MD5sum, AND SHA-1 hash, all at the same time -- at least accidentally.

[Nightcrawler]

I am anti-header because I play my ROMS on a copier and actually USE the header. Perhaps if the header would still work on the copier, I would support it.

I'm not sure what my copier does with the unused portion of the header. I would like to think it ignores whatever bytes it doesn't need and you could use the unused portion of the header for whatever you want and not affect it, but I haven't actually tried that yet to see what happens.

On a side note to what Byuu was saying. I know you don't care to do anything Win32 platform specific, but I think lack of ease of use and flexibility holds back your format. Zip files are nearly transparent these days with shell integration. I can extract and create zips with just a right mouse click and selecting a path. There are more -'s rather than +'s for switching to JMA. The only thing gained is smaller file size and you lose flexibility and ease of use.

I suppose for the average user who compresses their whole collection and never touches it again, this is fine, but for other people such as myself, who are compressing and uncompressing fairly often for hacking and/or copier purposes, JMA doesn't seem like a better alternative.

And lastly since you can get a 400GB hard drive if you want these days and/or a cheap 80GB drive for just $55, your SNES ROM collection is lucky to take up more space than the formatting information overhead does on those drives!