bsnes v0.038 released

[FitzRoy]

If anyone unleashed a useless format on the world, it's you Nach with JMA. Its benefit was lost mere months after its release as GBper$ increased permanently by the same amount your format purported to save. Since then, it's offered nothing but wasted time in supporting/compiling and confusion/NSRT dependency for users. Theoretically, we could create thousands of different compression formats and programs to squeeze out the maximum space for each type of file.

This 'developer's only' club for discussing the new method is merely a front to avoid criticism, and I think I've proven that there is some to be made. A closed website database under anyone's ruleset deserves no further consideration. I have to give you free reign over PCB, though, because I don't understand it or consider it that important.

By the way, Nightcrawler, IPS and headers have as much of an agenda as UPS. If you want to disagree with how it addresses IPS' shortcomings, you'll have to attack it with more substance than that.

[tetsuo55]

the PCB and UPS/IPS discussion should be taken to their own dedicated threads

[byuu]

He released his UPS patcher before I was ready to release mine for example, along with all the side tools I made or have planned. Or before I put out new ZSNES and Snes9x releases with support for it.

I had valid reasons for that. If you were discussing it with us over at RHDN, we were on the verge of having a half-dozen patch formats released to replace IPS: NINJA1-3, XRIF, bsdiff, lately validated IPS, etc. I'm not saying any of those were bad -- I'd prefer if the best format won. I just wanted our side represented.

There was also serious talk of updating all the RHDN patches with the new format. That turned out to be a dead end, but if not -- whatever format was out there would've become the new de-facto standard. I found the other specs to be way too complex / ambiguous to implement (meaning we'd end up with the same patch creating different output between two utilities), and they all still had issues like fixed-pointer sizes, etc.

Further, the format was needed for both Der Langrisser and MOTHER 3. The situation would've been worse if we were still sitting around with no patcher today.

When did you come up with NPS? And when did we decide to work together on UPS? I found this post from RHDN:

http://www.romhacking.net/forum/index.p ... l#msg26563
October 25th, 2006; and Nightcrawler was well aware of the new format already. Nightcrawler made the point that we needed to get tools out there instead of talking about it forever.

http://www.romhacking.net/forum/index.p ... 059.0.html
Released March 31st, 2008.

It's no exaggeration that we've been talking about replacing IPS for at least two or three years and nobody had anything to show for it.

If we take UPS as an example, if byuu and I sat down and decided a month or two in advance on a particular UPS rollout date, I would've worked on UPS at that time as opposed to other projects, and you wouldn't be seeing the current situation.

So because I released my patcher 'early', you put it off for the last nine months? :/

With byuu, he's always been a fast moving target with bsnes, which is a good thing in itself.

I haven't released any kind of PCB mapping support yet. Despite wanting to since v032, I've put it off indefinitely and we're now discussing important issues like what six-letter word best describes a complex mapping strategy.

But yes, I move very fast compared to ZSNES. Unless it's for releasing something valued by their devs: NTSC filter support, WIP SPC7110 support, etc. I really mean no offense here. I sincerely appreciate how fast you guys moved with those.

I personally have a lot of projects to deal with including:

Just publicly:
bsnes
xkas (multi-target cross assembler)
UPS
hiro (multiple platform GUI toolkit wrappers)
ruby (20+ hardware API wrappers)
nall (compression, containers, strings, etc -- eg "The Wheel v2.0")
libco
bview (binary graphics viewer)
FEoEZ fan translation (min. 1,500 work hour project)
Helping others with their translations (which I enjoy doing)

... and doing all that with serious sleep issues and an OCD that severely hampers my programming speed.

But it's not a contest. We're both very busy people, surely. Should we really stop working on this stuff because one of us are too busy?

I don't consider bsnes to have any real effect on the scene as a whole. If anything, it's a nice testbed. If UPS turned out to be flawed, we could rename it VPS and go with that.

We (at least I) don't want everyone to join in with all their fear and skepticism without anything to actually judge.

There's definitely a point between Nach's insider and Nightcrawler's 'everyone matters' approach.

I will say that posting about UPS led to DMV27 pointing me to that awesome pointer encoding format. So instead of bragging about how our format handles 16 exobytes, or 2*10^53224834 bytes, we just state that it supports infinitely-sized files; and 95% of pointers will be encoded with a single byte. Longer pointers are only beneficial since they represent longer stretches of similar data, meaning the longer the pointer, the more exponentially smaller the resultant patch will be.

And I also recognize that not all developers' opinions are equal. I want emulator authors' opinions because they understand the issue the best. I want ROM hackers' opinions on how best to support their work. I want input from those who've designed online communication apps on how we should keep the database up to date. Security experts for how to future-proof the validation portion as best as possible (no, SHA-1 hashes are not secure.)

But I don't want input from people who have no expertise in the areas being discussed. We're never going to all agree, but can we all find a compromise? I'm not so sure right now, but we'll see.

I'm willing to sit around and debate this for another few months or so. But I'm not going to sit around for the next five years debating theories, either.

From what I understand, that's what he designed UPS for initially.

UPS was initially Nach's idea. It was renamed from NPS. I just added the pointer encoding format -- which was recommended by DMV27 -- and then pushed for it as best I could. And so far, I've failed miserably.

[Turambar]

Firstly I know almost nothing about rom hacking or Snes hardware. Still I love to read discussions like this about them.

I've read all this rant rather quickly, and I dimly grasp why you want to create this PCB mapping format, and what it means. However three or four people have only argued whether or not it should be done this or that way. The problem is that you are arguing over something that doesn't exist. Someone already mentioned that you should establish a specification, a plan, and then discuss about it and improve it.

I have a suggestion for byuu: write an article. Clarify and lay out your thoughts. By posting here and answering to multiple persons in the same post, your opinions become harder to understand and scattered.

[DataPath]

It's great and all to talk about how important it is to have a specification, but a specification that is not based on a reference implementation is begging for trouble.

Paper specifications are just that - so much paper.

[Nach]

Alright, well clearly your methods and process for coming up with standards that could affect everyone differ from mine. I'd see what the people it affects most have to say, hear potential alternative ideas, and take a more informed path to balance the interests of all parties affected. Working from there, you can create something that already has some support, is more likely to be adopted, and more likely to receive help on. You can't do that behind closed doors.

That is 100% correct, which is why it's being done with people who understand the topic under discussion.

The big issue right now is that TOO MUCH is being discussed in the open, and it's not healthy towards moving in the right direction.

It's not healthy toward moving in your direction. It is very healthy in moving in a direction to determine what's best for all involved. You'd rather create your own bold new initiative, then see how people react after the fact.

No, that's not what I'm saying.

So far in this forum alone there's been a dozen different opinions expressed on using XML or something else.
The data to be stored is still a major question to consider before considering how best that data should be stored. We can determine after we see the data in itself whether it belongs in XML, SQL, CSV, or a one of a bajillion custom formats.

There's also questions on how this will look in an emulator from a user perspective. These questions simply aren't ones that should be asked at this stage. Stop trying to worry about the exact set of wallpaper in the attic when we don't even have a stable basement.

People can be presented with choices that they're able to make informed decisions on. 99% of the people here DO NOT need to give their $0.02 on what mapping specific info need to be stored, nor do they want to.

Smart developing always has a straight forward method to achieving the goal, top down, bottom up, or any other valid method out there. Jumping from topic to topic to topic, completely haphazardly, with many people only jumping in to say something because they want to feel involved is not a valid paradigm for design.

Don't waste any more time on this thread and go develop what 'we plan to develop'. Part of developing a standard is talking about it, so I don't see how that is a waste of time. I thought you were interested in developing a collaborative solution balancing the interests of all parties, but I appear to have been mistaken. You've got what you want to do, so go do it. I was genuinely interested in seeing the best possible solution to all come to light. Don't get upset at me for using the word agenda when your actions disregard interests outside your circle.

I'll be interested in talking to parties when, and only when their opinions are needed, not when I have to present them with 50 maybes and if so, which layer would they like on top of any of those cases.

If anyone unleashed a useless format on the world, it's you Nach with JMA. Its benefit was lost mere months after its release as GBper$ increased permanently by the same amount your format purported to save. Since then, it's offered nothing but wasted time in supporting/compiling and confusion/NSRT dependency for users. Theoretically, we could create thousands of different compression formats and programs to squeeze out the maximum space for each type of file.

Nobody unleashed a useless format. In fact, the UPS format is my format, and I want it unleashed. I just didn't think the massive roll out was ideal for what it was. It may be, once things are ready, we'll now have to roll it out under a slightly different name just so people notice it.

As for JMA, it's not confusing anybody, as the average user doesn't even notice it. If you want it for whatever reasons you want it, it's there, you're not forced to use it. There are many people who don't care what GB/$ is and compress everything, don't bash everyone out there from living in a different ideal world than you are.

This 'developer's only' club for discussing the new method is merely a front to avoid criticism, and I think I've proven that there is some to be made.

When there is a new method that is multi-faceted, what proofs have you provided that there needs to be open discussion with non developers on topics which are solely developer related?
I'll happily hear your input on things which we can all agree someone of your knowledge and abilities can provide valuable input on. But please like Nightcrawler stop worry about details which simply are not relative yet.

A closed website database under anyone's ruleset deserves no further consideration.

If you would like to discuss privately with me and byuu what rules are okay and not okay for a DB website, I'd happily discuss it with you. In fact, if people want to seriously discuss an open DB design, but not a free for all like Wikipedia, and to seriously get a move on things, I'll setup a private forum for the interested parties who are willing to contribute (and not just throw in $0.02 for the sake of $0.02).

He released his UPS patcher before I was ready to release mine for example, along with all the side tools I made or have planned. Or before I put out new ZSNES and Snes9x releases with support for it.

I had valid reasons for that. If you were discussing it with us over at RHDN, we were on the verge of having a half-dozen patch formats released to replace IPS: NINJA1-3, XRIF, bsdiff, lately validated IPS, etc. I'm not saying any of those were bad -- I'd prefer if the best format won. I just wanted our side represented.

If that was the case, then I'm sorry. Although I didn't know this was a competition like the NSA is currently holding with SHA-3 where there's deadlines for submissions, and there can be only one winner.

When did you come up with NPS?

2002.
I gave out a spec and binaries to everyone in the SNES community back then, and had it up on the NSRT site for a while. Also made 2 or 3 threads about it, and discussed it with Gideon Zhi and another translator he introduced me to.
Although interest didn't seem very strong, and some strong criticisms were made, so I decided to hold off putting it in NSRT and ZSNES until there was more of an interest, and I had the time to deal with some of the criticism.

And when did we decide to work together on UPS?

I don't recall exactly, 1.5 years ago?

It's no exaggeration that we've been talking about replacing IPS for at least two or three years and nobody had anything to show for it.

6 years by my count, and I had what to show for it 6 years ago.

So because I released my patcher 'early', you put it off for the last nine months? :/

I wasn't actively working on it outside discussions with you, since other stuff were deemed more important at the time, and you didn't tell me we gotta have something new till literally 2 days prior to when you decided to release DL. I thought you had it on the back burner like I did for much of that time.

So because I released my patcher 'early', you put it off for the last nine months? :/

I still haven't released my patcher beyond a select few because I've been exceptionally busy recently, and still have not had time to really polish it like it deserves. I would probably be working on it with the free time I've had recently, but as you know, I'm spending it on stuff like discussing a PCB format, which seems to be more important at the moment.

If you feel we should hold off everything else and concentrate on UPS for the next month, I will, and hopefully release my UPS stuff then.

I haven't released any kind of PCB mapping support yet. Despite wanting to since v032, I've put it off indefinitely and we're now discussing important issues like what six-letter word best describes a complex mapping strategy.

Wanting to, and ready to are two entirely different things. And for something like this, I don't think bsnes in itself can be the only player in a rollout.

But yes, I move very fast compared to ZSNES. Unless it's for releasing something valued by their devs: NTSC filter support, WIP SPC7110 support, etc. I really mean no offense here. I sincerely appreciate how fast you guys moved with those.

I had nothing to do with NTSC filter support, nor do I care about it, since it annoys me, no offense intended to blargg or anyone else who likes his NTSC filter.

SPC7110 on the other hand does matter to me, it's been something that has been annoying me for 8 years. And it happened to coincide with when I had free time, so I worked on it, and it wasn't MUCH work either. But you'll notice I only put out a WIP, not a main release, because I didn't have time for it (but hope to in the near future).

Other stuff with ZSNES has taken ages. Our path system rewrite took us 3 months. Which an end user doesn't even notice unless they really look for it.

And unfortunately, due to the code base, we haven't been able to just swap in better implementations of core features just because we wanted them. Yes I know you want us to start from scratch, maybe we would be faster if we did. But notice that neither is Snes9x keeping up with your rate of changes either.

But it's not a contest. We're both very busy people, surely. Should we really stop working on this stuff because one of us are too busy?

That's not what I'm saying at all.
People are busy, they take time on things from their end. Some things may happen quickly, or slowly. However, something which is a huge change shouldn't be forced out there until everything is ready.

If just one small area which is absolutely required is not ready, because one developer is busy, that doesn't mean that the project should be dropped, but perhaps in such a circumstance the other developers should pick up the slack instead of just ignoring the other work that needs to be done.

If bsnes just rolled out with any PCB format without ZSNES, Snes9x, NSRT, a central website, and several core components in place, I think it will be nothing short of a disaster. If you feel I'm slacking with something, then please pitch in on helping me complete some work which I don't have the time to complete myself, even though I'd like to.

I will say that posting about UPS led to DMV27 pointing me to that awesome pointer encoding format. So instead of bragging about how our format handles 16 exobytes, or 2*10^53224834 bytes, we just state that it supports infinitely-sized files; and 95% of pointers will be encoded with a single byte. Longer pointers are only beneficial since they represent longer stretches of similar data, meaning the longer the pointer, the more exponentially smaller the resultant patch will be.

Which is definitely a good thing. However, I didn't hide anything related to NPS or UPS, anyone who asked me got exact specs, and I agree to DMV27's suggestions. I just wasn't running a constant referendum on the subject.

And I also recognize that not all developers' opinions are equal. I want emulator authors' opinions because they understand the issue the best. I want ROM hackers' opinions on how best to support their work. I want input from those who've designed online communication apps on how we should keep the database up to date. Security experts for how to future-proof the validation portion as best as possible (no, SHA-1 hashes are not secure.)

But I don't want input from people who have no expertise in the areas being discussed.

Yes, and I think some people miss that. Or even more so, they miss that we're simply not ready to discuss a certain aspect right now, and instead complain that everything is behind closed doors, and we need to know which box to ship our lovely package in before we even know the dimensions of it. I'd like to think our developing process isn't 100% ran by the marketing division like it is in some badly managed companies.

We're never going to all agree, but can we all find a compromise? I'm not so sure right now, but we'll see.

I think we can, but we're not going to get there if everyone is constantly throwing in their $0.02 on whether we should charge for this format via VISA or MasterCard or PayPal.

I'm willing to sit around and debate this for another few months or so.

I certainly hope not. I don't think any single area should be up for debate for more than 2-3 weeks.

But I'm not going to sit around for the next five years debating theories, either.

Neither am I. But when everything is decided on, and it's time to write all the libraries and code and update all the related programs, I don't want one emulator to rush a release for the sake of rushing it. If one program falls behind, the other capable developers MUST be able to lend a hand. And if it takes time, then the proper time must be alloted, and not just rush out some buggy junk just to be out in time for the holidays.

UPS was initially Nach's idea. It was renamed from NPS. I just added the pointer encoding format -- which was recommended by DMV27 -- and then pushed for it as best I could. And so far, I've failed miserably.

Yeah, and I keep on getting people bothering me why don't I like UPS, or how come I don't support byuu's format, or agree with the UPS spec. Even when I tell people IT IS MY SPEC, stop bothering me, I just get back "Nah, you're just saying that so I leave you alone, we both know it's not your spec and you hate it and therefor don't want to implement it anywhere".

[Deathlike2]

Alright, well clearly your methods and process for coming up with standards that could affect everyone differ from mine. I'd see what the people it affects most have to say, hear potential alternative ideas, and take a more informed path to balance the interests of all parties affected. Working from there, you can create something that already has some support, is more likely to be adopted, and more likely to receive help on. You can't do that behind closed doors.

Normally I agree with that, but for myself seeing a smaller glimpse into what Nach has invested in.. (and sure that is bias you can question), the reality is that he has the right idea on getting something done. Now whether time is available for that is always another issue altogether.

Kinda like cooking... when you get so much feedback into what users think instead of something a programmer is supposed to deal with (with reasonable consideration but also no outside interference), it starts to become a mess... like the simple "too many chefs messing with the stew argument". Sometimes it doesn't always sound desirable, and I'm sure it always like a proprietary deal, but sometimes it's for the betterment of things. You can't just say yes to everyone.

The big issue right now is that TOO MUCH is being discussed in the open, and it's not healthy towards moving in the right direction.

It's not healthy toward moving in your direction. It is very healthy in moving in a direction to determine what's best for all involved. You'd rather create your own bold new initiative, then see how people react after the fact.

Don't waste any more time on this thread and go develop what 'we plan to develop'. Part of developing a standard is talking about it, so I don't see how that is a waste of time. I thought you were interested in developing a collaborative solution balancing the interests of all parties, but I appear to have been mistaken. You've got what you want to do, so go do it. I was genuinely interested in seeing the best possible solution to all come to light. Don't get upset at me for using the word agenda when your actions disregard interests outside your circle.

He's certainly interested, but I think isolating the noise (other people who think their two cents is worth it, but sometimes isn't worth exploring when properly thought through) and isolating the important stuff (not the silly details like how things should look pretty, unless we're examining a GUI), then you will find something reasonable that will come out from all of this.


I'm just seeing a lot of opinions and issues and agendas.. not a lot of "hm, why is my idea not so good?" type of self-analysis. This kind of complaining is like the elections... sometimes people even forget they voted an idiot into office.

The unfortunate launch of UPS is more or less a disaster. I'm sure anyone can point fingers and lay blame, but the reality is that when you launch something that's not totally ready, you're doomed to fail. It's that simple. It's just as real is not knowing the name of byuu's UPS patching app. I joked about it in the RHDN forum, but the seriousness of why noone has adopted it is because it wasn't ready. I'm sure if byuu waited for across the board implementation as Nach suggested with JMA, these issues would be lessened (although, I don't take JMA very serious until a seperate GUI based app or something integrated to NSRT would be available).


Sometimes, the obvious problem is the obvious issue. I hope people aren't making dumb comments when the answer is in front of their faces. The continuing whining it seems from all non-involved parties is not going to magically make this issue better or improve anything. Let things work themselves out first.. it's not as if emulator updates come on user demand.

[byuu]

Okay, well the PCB discussion really isn't going all too great at this point here. Sorry for bringing it up and causing trouble, I just like to keep things as transparent as possible from the beginning.

Nach is right, we need to focus on the foundation: what do we need to map all cartridges using this format? We don't have a single proposition, so discussing databases, what certain emulators are going to do with the new spec, and online distribution is kind of silly. None of that is finalized, we aren't going to cut anyone out of the loop on that.

Let's stick to the underlying format. If you have technical expertise on mapping out SNES boards and how the bus and coprocessors interact with the base unit, I welcome your feedback. All such people (FitzRoy, neviksti, Nightcrawler et al) should know where to discuss this, if not please check with me or Nach. If you don't understand that, then I'm sorry to be rude -- but feedback probably won't be too helpful for this stage alone. Once we get that settled, we'll get people and discuss the next stage, whatever that may be.

I have a suggestion for byuu: write an article.

That sounds like a good idea, but I'm not qualified to discuss the header detection methods we use now. The first part would have to be written by eg Nach.

The data to be stored is still a major question to consider before considering how best that data should be stored.

Very true, I'm guilty of jumping all over as well. Point taken, we'll stick to the C interface first.

If that was the case, then I'm sorry. Although I didn't know this was a competition like the NSA is currently holding with SHA-3 where there's deadlines for submissions, and there can be only one winner.

That seems a bit harsh, too. I'm sure Nightcrawler will contest this, so let me just clarify my statement.

Back when NINJA1 came out in 2006ish, most people were fed up with IPS and wanted something better. There was definitely talk of converting the existing patches over to it. But relations fell apart between the author and key members of RHDN before a consensus could be reached either way.

Now, I personally don't believe it would've ever happened, even with NINJA1 back then. If there's one thing about ROM hackers, is that no two are alike. The biggest discussion I've ever seen on that site was a minor deletion policy change. To even consider updating an archive with a newer patching format that would work in new emulators ... I'm sure it would cause World War ROMH over there.

But what I did see while discussing UPS with DMV et al was a half-dozen people who wanted to add the kitchen sink to IPS: cut, copy, move, weird bit shifting patterns, really high-end compression methods ... all kinds of things, when all we really needed was a simple update to IPS to validate that the target was good, and community-enforced rules on how to make a patch (eg header or no header.) And we needed to expand from the 16MB limitation.

Sure, a delta patcher like xdelta or bsdiff will be essential for DVD image patching and such in the future ... but that's a completely different ball-game. No NES emu author is going to jump through the hoops to write such a patcher for ~64kb games. It's a matter of the right tool for the right job.

I feared that if we continued to sit by, all of these formats would come out and nobody would ever agree on anything. I didn't want to kill them off, certainly. Again, I think they'd be great if targeted to ISO patching and other tasks like that.

It was also convenient timing for DL's release, no disputing that. But that wasn't my sole motivator. In fact, for the first release, we didn't even use UPS. We had a custom EXE patcher using a proprietary format.

2002.

And that spec was 99% the same as it is today. And had I waited, would a single SNES emu support anything but IPS today? I'm not entirely sure.

I wasn't actively working on it outside discussions with you, since other stuff were deemed more important at the time, and you didn't tell me we gotta have something new till literally 2 days prior to when you decided to release DL. I thought you had it on the back burner like I did for much of that time.

You're right that it was rushed. We didn't know SNESGT could run DL, and I wasn't about to implement IPS. UPS was the only way to offer soft-patching with an emulator.

But the file specification itself ... that was golden. Aside from a few minor things like swapping the order of the checksums (which is purely a cosmetic issue), we had the spec finalized and ready for several years, at least.

The only thing bad about UPS is that the current patching tool isn't fantastic. And by releasing it, though it isn't super popular ... a lot of people have heard about it.

I don't think that's a bad thing at all. I don't see how we're any worse off for having released the patcher in April than if we were to have waited until today.

On the plus side, 100,000+ people have downloaded UPS patches now between DL and M3 (mostly the latter.) It went very, very well; if not perfect. We also have four of the best emulators for each system supporting it out of the box now.

Anyway, I do apologize that I didn't discuss this more with you. I was honestly under the impression we were ready to go, and if anything, I thought you'd be happy that I finally got your spec out there for people to use, with real-world patches, emulator support and a patcher.

Even when we discussed it in April, I still had that impression that we both felt the spec itself was golden. We had discussed it off and on at least a dozen times, and I had open threads on RHDN for at least 18 months about it. I figured we were ready.

But again, I still don't believe my actions have caused more harm than waiting: many more people know of UPS than of NPS, and there's no such thing as bad publicity. But if you really disagree, we can revert to NPS and you can roll that out at your discretion. I don't know what we would change spec-wise.

If you feel we should hold off everything else and concentrate on UPS for the next month, I will, and hopefully release my UPS stuff then.

No, absolutely not. The PCB spec affects all games, UPS affects three dozen SNES translations or so.

If anything, it's my fault. I should be the one to go back and write a really nice patcher. I'll put that on my to-do list, along with coming up with a better name for it ("byuu PS"? :P)

And for something like this, I don't think bsnes in itself can be the only player in a rollout.

No, but by not being mainstream, it can be used as a testbed. We don't have to claim anything in test releases is final, and we can give people a chance to beta test the ideas before a 100% rollout. I think that's a good thing.

If there's one thing I'm blessed with, it's absolutely awesome users with high technical aptitude. I'm sure we wouldn't have any problem users.

SPC7110 on the other hand does matter to me, it's been something that has been annoying me for 8 years.

It mattered to me a lot too, which is why we both got it out in record time of ~24-48 hours ;)

My point was more that things can move fast in ZSNES, they just have to matter enough and not be too difficult to pull off. Given, I don't know how hard it'd be to modify the IPS code in ZSNES.

And unfortunately, due to the code base, we haven't been able to just swap in better implementations of core features just because we wanted them. Yes I know you want us to start from scratch, maybe we would be faster if we did. But notice that neither is Snes9x keeping up with your rate of changes either.

I did want you to start over. And I was especially irked to hear the new cores were being written in assembler again.

As my own codebase ages, I'm seeing more how painful it'd be to truly start over: having to track down why D3D is blurring your 1:1 scaler again, re-tracing and fixing all 300 of those crazy Japanese golf games again (you'll get stuck with that on a new core anyway, but hopefully my memory can help), and all that without any of the original devs ...

Short of pagefault really pulling a miracle, I'm not sure ZSNES (or even bsnes) would survive a total rewrite at this point.

I'm still not happy with any emulator though ...
SNESGT has the language barrier and is closed source
SS is closed source
bsnes is crazy slow, lacks features and is half-closed source
SNEeSe is on life support
Snes9X is a mess, on life support, and has no real lead without anomie
ZSNES is still mostly assembler, not portable and very hard to work with

ZSNES definitely has the best balance, as evidenced by its popularity. But I still feel there's no definitive Nestopia, gambatte or Fusion for it. They all have serious downsides.

And none of us really have time for real hardware research or for working on a new, penultimate emulator project.

Yeah, and I keep on getting people bothering me why don't I like UPS, or how come I don't support byuu's format, or agree with the UPS spec.

Wow o.O

Didn't even realize I was popular enough for people to bug you. I've tried to make it as clear as I could that this is your spec from the beginning.

[Deathlike2]

Yeah, and I keep on getting people bothering me why don't I like UPS, or how come I don't support byuu's format, or agree with the UPS spec.

Wow o.O

Didn't even realize I was popular enough for people to bug you. I've tried to make it as clear as I could that this is your spec from the beginning.

Sometimes.. I think you don't realize what you've done until someone beats you over the head and tells you. Really, it's not a mean comment, but it seems like the reality.

[blargg]

Just skimming here, but any new patch format should allow avoidance of any of the original file's data appearing in the patch, so that patches of copyrighted works can avoid being legally considered derivitive works. As an example, the patch format could XOR the patch data with that from the original file or encrypt it based on the hash of the original file. This would ensure legality of patches that move or slightly alter data from the original file, where they'd otherwise have copies of this data in them. Maybe the technical approach wouldn't be sufficient (I'm not a lawyer), but it would be nice for patches to have no hint of copyright infringement issues.

[byuu]

Sometimes.. I think you don't realize what you've done until someone beats you over the head and tells you. Really, it's not a mean comment, but it seems like the reality.

I'm definitely out of my league in the company of most of you. But eh, it's not all bad. I hope you'd at least agree that despite all my mistakes, I've still been more overall beneficial than harmful.

I'm a stubborn ass at times, but I learn. If ever so slowly.

Nach ... I just can't understand him 90% of the time. I hear that's a common issue when two people are that far apart from each other intellectually. Not that it's his fault in any way, but it would certainly help if he made his intentions more clear to me from the start.

He never said that he really cared about NPS and that me releasing it would in some way damage UPS's chances of success. I still don't even see how that's the case, in case anyone would like to explain :/

I was going to make my own format, saw he was working on one, and wanted to help out rather than make my own.

As an example, the patch format could XOR the patch data with that from the original file or encrypt it based on the hash of the original file. This would ensure legality of patches that move or slightly alter data from the original file, where they'd otherwise have copies of this data in them.

I thought about this a lot ... it's really an untested legal gray area. But I don't think it would hold up in court. Most of these works are illegal per the Berne Convention the moment they include translated bits of script. The graphics hacks probably so as well, unless they completely refrain from having parts of the original work.

If a simple obfuscation XOR were enough to make something legal, then one could XOR an MP3 with a Word document, and distribute the patch and Word doc, right? It seems to be too convenient of a legal loophole. Then again, US judges at least aren't known for their technical competence. Just need a really expensive lawyer (good luck.)

The reason I chose not to XOR the expanded data was because a) it made even the compressed patches significantly larger (eg pad your 2MB game to 4MB, but only use 200kb of that); and b) how do you contain a reversible patch where one direction is an empty file? This would be needed for a multi-patch format that is planned in the future. The goal of the spec is to allow conversion in either direction, without ever requiring one have the original file in the first place.

I think if you really wanted to be legal, you should use a one-way patch that in no way contains even XORed data from the original file.

[Deathlike2]

Nach ... I just can't understand him 90% of the time. I hear that's a common issue when two people are that far apart from each other intellectually. Not that it's his fault in any way, but it would certainly help if he made his intentions more clear to me from the start.

I don't understand him all the time either, but I'm willing to listen.

He never said that he really cared about NPS and that me releasing it would in some way damage UPS's chances of success. I still don't even see how that's the case, in case anyone would like to explain :/

Just apply the "tree falling in a forest and noone hears it" analogy. It could be argued that UPS suffers from it. It's not hopeless. The primary issue is that it's missing is a reference implementation and the tools released to test other implementations... you can't just release a format unless you make sure the obvious stuff is covered. There is one thing you can be sure of from Nach.. when he says he's ready, he's ready. Jumping the gun on UPS currently hinders any sort of across-the-board adoption in the near future until the obvious issue is addressed (it's all in italic just in case it wasn't already obvious). It has nothing to do with the "paper" that people claim the spec is written on.

In any case, you can't get rid of IPS... you can only progressively move forward with better tools equipped to transition. You don't improve adoption without the tools necessary in the first place. You can't just "force" adoption. That's the clear and obvious issue you need to realize. Heck, I'm always tempted to just make a IPS patch for Der Lang (for myself) just because I have one option only. I like multiple options... as long as I know what I'm doing. Headers and all this silly talk is irrelevant if you don't educate people to use the tools like NSRT to make sure you can't screw up. UPS will not fix the SNES header issue, not matter how hard you try. To fullproof something is to make the tools possible... even Nightcrawler's solution.. which is nice for the end user, is not something one should spend extra time on in general. There are obvious solutions and the means to do it. That's why the tools exist. The UPS patcher is one tool in a set of tools that needs to be deployed to get UPS deployed properly. It should be plenty obvious here once it's understood.

[Thristian]

If a simple obfuscation XOR were enough to make something legal, then one could XOR an MP3 with a Word document, and distribute the patch and Word doc, right? It seems to be too convenient of a legal loophole. Then again, US judges at least aren't known for their technical competence. Just need a really expensive lawyer (good luck.)

I read a really interesting article recently that tried to explain the technical vs. legal definitions of 'derivative works'. I found it quite well done - depressing, from a technical point of view, but well done.

[Panzer88]

I think it's clear that some rom hackers would make a fuss if we tried to update the RHDN archive (although I don't know why) but I will say this byuu. Since that is probably not going to happen the next fastest and really the best is to just ask prominent hacking and translation groups (e.g. Aeon Genesis, Transcorp, ) if they would be willing to post a UPS version of their projects, both old and new. Sure they can say no, but they might just do it, and that way the only ones be supported are by the authors who WANT to support it. Nobody would have to take down IPS patches, and if you got enough support from big groups (note: just because they want to because it's useful, people are too defensive or paranoid and think this is some kind of hostile takeover) then it might just become the new standard.

that's just my unqualified two cents.

I know this is a "no duh" situation, but it actually needs to be done. for example, Nightcrawler, what would you think of supporting this format with transcorp releases?

[Nach]

If that was the case, then I'm sorry. Although I didn't know this was a competition like the NSA is currently holding with SHA-3 where there's deadlines for submissions, and there can be only one winner.

That seems a bit harsh, too. I'm sure Nightcrawler will contest this, so let me just clarify my statement.

Back when NINJA1 came out in 2006ish, most people were fed up with IPS and wanted something better. There was definitely talk of converting the existing patches over to it. But relations fell apart between the author and key members of RHDN before a consensus could be reached either way.

Now, I personally don't believe it would've ever happened, even with NINJA1 back then. If there's one thing about ROM hackers, is that no two are alike. The biggest discussion I've ever seen on that site was a minor deletion policy change. To even consider updating an archive with a newer patching format that would work in new emulators ... I'm sure it would cause World War ROMH over there.

But what I did see while discussing UPS with DMV et al was a half-dozen people who wanted to add the kitchen sink to IPS: cut, copy, move, weird bit shifting patterns, really high-end compression methods ... all kinds of things, when all we really needed was a simple update to IPS to validate that the target was good, and community-enforced rules on how to make a patch (eg header or no header.) And we needed to expand from the 16MB limitation.

Sure, a delta patcher like xdelta or bsdiff will be essential for DVD image patching and such in the future ... but that's a completely different ball-game. No NES emu author is going to jump through the hoops to write such a patcher for ~64kb games. It's a matter of the right tool for the right job.

I feared that if we continued to sit by, all of these formats would come out and nobody would ever agree on anything. I didn't want to kill them off, certainly. Again, I think they'd be great if targeted to ISO patching and other tasks like that.

It was also convenient timing for DL's release, no disputing that. But that wasn't my sole motivator. In fact, for the first release, we didn't even use UPS. We had a custom EXE patcher using a proprietary format.

See, this just shocks me.
Back in 2002 when I tried discussing with several people that the IPS format is severely flawed, and I offered a similar simple format which can be used to upgrade to and solve all those issues, people basically laughed in my face. What reason was there to change? Everything works great with IPS. The occasional issue can be covered with a readme, and the user always worked with a backup copy.

Nobody seemed interested in hearing about a new format then. Now they were demanding it? Guess I was just too early.

I've really been out of touch with translators aside from the occasional request for help since people told me they didn't need a new format. I had no idea they were finally ready.

2002.

And that spec was 99% the same as it is today. And had I waited, would a single SNES emu support anything but IPS today? I'm not entirely sure.

Well, if you want to get technical, I've always had my own private tree of ZSNES with NPS support, and now UPS support.
I haven't released it though, due to a feature addition with IPS.

For the ZSNES 1.5 branch, stronger requests were made for multiple IPS patches one after another, so I went ahead and added, I literally only spent a few minutes on it.

Now, if I were to merge UPS into the mainline ZSNES branch, I wouldn't do it without support for multiple UPS patches one after another. The problem here though is, this is a lot more complicated from an emulator's perspective, because UPS has hashes that need to match, and UPS patches that cover two different completely parts of a ROM image should be able to merged together. The issue is finishing my code to handle all the various cases to make sure this works flawlessly.

You have to understand two things about me, I don't like doing a partial job, if I'm doing something, it should be expected to work even beyond the extent a user expects it to work. It should make users see a feature and think to themselves, "oh how cool, but such a thing would really be nice in conjunction with X", and when they go to request X, they're flabbergasted that we already thought of X, and it's implemented.

The other thing is that sometimes I'm too smart to get work done. For example, I'm really good at Chess (or used to be when I played daily competitively, 0 losses for a 3 year running stretch!), I see a lot of the possibilities on the board, and can really all out analyze a bunch of situations. I get the same way about software. I see a lot of situations that most of the time don't cross other people's minds. I don't like doing something unless I can handle ALL of those situations, and it's way more situations than others foresee. Which means quite often, to add a new feature, I'm working for a long time on it, even though, other people end up first to market. But in the end, I offer a much more solid product.

I wasn't actively working on it outside discussions with you, since other stuff were deemed more important at the time, and you didn't tell me we gotta have something new till literally 2 days prior to when you decided to release DL. I thought you had it on the back burner like I did for much of that time.

You're right that it was rushed. We didn't know SNESGT could run DL, and I wasn't about to implement IPS. UPS was the only way to offer soft-patching with an emulator.

But the file specification itself ... that was golden. Aside from a few minor things like swapping the order of the checksums (which is purely a cosmetic issue), we had the spec finalized and ready for several years, at least.

The only thing bad about UPS is that the current patching tool isn't fantastic. And by releasing it, though it isn't super popular ... a lot of people have heard about it.

I don't think that's a bad thing at all. I don't see how we're any worse off for having released the patcher in April than if we were to have waited until today.

On the plus side, 100,000+ people have downloaded UPS patches now between DL and M3 (mostly the latter.) It went very, very well; if not perfect. We also have four of the best emulators for each system supporting it out of the box now.

Anyway, I do apologize that I didn't discuss this more with you. I was honestly under the impression we were ready to go, and if anything, I thought you'd be happy that I finally got your spec out there for people to use, with real-world patches, emulator support and a patcher.

Even when we discussed it in April, I still had that impression that we both felt the spec itself was golden. We had discussed it off and on at least a dozen times, and I had open threads on RHDN for at least 18 months about it. I figured we were ready.

You have to understand my idea of ready.

Ready would mean ZSNES and Snes9x support to the extent I mentioned above. Ideally bsnes too. It'd mean command line utilities for easy scripting of an installer, which is straightforward. A GUI that would need a very futuristic idiot to screw up anything. A converter that can take ROM+IPS and spit out UPS, asking with or without header. A more stupid version of a converter that shows the user several captured frames from running with and without a header in an actual emulator and asks them which one seems better, for when they simply don't know if there's a header or not. This also has to be done with interleaving. A simple lightweight library in C and in C++ to make it as easy as possible for other developers to drop into their emulators. The developers also need a series of test cases to test to make sure they handle all of them properly. Also, I made a program to test a C/C++ library or a command line creator/patcher against in a few thousand cases, which I have given you, and I still don't know if your current implementation passes all of them. I know your initial release failed way too many.

When all that's in place, I'll say we're ready. And if I have to do it all singlehandedly, it's easy a month or two of solid work on those things, without time for any other projects I might want to work on at the time.

But again, I still don't believe my actions have caused more harm than waiting: many more people know of UPS than of NPS, and there's no such thing as bad publicity. But if you really disagree, we can revert to NPS and you can roll that out at your discretion. I don't know what we would change spec-wise.

Well, in the long term, I don't think it will hurt either, and all advertisement is generally a good thing.

But when we finally are *ready*, many people will just see UPS and think, "Oh that's the thing they've been going on and on about, last I heard, nothing supported it nicely, so I'm not going to go look at it again now, waste of time".

I don't think the spec needs to change though.

If you feel we should hold off everything else and concentrate on UPS for the next month, I will, and hopefully release my UPS stuff then.

No, absolutely not. The PCB spec affects all games, UPS affects three dozen SNES translations or so.

If anything, it's my fault. I should be the one to go back and write a really nice patcher. I'll put that on my to-do list, along with coming up with a better name for it ("byuu PS"? )

Hardly it's your fault. I'm the crazy long term visionary who is parallelized for anything less than my ideal long term goal.

Also at that time back then, you can't even imagine how crazy busy I was, to really devote time like this needs devotion to.

A brand new start up company tried to release a service for a very specific field online. After their initial attempts to release, they realized how swamped and overwhelmed they were with the competition and what it would mean to really stand out. To advance, they would need integration of several vastly different programs and services, and bring them altogether in an intelligent, mostly automated, and easy to use way. They literally had to have several secretaries working around the clock to pass data back and forth between several services, since none of them even offered any API to bind them together. They were for a while, but they really couldn't keep up.

The founder of the company bemoaned of his issues to his son-in-law telling him how he had a paddle without a creek to get anywhere with. His son in law actually knew me personally, and has seen some crazy stuff I pulled with NSRT, which seemed similar to about half of what they needed done.

So they called me up to work on it, with the basic goal of everything needs to be done yesterday. I of course accepted, after being offered a huge bundle of cash if I could save them.

I then had to study APIs that were available to some of the services, although some documented very badly and had to really quantify how they worked, and what they could and could not do. Then I had to cover all the things without an API, and create an API which would basically be bindings to a robot to emulate a user interacting with a frontend to a service. They needed an automated program which could collect a bunch of different files from all over, with no initial markers that they were in any way related, and bundle them together, finding the correlation between them as new versions come out - automatically. They needed some Cisco stuff hacked so they could interface directly. They needed to have series of files named in such a way that the names made sense to end users, although they'd be stored in a directory which was unguessable by users in advance, but completely deterministic from the software internally, so it can all work together. They needed passwords stored in such a way that the actual passwords would be distributed across multiple servers, so the actual passwords could be gotten by our software (since these passwords were in turn used by a robot to other services automating stuff for each user when they weren't around, and had to log in as them), but secure enough, that if any one of our servers were hacked into, it would be impossible to have any information about the user's password, they'd need to hack them all to get anywhere.

I was swamped. I had to work around the clock for days. I even had members of the start up calling me at 4 o'clock in the morning to add urgent new features, or debug and fix stuff done by other developers of their's who just couldn't figure out what was going wrong with their code.

That's what I was doing in April, and in March, and in May. If you think what I want to do with UPS is tricky, it's not even a fraction of what this other project entailed.

On the other hand, two higher ups of this start up don't like how it now is obvious their service can do so much more, but the rest of the company is afraid to try anything so bold after what was initially done. I've been asked by these two to join a start up they want to do, even more advanced, and this time get a 25-30% share in the company too. If/When they get some venture capitol in place (not easy with the current economy unfortunately) to do what they want, I'll be joining them, and probably be busy like mad for several months then too. Probably even more than I was back then.

And for something like this, I don't think bsnes in itself can be the only player in a rollout.

No, but by not being mainstream, it can be used as a testbed. We don't have to claim anything in test releases is final, and we can give people a chance to beta test the ideas before a 100% rollout. I think that's a good thing.

A testbed to what extent though? Can it be a testbed for integration with a central website when there is no central website?

SPC7110 on the other hand does matter to me, it's been something that has been annoying me for 8 years.

It mattered to me a lot too, which is why we both got it out in record time of ~24-48 hours

I think it took me closer to 72 hours till my initial WIP. And another 2 days till I actually felt the code was ready. Notice how much I put into that code. I've actually ran all the SPC7110 games on 300 Mhz machines at full speed. Notice the amount of side features I offer with it. I really hate skimping on what can and ideally be done.

My point was more that things can move fast in ZSNES, they just have to matter enough and not be too difficult to pull off. Given, I don't know how hard it'd be to modify the IPS code in ZSNES.

Piece of cake really. Just to plainly add on a simple patch format shouldn't even take a bad programmer more than two hours of work. But to add on a patch format to the extent I do is a completely different ballgame.

Wow o.O

Didn't even realize I was popular enough for people to bug you. I've tried to make it as clear as I could that this is your spec from the beginning.

Heh, notice above how FitzRoy got all defensive for you, because he thinks I'm bashing your format.

[Nach]

Just skimming here, but any new patch format should allow avoidance of any of the original file's data appearing in the patch, so that patches of copyrighted works can avoid being legally considered derivitive works. As an example, the patch format could XOR the patch data with that from the original file or encrypt it based on the hash of the original file. This would ensure legality of patches that move or slightly alter data from the original file, where they'd otherwise have copies of this data in them. Maybe the technical approach wouldn't be sufficient (I'm not a lawyer), but it would be nice for patches to have no hint of copyright infringement issues.

UPS covers for all this, except in the circumstance where the new file is smaller than the original file.

I don't even see how we'd be able to make it that we'd be able to avoid it on a shrink, to the extent that it'd be impossible to generate that chunk of data solely from the patch file.

Unless we somehow came up with a method that constantly xor'd that extra chunk of data, with the beginning chunk of non changed data. Except that's an issue when the data is 0.

[FitzRoy]

When there is a new method that is multi-faceted, what proofs have you provided that there needs to be open discussion with non developers on topics which are solely developer related?

The database is not solely developer related. Your idea basically turned it into one.

A closed website database under anyone's ruleset deserves no further consideration.

If you would like to discuss privately with me and byuu what rules are okay and not okay for a DB website, I'd happily discuss it with you. In fact, if people want to seriously discuss an open DB design, but not a free for all like Wikipedia, and to seriously get a move on things, I'll setup a private forum for the interested parties who are willing to contribute (and not just throw in $0.02 for the sake of $0.02).

No, you still don't understand. We fundamentally disagree on the "rules." That includes documented info, naming, inclusion, verification requirements, and what of all that should be public knowledge. And if you haven't realized this by now, we conflict on pretty much all of them. This is just one of the reasons a central database will never do, nevermind the technical dependencies I cited. The behavior I'm suggesting is immune to all of that.

It's simple. The emulator's 2nd detection method would be to check for a spreadsheet file in the executable directory. It scans within a row to find a checksum and pcb serial. It doesn't expect a certain column order or care if other columns are present. That allows any emulator author to bundle any dumper's spreadsheet they want with releases. It furthermore allows any user to replace it with one of their choice should they dislike the author's selection. It's also not something that requires emulator consensus, bsnes could do this ASAP to its advantage, and I think that it will. Whether you want to do something similar and achieve consensus behavior would simply be a matter of agreeing on what checksums to include.

And if this wasn't clear already, this has no negative effect on community gathering of information. I welcome other databases such as your own if you made one to include my verifications. I'm sure you welcome me to include yours, it's just that my standards would make most of yours inadmissible.

If bsnes just rolled out with any PCB format without ZSNES, Snes9x, NSRT, a central website, and several core components in place, I think it will be nothing short of a disaster.

Though I agree that consensus is far more important for this method, I think you're fearmongering a little that a proprietary format should be avoided like the plague. Unlike the patching formats, it would burden practically no one but yourselves to scratch it in favor of a superior one should one arise. Rom distribution would have to change to include these files with the roms in order for that external worry to exist, and that's not going to happen.

Heh, notice above how FitzRoy got all defensive for you, because he thinks I'm bashing your format.

I was talking to Nightcrawler.

[Nach]

When there is a new method that is multi-faceted, what proofs have you provided that there needs to be open discussion with non developers on topics which are solely developer related?

The database is not solely developer related. Your idea basically turned it into one.

A closed website database under anyone's ruleset deserves no further consideration.

If you would like to discuss privately with me and byuu what rules are okay and not okay for a DB website, I'd happily discuss it with you. In fact, if people want to seriously discuss an open DB design, but not a free for all like Wikipedia, and to seriously get a move on things, I'll setup a private forum for the interested parties who are willing to contribute (and not just throw in $0.02 for the sake of $0.02).

No, you still don't understand. We fundamentally disagree on the "rules." That includes documented info, naming, inclusion, verification requirements, and what of all that should be public knowledge. And if you haven't realized this by now, we conflict on pretty much all of them. This is just one of the reasons a central database will never work, nevermind the technical dependencies I cited. The behavior I'm suggesting is immune to all of that.

It's simple. The emulator's 2nd detection method would be to check for a spreadsheet file in the executable directory. It scans within a row to find a checksum and pcb serial. It doesn't expect a certain column order or care if other columns are present. That allows any emulator author to bundle any dumper's spreadsheet they want with releases. It furthermore allows any user to replace it with one of their choice should they dislike the author's selection.

And if this wasn't clear already, this has no negative effect on community gathering of information. I welcome other databases such as your own if you made one to include my verifications. I'm sure you welcome me to include yours, it's just that my standards would make most of yours inadmissible.

If bsnes just rolled out with any PCB format without ZSNES, Snes9x, NSRT, a central website, and several core components in place, I think it will be nothing short of a disaster.

Though I agree that consensus is far more important for this method, I think you're fearmongering a little that a proprietary format should be avoided like the plague. Unlike the patching formats, it would burden practically no one but yourselves to scratch it in favor of a superior one should one arise. Rom distribution would have to change to include these files with the roms in order for that external worry to exist, and that's not going to happen.

I don't see how having an online community database which opts to store as much info as possible from as many sources as people are willing to contribute (with a clear privacy policy of course, as needed) is a bad thing.

Furthermore, I'd have it that one would be able to check a couple of boxes as to which data they want, with whatever filtering shenanigans, and have it spit out a file in several different formats for a user to do with as they see fit.

So what if my preference of what it spits out is different than yours? I still don't see why you're up in arms about that.

Heh, notice above how FitzRoy got all defensive for you, because he thinks I'm bashing your format.

I was talking to Nightcrawler.

If you say so. "If anyone unleashed a useless format on the world, it's you Nach" seems to be directed clearly at me, if not, we seem to have major communication issues.

[byuu]

I know this is a "no duh" situation, but it actually needs to be done. for example, Nightcrawler, what would you think of supporting this format with transcorp releases?

There's a slight cost to supporting it now. It'll confuse a small number of people who don't know which emulator supports which auto-patch method.

I've really been out of touch with translators aside from the occasional request for help since people told me they didn't need a new format. I had no idea they were finally ready.

Who knows. They seemed receptive to NINJA, too. Maybe you have to have inside credibility first or something :P

The problem here though is, this is a lot more complicated from an emulator's perspective, because UPS has hashes that need to match, and UPS patches that cover two different completely parts of a ROM image should be able to merged together.

Both IPS and UPS will fail in a multi-patch scenario when two patches or more modify the same byte. The final result will vary based upon the order of patching. UPS gets it a bit worse by having a XOR result, whereas IPS has the final real result, but it's still very bad.

As for patching in order, just read in all the patch checksums, locate the next one in the list, apply it, and repeat. Not easy, but not several-months difficult, either -- hopefully. Or we could treat it like IPS, accepting any kind of arbitrary combination, in which case we'd ignore the checksum field; and patching order wouldn't matter.

My biggest concern at the moment is that we need a lib that can do chunks at a time. Patching a very large file may freeze up a GUI in some cases otherwise.

But when we finally are *ready*, many people will just see UPS and think, "Oh that's the thing they've been going on and on about, last I heard, nothing supported it nicely, so I'm not going to go look at it again now, waste of time".

If they weren't interested when three other emus added it, I don't think adding it to ZSNES is going to be that much more exciting. But we'll have to see ...

I think more people would give it a shot just as soon as both ZSNES and Snes9X supported it out of the box for a while.

I've actually ran all the SPC7110 games on 300 Mhz machines at full speed.

My Atom processor and its battery will love you for that.

Unless we somehow came up with a method that constantly xor'd that extra chunk of data, with the beginning chunk of non changed data. Except that's an issue when the data is 0.

That's the trick I was mentioning before ... it increases zipped patch size of Tekkaman Blade from =~150kb to ~850kb. Not very nice when IPS is only ~180kb.

Nach mentioned this one to me in the past:

Let's say Super Mario World and Super Metroid are the exact same file size. You make a UPS of the difference between the two. By definition, not a single byte is a direct value from either file.

Yet now, a person who only owns Mario can use your patch to create Metroid, or vice versa. The obvious point is you don't own the copyright on either game, but if it's only illegal to have literal data from a game ...

The first and foremost requirement for a patch to be legal would be for all work in it to be legal: no text translation, no modified copyrighted graphics, etc. Eg ~99% of RHDN patches ruled out right off the bat.

After that, if you're applying it to a copyrighted work, you really wouldn't want to use a XOR patch at all. Instead, you'd want a single direction patch, let's call it foo.patch. When you apply the patch, the patcher could then create foo.unpatch locally.

Of course, it's true that when your modified file is smaller, you'll get raw data from the original file. Only a portion of it, but it may be a concern if you have a truly otherwise legal patch.

UPS though wasn't meant solely for translators, but more as a completely generic patching format that is also reversible. A property that to my knowledge is not supported by any other format. Could be really useful for something like a binary version repository able to walk revisions in each direction, while only requiring a single complete file for any one point.

[Nach]

The problem here though is, this is a lot more complicated from an emulator's perspective, because UPS has hashes that need to match, and UPS patches that cover two different completely parts of a ROM image should be able to merged together.

Both IPS and UPS will fail in a multi-patch scenario when two patches or more modify the same byte. The final result will vary based upon the order of patching. UPS gets it a bit worse by having a XOR result, whereas IPS has the final real result, but it's still very bad.

As for patching in order, just read in all the patch checksums, locate the next one in the list, apply it, and repeat. Not easy, but not several-months difficult, either -- hopefully. Or we could treat it like IPS, accepting any kind of arbitrary combination, in which case we'd ignore the checksum field; and patching order wouldn't matter.

I'm aware of all of that.
However, I want to go above and beyond just the cases you listed here, hence why it's not so easy.

And this isn't why it's several months difficult, but because of all the converters and other things I feel must be put out. Many people have told me way back they won't touch a new format without smart converters.

My biggest concern at the moment is that we need a lib that can do chunks at a time. Patching a very large file may freeze up a GUI in some cases otherwise.

I'm not so concerned about that. However, if I finish porting my super optimized one to a C++ class, and add on the two extra levels of buffering I want, it'll be done.

[byuu]

So what if my preference of what it spits out is different than yours? I still don't see why you're up in arms about that.

For one person's DB to be "complete", it has to have all relevant fields filled in -- no blanks.

If a submission to NSRT doesn't require the PCB serial, but one to FitZert does, then the two cannot completely coexist as complete entries. Likewise if NSRT is recording genre and controller connection options, but No-Intro doesn't care about that. Though luckily the latter can be gathered without access to the physical carts, at least. And it doesn't really hurt us if that info is fake, though I still don't like anything being a Wikipedia-style free-for-all.

And how do we settle naming disagreements? Eg person A prefers "Daikaiju", person B (the sane one :P) prefers "Daikaijuu", person C wants "Big Shell Monster".

It scans within a row to find a checksum and pcb serial.

Short of an intelligent parser, this is going to need some sort of formal design. Eg try supporting CSV + XML + IBM dBase III formats all at the same time with a generic scanner. May be possible, but you get the idea: not practical to be universal.

I'll reiterate that I really like the PCB serial#s for specifying exact mappings: they're unbelievably compact, they're official, and they give all the info we need. Best of all there's only ~50-100 of them, tops.

Whereas ten lines of ROM, RAM, MMIO and special chip mapping commands can vary so much more and we can debate naming of each field for eternity. But damn if the latter isn't more flexible, and possibly even required. Some fan translations may not map to any known PCB.

I'd really like to support both methods. But yes, I'd like to get the underlying format for specifying these boards in C++ code taken care of first. Then we can discuss the text file layout.

I'd like it if we could all get together on IRC and discuss that portion together. If we can't share the same universal database (which would be a shame, but double verification is always nice I guess), it would at least be nice to allow people to choose the database provider they want. Assuming we can compromise enough to make that possible.

Many people have told me way back they won't touch a new format without smart converters.

Eh, can't please everyone :P

[Nach]

So what if my preference of what it spits out is different than yours? I still don't see why you're up in arms about that.

For one person's DB to be "complete", it has to have all relevant fields filled in -- no blanks.

Not entirely true. Lots of things can be incomplete till someone contributes it.

If a submission to NSRT doesn't require the PCB serial, but one to FitZert does, then the two cannot completely coexist as complete entries. Likewise if NSRT is recording genre and controller connection options, but No-Intro doesn't care about that. Though luckily the latter can be gathered without access to the physical carts, at least. And it doesn't really hurt us if that info is fake, though I still don't like it anything being a Wikipedia-style free-for-all.

An exact minimal will have to be agreed on. But beyond that? I don't see why we can't offer a bunch more optional fields that people can have their own preferences about.

And how do we settle naming disagreements? Eg person A prefers "Daikaiju", person B (the sane one ) prefers "Daikaijuu".

Well, the big fight in the DB'ing community and naming revolves around the following:
In game title.
Cart title.
Box title.
Official company title (company game list or website or order forms).

Then, what language is the title in? Native? Always English? Transliterated?

I personally prefer to collect it all, people can have their own preferences. I don't think any name is required right off the bat, except knowing which cart it was dumped from, where perhaps cart label + serial is what is required.

As how to handle transliterations, if there are several valid systems, I'd gladly support ALL of them provided they were always consistent about it.

I myself am working with a group writing a commentary on a text. Since the original language isn't English, there calls for transliteration a lot to make it easy to read in English. We have 3 methods for transliteration, every person in the group has their own preference. We cope by storing each method of transliteration, I personally transliterate using all 3 methods since I know them all. When viewing a document (all web browser based), it just uses your preference cookie to display the transliteration style you prefer.

A worthwhile option would also be to not actually store transliterated names at all, but only original names. And when a user wants a transliteration, we can do what PHP offers, and offer options how to handle certain Japanese situations..

Many people have told me way back they won't touch a new format without smart converters.

Eh, can't please everyone

No, but I think this is one qualification desired that can be satisfied.

[FitzRoy]

I don't see how having an online community database which opts to store as much info as possible from as many sources as people are willing to contribute (with a clear privacy policy of course, as needed) is a bad thing.

Furthermore, I'd have it that one would be able to check a couple of boxes as to which data they want, with whatever filtering shenanigans, and have it spit out a file in several different formats for a user to do with as they see fit.

So what if my preference of what it spits out is different than yours? I still don't see why you're up in arms about that.

It sounds all warm and fuzzy, but there are too many issues to deal with. How do you intend to database every naming and inclusion convention? Are you going to attribute everything to its dumper, or just the checksum? No-intro already tried a wiki like this and failed to account for that. You'd have 5 dumpers listed and yet you wouldn't know which of them had provided the other information. If one of the dumpers was later exposed as a fraud, everything but the checksum would come into question because it wasn't documented who submitted them. How much webspace and bandwidth do you suppose 6000 pcb and cart scans would take up and who's paying for it? Seeing as how there's no central authority capable of accepting submissions on sight of them, you'd have to host them. Even if you managed this, there are components and etchings that are bent into occlusion. If you allow people to upload their own images to any game profile, who's going to police vandalism and errors? The game entries aren't threads that rise to the top with updates. Who's backing up the information every day to make sure a hacker or a hard drive crash doesn't wipe out everyone's work? My concerns are endless, but you lost me as soon as I couldn't use my own naming convention or work offline without having to duplicate my efforts.

If you say so. "If anyone unleashed a useless format on the world, it's you Nach" seems to be directed clearly at me, if not, we seem to have major communication issues.

No, I said JMA was a useless format. What of byuu's am I defending here? Sorry you continue to think I'm his henchman or something even after debating him several times in this thread alone.

Short of an intelligent parser, this is going to need some sort of formal design. Eg try supporting CSV + XML + IBM dBase III formats all at the same time with a generic scanner.

I did not consider that and I'm not technically proficient here. Is there something wrong with using the one I posted, ODS? OpenOffice is free and works fine for me. I think it allows me to save in other formats if there's a better one, but we should probably agree on just one.

[Nach]

I don't see how having an online community database which opts to store as much info as possible from as many sources as people are willing to contribute (with a clear privacy policy of course, as needed) is a bad thing.

Furthermore, I'd have it that one would be able to check a couple of boxes as to which data they want, with whatever filtering shenanigans, and have it spit out a file in several different formats for a user to do with as they see fit.

So what if my preference of what it spits out is different than yours? I still don't see why you're up in arms about that.

It sounds all warm and fuzzy, but there are too many issues to deal with. How do you intend to database every naming and inclusion convention? Are you going to attribute everything to its dumper, or just the checksum? No-intro already tried a wiki like this and failed to account for that.

A wiki is the most retarded thing to use for this. Databases exist for a reason.

My design calls for actual fields listed, with a history on each field, and who initially added it, who changed it, and when. This is not a problem.

How much webspace and bandwidth do you suppose 6000 pcb and cart scans would take up and who's paying for it?

I've thought of this. Initially, I'll pay for it out of my own pocket, but it won't exist that way long term, especially considering ideally I'd like to expand beyond SNES and cover more than 6000 PCBs.

I can see two viable solutions.
The site could have some minor Google ads on the side, and try to be unobtrusive.

Or offer a subscription method. Perhaps $2 a month or $10 a year. (Contributors get free accounts of course)
Basically only subscribers can generate a DB with the latest data they want whenever they want, how they want. Non subscribers can only get certain prepackaged databases that are regenerated weekly.

Seeing as how there's no central authority capable of accepting submissions on sight of them, you'd have to host them. Even if you managed this, there are components and etchings that are bent into occlusion. If you allow people to upload their own images to any game profile, who's going to police vandalism and errors?

Like I said, I don't want a free for all like Wikipedia. I want a careful hiarchy, with clear permissions on who can access and modify what. Casual users may only be able to view and offer a comment, not change something which has to be okay'd by staff.

The game entries aren't threads that rise to the top with updates.

Why not? That's exactly what I want to do. Every game entry should have an associated forum thread to discuss it.

Who's backing up the information every day to make sure a hacker or a
hard drive crash doesn't wipe out everyone's work?

The web providers I use have daily MySQL DB backups. I myself backup my own SQL DBs frequently as well. I'd allow for any admin to pull a complete backup as they desire.

My concerns are endless, but you lost me as soon as I couldn't use my own naming convention or work offline without having to duplicate my efforts.

I would be happy to support your naming convention, it just might not be the one I prefer, or the one I have as default in NSRT, but I'd like to record it and allow users to use it nontheless.

As for working offline, I am a lot more capable with cross platform + web platform apps now than I was even just a few months ago.

If it was really needed, I could see writing a program where you browse files, fill in fields and whatever, and when you're online, you'd hit a sync to web button. I don't think it can be done right off the bat though, unless a lot of people want to pitch in with the software development.

If you say so. "If anyone unleashed a useless format on the world, it's you Nach" seems to be directed clearly at me, if not, we seem to have major communication issues.

What of byuu's am I defending here

"If anyone" suggests an exclusion. As in I unleashed a useless format but no one else did.

Directed at me "you Nach", it serves to provide a contrast as to what you perceive I think about others, namely in this case, UPS.

If you subconsciously perceived I was against the UPS format, that would only occur if you believed that it's not my format.

[FitzRoy]

Alright, forget about the wiki misconception.

I would be happy to support your naming convention, it just might not be the one I prefer, or the one I have as default in NSRT, but I'd like to record it and allow users to use it nontheless.

As for working offline, I am a lot more capable with cross platform + web platform apps now than I was even just a few months ago.

If it was really needed, I could see writing a program where you browse files, fill in fields and whatever, and when you're online, you'd hit a sync to web button. I don't think it can be done right off the bat though, unless a lot of people want to pitch in with the software development.

It isn't just about placating me personally, I'm not trying to blackmail you. It's about what every emu author, user or dumping organization wants individually, and you just can't do that with a central database.

"If anyone" suggests an exclusion. As in I unleashed a useless format but no one else did.

Directed at me "you Nach", it serves to provide a contrast as to what you perceive I think about others, namely in this case, UPS.

If you subconsciously perceived I was against the UPS format, that would only occur if you believed that it's not my format.

I didn't mention JMA to implicate all formats you've worked on as being the equivalent. I mentioned it as an example to cast doubt on your suggestion that we all just go away and trust you to come up with the best solution for the database method in private. It's clear now that my concerns were warranted, I wouldn't have been happy with these solutions. It's far easier to defeat a bad idea before you've spent many man hours laboring it into existence.