State of the N64 emulation scene

Post by **grinvader** » Tue Apr 15, 2008 9:03 pm

Mainly on the documentation side. How bad is it ?

Is everyone stuck with stuff barely readable or accurate ?
Noone with appropriate hardware is interested in gathering better results ?

Or is it pretty much laid out and only tiny obscure details are left without easy tests to find what they do/how to do them ?

powerspike · Post by **powerspike** » Tue Apr 15, 2008 9:15 pm

I guess the only real documentation they have so far is probably the mame/mess n64 source. It's not even complete yet or anything like that. I've also been watching mooglyguys progress of it in his blog for awhile. There's always a chance too that the developers have their own documents. You'd probably have to ask Ville lindie or mooglyguy.

http://moogle-tech.com/blog/

Then he made a wiki for the hardware status and for reporting which games boot.
http://www.moogle-tech.com/wiki/index.p ... nce_status

mudlord · Post by **mudlord** » Wed Apr 16, 2008 12:56 am

I guess the only real documentation they have so far is probably the mame/mess n64 source.

Indeed, there's very little hardware documentation apart from that. However, Nintendo has several docs relating to its software ucode, namely Turbo3D, Fast3D, F3DX deratives and others..

So basically, the emu developers have thier own docs. Which is a huge shame.

Post by **grinvader** » Wed Apr 16, 2008 6:25 pm

mudlord · Post by **mudlord** » Thu Apr 17, 2008 4:27 am

Yeah, that should answer your question, and show why there is not much progress made at all.

snkcube · Post by **snkcube** » Thu Apr 17, 2008 5:39 am

That's kind of sad to hear.

I.S.T. · Post by **I.S.T.** » Thu Apr 17, 2008 6:01 am

mudlord wrote:
I guess the only real documentation they have so far is probably the mame/mess n64 source.
Indeed, there's very little hardware documentation apart from that. However, Nintendo has several docs relating to its software ucode, namely Turbo3D, Fast3D, F3DX deratives and others..

So basically, the emu developers have thier own docs. Which is a huge shame.

I take it they don't really share their docs that much?

Rashidi · Post by **Rashidi** » Thu Apr 17, 2008 6:28 am

I.S.T. wrote:I take it they don't really share their docs that much?

the Non-Disclosure thingy?

Deathlike2 · Post by **Deathlike2** » Thu Apr 17, 2008 6:33 am

Rashidi wrote:
I.S.T. wrote:I take it they don't really share their docs that much?
the Non-Disclosure thingy?

No.

I get the feeling there are separate agendas at work, similar to the days where ZSNES (and I think Snes9x as well) at the time were not open sourced. On the other hand (IIRC), they did share quite a bit of info back in the day.

Post by **grinvader** » Thu Apr 17, 2008 7:29 pm

mudlord wrote:Yeah, that should answer your question, and show why there is not much progress made at all.

This blows beyond measure.

ShadowFX · Post by **ShadowFX** » Thu Apr 17, 2008 10:00 pm

I've been a beta-tester for PJ64 for about 2 years now. What I can tell you is that PJ64 hasn't been updated since June 2007 (version 1.7). Yes, you can say there hasn't been progress at all since this was more or less the most active emulator in development at the time.

After that, I put my hopes up for Z64, an LLE based graphics plugin. Nice development in the beginning but was abandoned too soon. Last but not least, Glide64 and MooglyGuy's progress are the only things currently in progress, as far as I know.

Post by **grinvader** » Thu Apr 17, 2008 10:29 pm

ShadowFX wrote:LLE based

Good.

graphics plugin

Bawww.

Also, other emus 'progressing' isn't exactly what I'm looking for. I'm looking for what they're progressing with, if any (trial and error only leads you that far - and most importantly, would lead me nowhere, since I cannot 'try' and see my errors.).
Mostly it seems they just try to mimic how stuff looks. It's obviously a direct path to HLE.
Which blows.
These guys make awesome dynarecs and have an emu accuracy the level of a blind chinese kid making fake nikes. What a waste.

ZH/Franky · Post by **ZH/Franky** » Thu Apr 17, 2008 11:17 pm

Hey, I have a question. What is "dynamic recompilation"?

Is it some kind of speed optimization technique? (if so, does it affect accuracy in any way)

Does bsnes use it?

mudlord · Post by **mudlord** » Thu Apr 17, 2008 11:47 pm

Mostly it seems they just try to mimic how stuff looks. It's obviously a direct path to HLE.

Mainly, they are emulating how the graphics microcode works. So, yes, its HLE. Though Z64 uses shaders to emulate the graphics pipeline, as well as doing internal emulation of the RDP/RSP.

Nice development in the beginning but was abandoned too soon.

Not much you can do if the Frenchman wanted to get a decent job and sort out his life. Same with zilmar, who I know works for a AV company coding backends to firewalls. And plus, he has a contract job for a project, which eats up his time.

Palin · Post by **Palin** » Thu Apr 17, 2008 11:48 pm

Franky wrote:Hey, I have a question. What is "dynamic recompilation"?

It means that instead of following a literal code execution path, it takes short cuts.

A current gen CPU can do things that an SNES or N64 chip can't, yes? So if you run across a line of code that does something like:

empty accumulator
add A to accumulator
add B to accumulator
return accumulator

when you know that your chip allows you to do:

return A plus B

Changing it on the fly to use the second (faster) method would be called dynamic recompilation.

usual disclaimer: I know nothing about emulation and am not a programmer. This post may be blatantly false

byuu · Post by **byuu** » Fri Apr 18, 2008 12:35 am

I get the feeling there are separate agendas at work, similar to the days where ZSNES (and I think Snes9x as well) at the time were not open sourced.

Luckily, the N64 is really starting to get too old for these egotistical popularity contests. So if it hasn't yet, it should change soon. Then again, there's the YM2612 ...

Mostly it seems they just try to mimic how stuff looks. It's obviously a direct path to HLE. Which blows. These guys make awesome dynarecs and have an emu accuracy the level of a blind chinese kid making fake nikes.

Man, low-level N64 emulation. I strongly believe we'll never see true low-level emulation of even the main processor, let alone the entire system.

Take for example: the SNES CPU (65816) is a two-stage pipeline, 16-bit, ~3.58MHz processor, and the N64 CPU (VR4300) is a five-stage pipeline, 64-bit, ~94MHz processor.

I would be surprised if a cycle accurate N64 CPU emulator alone could reach full speed on any modern CPU, regardless of overclock. And it's the synchronization between chips that's the slowest part. Yes, I know we'll keep getting faster and faster systems in the future, but there are limits. I don't think anyone will tolerate an emulator that needs ~100GHz to get full speed, when there are HLE emulators which run all known games with a few hacks on ~1GHz systems. And yes, I believe the differences between low-level and high-level emulation requirements scale exponentially, not linearly. The faster the system, the easier it is to employ more and more complex HLE techniques, and skimp even greater on exact timing.

But more importantly than performance is the work involved. I know of so many interesting effects just from a two-stage pipeline ... I can't even comprehend the amount of pain and suffering involved in a five-stage pipeline. Who in the hell would bring that pain upon themselves when clearly no games even rely on it? And just to create an emulator that nobody will be able to play before the world has completely forgotten about the N64 in the first place?

Sadly, the fourth generation of video game consoles and the sixth generation of handhelds (with some obvious exceptions here and there) are very likely to be the last that will ever be emulated at the lowest possible level feasible by software.

What a waste.

I say that to myself every day when I look at SNES PPU emulation, and then compare it to the discussions over at NESdev. Really, we're the last to be throwing stones about accuracy vs playability tradeoffs.

creaothceann · Post by **creaothceann** » Fri Apr 18, 2008 6:59 am

Franky wrote:What is "dynamic recompilation"?

http://en.wikipedia.org/wiki/Dynamic_recompilation

Game code (e.g. SNES ASM) is translated at runtime into code that can be executed directly by the host CPU (PC ASM).

Deathlike2 · Post by **Deathlike2** » Fri Apr 18, 2008 7:07 am

byuu wrote:Luckily, the N64 is really starting to get too old for these egotistical popularity contests. So if it hasn't yet, it should change soon. Then again, there's the YM2612 ...

AFAIK, there's no good Genesis emu source, just a good emu. I guess some people can live with that.

Mostly it seems they just try to mimic how stuff looks. It's obviously a direct path to HLE. Which blows. These guys make awesome dynarecs and have an emu accuracy the level of a blind chinese kid making fake nikes.
Man, low-level N64 emulation. I strongly believe we'll never see true low-level emulation of even the main processor, let alone the entire system.

Take for example: the SNES CPU (65816) is a two-stage pipeline, 16-bit, ~3.58MHz processor, and the N64 CPU (VR4300) is a five-stage pipeline, 64-bit, ~94MHz processor.

I would be surprised if a cycle accurate N64 CPU emulator alone could reach full speed on any modern CPU, regardless of overclock. And it's the synchronization between chips that's the slowest part. Yes, I know we'll keep getting faster and faster systems in the future, but there are limits. I don't think anyone will tolerate an emulator that needs ~100GHz to get full speed, when there are HLE emulators which run all known games with a few hacks on ~1GHz systems. And yes, I believe the differences between low-level and high-level emulation requirements scale exponentially, not linearly. The faster the system, the easier it is to employ more and more complex HLE techniques, and skimp even greater on exact timing.

It's a matter of time. Right now, I think people can forgive accuracy if their games play within a reasonable rate... it's happened with most emus that didn't quite have the resources at the time, so this trend will continue until the hardware is capable or someone is really good at optimizing (trying to be better than the compiler would take some thought).

But more importantly than performance is the work involved. I know of so many interesting effects just from a two-stage pipeline ... I can't even comprehend the amount of pain and suffering involved in a five-stage pipeline. Who in the hell would bring that pain upon themselves when clearly no games even rely on it? And just to create an emulator that nobody will be able to play before the world has completely forgotten about the N64 in the first place?

Sadly, the fourth generation of video game consoles and the sixth generation of handhelds (with some obvious exceptions here and there) are very likely to be the last that will ever be emulated at the lowest possible level feasible by software.

First things first. Emulating the damn system properly is more important than hacking it to be game specific. Then again, I've heard more about retexturing projects than emulation (and trying to make those silly requests here to boot).

What a waste.
I say that to myself every day when I look at SNES PPU emulation, and then compare it to the discussions over at NESdev. Really, we're the last to be throwing stones about accuracy vs playability tradeoffs.

N64 emulation hasn't even reached anything similar to the SNES at the same amount of time (and the SNES had great games via special chips and other related stuff). That says more about the scene than the hardware resources required for emulating the console in question.. and that's saying quite a bit. Maybe I'm not factoring complexity correctly, but shouldn't progress be better than what is currently available?

mudlord · Post by **mudlord** » Fri Apr 18, 2008 11:49 am

Maybe I'm not factoring complexity correctly, but shouldn't progress be better than what is currently available?

Well, byuu hit the nail on the head why cycle accuracy definately isn't gonna happen in the next couple of years. Though from my perspective, the scene itself is kinda dead, with zilmar and Jabo busy with thier jobs. Which leaves Mupen64, which still is active, so there is hope left I reckon. Though Glide64 now is very mature, but I guess people are after a more purist approach to emulation, than just emulating software APIs.

As for progress, I reckon it should be a bit better. I know Azimer did some recent research into the MusyX sound system, which is a bitch to emulate games that use it, since the sound system is very picky on timing.

Nightcrawler · Post by **Nightcrawler** » Fri Apr 18, 2008 1:58 pm

Deathlike2 wrote: It's a matter of time. Right now, I think people can forgive accuracy if their games play within a reasonable rate... it's happened with most emus that didn't quite have the resources at the time, so this trend will continue until the hardware is capable or someone is really good at optimizing (trying to be better than the compiler would take some thought).

First things first. Emulating the damn system properly is more important than hacking it to be game specific. Then again, I've heard more about retexturing projects than emulation (and trying to make those silly requests here to boot).

N64 emulation hasn't even reached anything similar to the SNES at the same amount of time (and the SNES had great games via special chips and other related stuff). That says more about the scene than the hardware resources required for emulating the console in question.. and that's saying quite a bit. Maybe I'm not factoring complexity correctly, but shouldn't progress be better than what is currently available?

Man, you make it sound like it's trivial to pump out a low level, accurate N64 emulator. It almost sounds naive or ignorant, which is surprising coming from you.

You can't possibly compare the SNES to the N64 like that. Why would you expect N64 emulation to progress at the same rate as SNES emulation or the same degree of accuracy? They are nothing alike and the N64 is far more complex a system on the low level. The only saving grace as byuu mentioned is it's unnecessary to really get down to a cycle level on the N64 because games quit utilizing system resources in a way where that's critical. (Probably thanks to using high level programming languages).

So I ask you WHY SHOULD progress be better than what's currently available?

If there's been any set back in N64 emulation, it's been there are less people interested in that system versus many others. typically the more people involved in something, the faster developments take place. I do think N64 emulation could have been further along had more of our talented emulation folks been interested.

Of course it was also pointed out N64 documentation is poor to say the least. So, I'd imagine it would be very difficult for a new guy to jump into the mix without a lot of help from his peers to get up to date on all the N64 findings. You still need that even WITH good documentation, but at least you can get by fairly well by yourself if you have to.

byuu · Post by **byuu** » Fri Apr 18, 2008 2:55 pm

AFAIK, there's no good Genesis emu source, just a good emu. I guess some people can live with that.

Right, just kind of sad that we still have the separate agendas thing going on, even for a 20 year old video game console.

It's a matter of time. Right now, I think people can forgive accuracy if their games play within a reasonable rate... it's happened with most emus that didn't quite have the resources at the time, so this trend will continue until the hardware is capable or someone is really good at optimizing (trying to be better than the compiler would take some thought).

I really think you're underestimating the exponential growth factor I mentioned.

NES -> 1.68MHz
SNES -> 3.58MHz
N64 -> 93.75MHz

The honest truth is that the SNES is just a souped up NES. The N64 is far more than that. Though the SNES has a better sound system, as pathetic as that is.

You say, a matter of time ... I think a fully optimized, perfected SNES emu could maybe get 60fps on a modern machine. But for such a radical jump in power, it's going to be a very, very long time before N64 low level emulation at full speed is really feasible.

You have to wonder if anyone will even care by that point. The N64 wasn't even very popular during its prime.

First things first. Emulating the damn system properly is more important than hacking it to be game specific.

We agree on that, but I've yet to see a single major console system to be emulated properly before hacked up to run games. The latter is what the vast majority want.

N64 emulation hasn't even reached anything similar to the SNES at the same amount of time (and the SNES had great games via special chips and other related stuff). That says more about the scene than the hardware resources required for emulating the console in question.. and that's saying quite a bit. Maybe I'm not factoring complexity correctly, but shouldn't progress be better than what is currently available?

If you account for the exponential increase of complexity, then you should expect an exponential increase in time for the same level of improvement.

But I agree that N64 emulation should be more than it is now. Trying to be careful to avoid the "let's see you do better then" line here ...

I think the biggest reason is because the N64 was honestly just not that popular. The same reason we have more NES emulator authors today. If you look at the much more successful PSX, we have people like pSXAuthor who are making serious strides even today.

My personal beliefs are that we should stop at nothing in the pursuit of accuracy, but I find that after having worked on something for several years, you gain a real attachment to it and can't move yourself to make it so slow as to be unplayable on your own machine.

So I guess my optimism would hope for an N64 emulator that utilizes 100% of modern CPU power with the best possible tradeoffs, while still being playable. Maybe we already have that. I haven't even touched the scene, myself. I'd sooner buy the real system off eBay with the level we're at now :/

So I ask you WHY SHOULD progress be better than what's currently available?

... I don't like the sound of that :/
The same reason the NES scene strives onward, even though there's no known official NES game that doesn't run on at least one emulator. The hardware will eventually be non-existent (~50 years? ~100 years?), and there will be no way to improve emulation after that. Improvements help out fan hacks to work the same on hardware as software. Etc, etc.

AamirM · Post by **AamirM** » Fri Apr 18, 2008 4:37 pm

Hi,

Not to deviate too far from the topic but anyways.

byuu wrote:Right, just kind of sad that we still have the separate agendas thing going on, even for a 20 year old video game console.

How can you say that? What reasons do you have?
As far as I see, there are no separate agendas.

Deathlike2 wrote:AFAIK, there's no good Genesis emu source, just a good emu. I guess some people can live with that.

Most emu devers fail to understand that source can never replace good documentation of a system. It enables more people (especially newbies) to understand it. Sources are just a translation of a doc from english(or whatever) to C/ASM(or whatever). Sources are for computer to understand. I know I'll get ripped on for this

.

stay safe,

AamirM

Deathlike2 · Post by **Deathlike2** » Fri Apr 18, 2008 5:07 pm

Nightcrawler wrote:Man, you make it sound like it's trivial to pump out a low level, accurate N64 emulator. It almost sounds naive or ignorant, which is surprising coming from you.

I never said it was trival. I'm saying that N64 emulation (documentation) should be better than "can't help you there".

You can't possibly compare the SNES to the N64 like that. Why would you expect N64 emulation to progress at the same rate as SNES emulation or the same degree of accuracy? They are nothing alike and the N64 is far more complex a system on the low level. The only saving grace as byuu mentioned is it's unnecessary to really get down to a cycle level on the N64 because games quit utilizing system resources in a way where that's critical. (Probably thanks to using high level programming languages).

The complexity is there, period. Far more than the PSX obviously.

So I ask you WHY SHOULD progress be better than what's currently available?

I'll admit, I can't say it's a slam dunk that N64 emulation would be better. On the other hand, the information being kept by individual groups/people is slowing down the research done. Instead of massive time spent on verification and testing, it could be reduced to having a test program or going through established guidelines for testing a specific component.

If there's been any set back in N64 emulation, it's been there are less people interested in that system versus many others. typically the more people involved in something, the faster developments take place. I do think N64 emulation could have been further along had more of our talented emulation folks been interested.

Yea, that's always been the case for any system.

byuu wrote:I really think you're underestimating the exponential growth factor I mentioned.

Maybe I'm underestimating it a little, but I have no expectations of anything close to completion. I expect some sort of reasonable framework in place (though it may end up having to be rewritten in the end) that when new information is available that it doesn't require code implosion (aka an immediate rewrite). Then again, is the lack of documentation a result of having few (if any?) open sourced N64 emus, or that the N64 emu sources can't be made heads or tails of?

If you account for the exponential increase of complexity, then you should expect an exponential increase in time for the same level of improvement.

Agreed, but I wasn't looking for drastic differences if things were different. I'm saying the situation would be slightly better, but it's truly hard to measure that result.

Most emu devers fail to understand that source can never replace good documentation of a system. It enables more people (especially newbies) to understand it. Sources are just a translation of a doc from english(or whatever) to C/ASM(or whatever). Sources are for computer to understand. I know I'll get ripped on for this .

Sure, it hasn't been the first or last time that's happend.

The closed source emus that I'm aware of have certain valid reasons why they can't be open sourced (based on their situation)... it's a little disappointing there, but if they are available/accessible for discussing emulation details, it's still relatively helpful.

Post by **grinvader** » Fri Apr 18, 2008 5:23 pm

The main problem I see is that noone even considered emulating the base system at low level.
Everyone's waiting for a microcode to get RE'd or trial and error'd to death so they can wrap stupid standard PC video card calls over an obviously different architecture.

I don't even see where these damn µcodes are. The n64 architecture may be a mess, but my lack of knowledge makes it an obscure mess.

Let me detail:
For the snes, you grab the rom, load it (memory map stuff happens, due to how the cart is wired), go wherever you have to and start the core execution loop.

The n64 being another cartridge system, I'm gonna guess that there are gonna be steps relatively similar to the SNES, like memmapping, vectors and so on.
What sort of design pretty much forced everyone to work on a top-to-bottom approach for that ?
5-stage pipeline or multiple processors to synchronise is only an increase in complexity over a situation I could understand (and I don't see microcodes in these).

So... yeah. Pretty obscure to me. Feel free to enlighten with any method available.

byuu · Post by **byuu** » Fri Apr 18, 2008 6:22 pm

How can you say that? What reasons do you have?
As far as I see, there are no separate agendas.

Really? How's your YM2612 emulation coming along? Good as Kega's yet? And will that see its way into Gens Plus (or whatever)?

It's so nice that SNES emulation doesn't have these sharing issues. I would have never even bothered to get into emulation if it did.

Not trying to get into another argument on that topic though, sorry. I only brought it up because it has possible relevance to the lack of N64 emulation progress.

Most emu devers fail to understand that source can never replace good documentation of a system.

And most non emu devs fail to realize that documentation cannot be unit-tested like source code. It's much easier to properly verify source with appropriate unit tests (eg test ROMs) than it is to verify documentation is correct.

I'd rather have documentation than poorly written source code, but I'd rather have cleanly written, well commented source code with no speed hacks than documentation.

But still, you're right that both are very important.

Then again, is the lack of documentation a result of ...

No idea. Lots of emu devs simply don't want to write documentation. Most emu devs can read source code to figure things out, though. I only had Y0shi's and Qwertie's stuff, so Snes9x was much more valuable to me.

I don't even see where these damn µcodes are. The n64 architecture may be a mess, but my lack of knowledge makes it an obscure mess.

You got me curious, but I couldn't find any good info on this, either.

My (possibly incorrect) understanding of it is that the RSP/RDP is a lot like specialized SNES DSPs. We speculate that most (all?) of the DSP-n chips are the same internal chip, and that the only difference is the internal program ROM (and data ROM, obviously). I believe this is what is called "microcode". Our emulation of the DSP-n chips now is essentially to simulate what these chips do.

"Okay, Pilotwings wrote 08h to port 03h. That means we need to calculate the arctangent of n." -- but in reality, that's done via executing a bunch of math ops inside the DSP itself.

Only the N64 RSP/RDP implemented far more complex operations, or whatever.

The big difference between the SNES and N64, of course, was that the individual games contained the program data to send to the RSP/RDP, and these games could write their own custom implementations. This made it more of a co-processor than a DSP, but I digress. Most shared the same program, but it was refined over time (ala DSP-1A/B) and rewritten entirely sometimes (ala DSP-2/3/4.) Nobody has ever been able to dump the program data from the SNES DSP chips, so we couldn't emulate those at a low level, even if we wanted to.

But because of this, SNES DSP emulation has much of the same problems. Luckily, it isn't too noticeable, but our fake simulation of these chips cause many timing problems, such as track flickering in Mario Kart.

It does make me wonder, though. If we were to successfully dump those DSP program ROMs one day, and a low-level implementation caused a massive speed hit, would all of us emulate it, or stick with our high-level emulation, simply because our simulation is good enough for full compatibility already?

EDIT: not sure which chip is the one that is commonly emulated at a high level, the RSP or the RDP. Or if they're both emulated with HLE.