Resampler, vsync refinement

[blargg]

I still think the audio resampling and vsync can use some refinement. I've come up with a simpler conceptual model for thinking about this resampling, which I think would make things clearer for everyone. First, consder how many samples and frames the SNES generates per second:

Code: Select all

          One second
|--------------------------|
############################ 32040 samples
[][][][][][][][][][][][][][] 60.09 frames

Then, divide the samples among the frames to find out how many samples are generated per frame:

Code: Select all

## 533.2 samples
[] 1 frame

Since time was eliminated, things are simpler to think about. All we now need to keep in mind is that for every emulated frame, we'll get 533.2 samples along with it. Now, turn to the PC:

Code: Select all

          One second
|--------------------------|
############################ 48000 samples
[][][][][][][][][][][][][][] 60 frames

Again, divide the samples among the frames:

Code: Select all

## 800 samples
[] 1 frame

Again, we can ignore time. For every frame we display, we must generate 800 samples. So, it's a simple matter of generating 800 samples from every 533.2 that come in. Tell the resampler to use an input:output ratio of 533.2:800 = 0.6665. So the linear resampler shown in another thread would "consume" 0.6665 input samples for every output sample generated.

Now, there's the issue of the PC not being exactly what it claims. Let's say the display really runs at 59.9 frames per second. This means that we need to generate 48000/59.9 = 801.3 samples per frame. Another example, if the sound card actually runs at 47987 samples per second, then we need to generate 47987/60 = 799.8 samples per frame. The user might not know these exact values, so he can adjust the samples per frame until he doesn't get gaps in audio (too few per frame) and video frames displayed for twice as long (too many samples per frame) are at a minimum.

What comes from this is that deviations of the PC's display rate or sound card rate can both be compensated for by adjusting how many samples we generate per frame. All the user has to do is adjust the value. There are a few ways to do so. The user could be presented with the number of samples per frame, and adjust that, or he could simply adjust a percentage that is applied to the ideal value. For example, he could tell it to use 98% of the ideal value (800*0.98 = 784 samples), or 102% (800*1.02 = 816 samples). The percentage seems more user-friendly, since it's not exposing irrelevant internal details, and it probably would apply even if the sound card rate were changed (for example from 48000 to 44100), since deviations are probably due to the clock crystal rather than peculiar to a particular rate.

What should the "samples per frame" control be labeled? Fundamentally it's causing sound to run faster or slower, without affecting video rate. The more samples per frame, the lower the sound pitch. So 102% would lower pitch since it stretches things out more. Maybe the slider could simply be labeled

Code: Select all

A/V sync: Smoother video, [-----------O------------] Jumpy video,
          audio crackling                            clean audio

Summary:

SNES: 32040 samples per second, 60.09 frames per second = 533.2 samples per frame
PC: 48000 samples per second, 60 frames per second = 800 samples per frame
User adjustment: 800 samples per frame * value close to 1.0
Resampler: For every 533.2 input samples, output 800*adjust samples. Simple.

[byuu]

I was up for about five hours after we spoke last night working on this. This is what ended up happening:

The computation for the resampler grew significantly in difficulty. What used to be:
ResampleOutput = {AudioFrequency}
ResampleInput = {FrequencyAdjust} * Scale

Was now:
SPFAdjust = {SamplesPerFrame} * {AudioFrequency} / 32000
ResampleOutput = (32000 / NTSC ? 60 : 50) * SPFAdjust
CPUCPS = NTSC ? NTSCCPUClock / (1364 * 262) : PALCPUClock / (1364 * 262)
SMPSPS = NTSC ? NTSCSMPCLOCK / 768 : PALSMPCLOCK / 768
ResampleInput = CPUCPS / SMPSPS

{} placed around user-specified values.

I found that the rate of 32000(hz) for the PC sound card was not perfect, I also found that the monitor value of 60(hz) was not either. We are basically adding two new "default" variables that we don't really know into the equation.

I do like that it factors in the CPU / SMP clock rate, but those honestly aren't ever going to be changed by anyone from the defaults.

I had hoped that this new model would simplify things, but for me at least things were a little worse. With the former model, I needed a FrequencyAdjust of -40 (31960hz). With the latter, I needed a SamplesPerFrame of +0.00065% over the base (533.333.)

To get the same range as the old +/- up to 200hz in either direction, my slider had to step by increments of 0.00003%, which seemed overly verbose.

Where it got bad was that it did indeed cause issues when I tried changing audio resampling rate on the PC side. I had to adjust SamplesPerFrame by ~0.0001% to fix the audio/video; whereas I did not have to change FrequencyAdjust at all regardless of AudioFrequency. This could be a fluke specific to my system due to some unfortunate rounding, though.

I was also forced to recalculate the skew whenever a new ROM was loaded that changed between NTSC and PAL modes, because of the assumptions in the calculation. The reason being that output samples per frame are ~533.333 on a perfectly calibrated 32khz:60hz system, but 640 for PAL. Since we don't ask the user for their monitor refresh rate, we have to base it somewhere, and that was the most logical place to do it.

For either method, your adjustment value will only work in either NTSC or PAL, but never both. But you have to exit the emulator and change your refresh rate anyway, which I doubt anyone would really be doing. If we find that people are doing this, we can make both NTSC and PAL adjust sliders at that time.

Also, by assuming a monitor refresh rate, we were making it more complicated to compute a value that gets you a perfect 75/85hz video output with smooth audio. I can't possibly imagine why anyone would want to do this (audio would play way too fast), but by assuming a monitor rate of 50/60, the SamplesPerFrame value for 85hz will have to compensate for the incorrect assumption that SPF = 32k/(60,50), so it's no longer simply SamplesPerFrame.

In the end, I found that both methods required a value that was pretty far from the center, at least on my system. Both can get a correct value for any system. And both rely on magic values we simply cannot know. The end user cannot figure out these variables either, and that's not even a matter of technical proficiency. You'd need a test to spend as long as you want clean video+audio calibrating. Eg if you want it to be perfect for 10 minutes, you'd need to calibrate for 10 minutes.

Might as well use the simpler method, it's a lot easier to understand in the end. "The number of SNES input samples used to generate the audio data for the output." No internal assumptions throwing off what it really does. It also makes it easy to disable the resampler (set the value to zero and run audio output at 32khz), and to get emulation running at nearly the exact speed of real hardware, video tearing be damned (set the value to +40, eg 32040 to 32000).

[henke37]

Stuff being userfriendly is important, but what is even more important is it having the needed features. We need exact options where it is possible to just type a number. Add sliders and other fun stuff, but don't take away the needed parts. Sure, make two things control the same value, but don't remove the precision.

[Verdauga Greeneyes]

To get the same range as the old +/- up to 200hz in either direction, my slider had to step by increments of 0.00003%, which seemed overly verbose.

Can I suggest you use something like ppm instead? I doubt the adjustment will be anywhere near 1% for anyone by the sound of things..

Where it got bad was that it did indeed cause issues when I tried changing audio resampling rate on the PC side. I had to adjust SamplesPerFrame by ~0.0001% to fix the audio/video; whereas I did not have to change FrequencyAdjust at all regardless of AudioFrequency. This could be a fluke specific to my system due to some unfortunate rounding, though.

This is odd, but how often are you really going to change your AudioFrequency? I don't think this is a problem really.

I was also forced to recalculate the skew whenever a new ROM was loaded that changed between NTSC and PAL modes, because of the assumptions in the calculation. The reason being that output samples per frame are ~533.333 on a perfectly calibrated 32khz:60hz system, but 640 for PAL ... we can make both NTSC and PAL adjust sliders at that time.

I don't really understand how this got worse compared to the previous method, but what's stopping you from making sliders for NTSC and PAL in the first place? As long as the user is even vaguely aware that the two are different, they won't mind setting both to the perfect values for their system.

Also, by assuming a monitor refresh rate, we were making it more complicated to compute a value that gets you a perfect 75/85hz video output with smooth audio.

When I tried the WIP a few days ago I forgot to change my refreshrate to 60 hz before and found the video to be horribly blocky. Adjusting it so that it's perfect sounds crazy to me. Also, isn't assuming a 50hz refreshrate a bit odd? Who's going to manage that? Surely 60hz would be a better base assumption, if any have to be made. (again I don't really understand the need for these assumptions over the old model - how did the old model come up with the base values?)
Edit: what about querying the refresh rate on any driver that supports it, and returning 60hz to bsnes on any that doesn't? (or if an invalid value is returned)

In the end, I found that both methods required a value that was pretty far from the center, at least on my system. Both can get a correct value for any system. And both rely on magic values we simply cannot know. The end user cannot figure out these variables either, and that's not even a matter of technical proficiency. You'd need a test to spend as long as you want clean video+audio calibrating. Eg if you want it to be perfect for 10 minutes, you'd need to calibrate for 10 minutes.

Okay, this is a blatant feature request, but could you build in some sort of calibration that the user could run when first running an NTSC or PAL game? All this adjusting sliders business sounds more than a little tedious to me - I really like blargg's suggestion to simplify things, and his explanation about the PC being off makes a great deal of sense, but it does sound like that would mess up defaults no end.

[FitzRoy]

And both rely on magic values we simply cannot know. The end user cannot figure out these variables either, and that's not even a matter of technical proficiency.

They don't need to know what you guys know. All they need to know is what slider to use to improve their sync and what direction incurs what. My note would have solved that.

(set the value to zero and run audio output at 32khz), and to get emulation running at nearly the exact speed of real hardware, video tearing be damned (set the value to +40, eg 32040 to 32000).

So that's why you left the positives in there. Here I was thinking that 0 was actually 32040. I guess advanced would have revealed that.

Okay, this is a blatant feature request, but could you build in some sort of calibration that the user could run when first running an NTSC or PAL game?

Ugh, popups suck.

[Verdauga Greeneyes]

Ugh, popups suck.

I agree, which is why I would like it to be a button placed next to the slider. If you press it you'd be asked how long calibration should take (with some sensible default value), after which bsnes would run the game for that amount of time and adjust the slider. (it'd be nice if the user could interrupt the process manually as well, of course)

[byuu]

We need exact options where it is possible to just type a number.

And you have it. Set advanced.enable to true, and you can type any number you want into a textbox, rather than a slider.

Also, isn't assuming a 50hz refreshrate a bit odd?

It is uncommon, but it's the only way to get PAL running smoothly at normal speed. I have both a television and a monitor that can manage 50hz. Ironically, the most expensive one, my main LCD monitor, only does 60hz.

Edit: what about querying the refresh rate on any driver that supports it, and returning 60hz to bsnes on any that doesn't?

I've already stated that you can't get exact values that way. The monitor will just say "60", even though it's 59.998hz. And if it returns something other than 60/NTSC or 50/PAL, you'll get horrible video anyway. May as well assume 60/50, then.

Okay, this is a blatant feature request, but could you build in some sort of calibration that the user could run when first running an NTSC or PAL game?

If I do anything, I'll make a separate program that you can run as long as you like, and it will converge on the best value for you. I won't offer binaries for Linux, because of OpenGL / Xv pains. I'll leave that in source.

I really like blargg's suggestion to simplify things, and his explanation about the PC being off makes a great deal of sense, but it does sound like that would mess up defaults no end.

I'm not sure where you're getting that from. I find both methods to be very straight-forward to understand conceptually. blargg's tries much harder to get a centered value to be "ideal", whereas mine forces it off to the left due to not accounting for normal PC<>SNES differences (eg 32khz vs 32.04khz output rates.) blargg's advantage is clear, eliminate differences the user isn't going to know about; but it relies on PC rate values we do not know, which can also be wrongly assumed.

And either way, we end up with magic values. But the situation really isn't that bad. It takes maybe three or four minutes to get a near-perfect value. I just played an hour of DKC and heard one audio pop, with zero video doubling. Need to move the slider right some, I guess.

Most people should be quite happy with the defaults, video will only double once every few seconds. How much time they spend perfecting it is up to them.

Speaking of which, how many emulators have absolutely perfect video+audio sync, anyway? I noticed Kega skips every ~5 seconds on my system, and I've never heard anyone complain about it.

My note would have solved that.

Yeah, I hate littering the UI with notes. The one on the driver tab was bad enough. What I wanted to do was add in tooltips, or at least a built-in help or something. Not sure yet. Sorry for omitting the idea for this release, I wasn't ignoring or discounting it, I do think it's a good idea.

So that's why you left the positives in there. Here I was thinking that 0 was actually 32040. I guess advanced would have revealed that.

Yes, the offset value is just saying how many samples extra you need to generate 32khz output. -1 means one less, or 31999. +1 means one more, or 32001. I didn't want to factor in the SNES clock rate of 32040hz. You can also control S-SMP clock rate (and thus SNES sample rate) from the advanced panel, which would make the current system have another "magic value."

Plus, I know we had five or so people test; but there's going to be people with ~61-62hz displays out there, I'm convinced of that. Supporting a bit of tolerance in each direction is a very good thing. And if they end up needing a lot, we have the advanced mode now.

[FitzRoy]

Ugh, popups suck.

I agree, which is why I would like it to be a button placed next to the slider. If you press it you'd be asked how long calibration should take (with some sensible default value), after which bsnes would run the game for that amount of time and adjust the slider. (it'd be nice if the user could interrupt the process manually as well, of course)

I don't think bsnes can detect when tearing or crackling is occurring and automatically adjust the slider for you. That would be pretty cool, though.

Yeah, I hate littering the UI with notes.

Heh, and you think I want to? I'm the last person you'd expect to want that. But I wouldn't have suggested it unless I thought it was absolutely necessary. It's simply too confusing without it. I would like the tooltips idea, but I foresee something like that going overboard when in reality you currently have a paltry 2 things needing warnings or explanations.

[byuu]

I don't think bsnes can detect when tearing or crackling is occurring and automatically adjust the slider for you.

I can detect it on some drivers, but not all. But you run into issues with the PC suddenly getting a spike in CPU usage and it throwing your values way off to compensate. Then you end up with much worse video and sound for a longer period of time. A manual value would correct itself as soon as the CPU spike went away.

[Verdauga Greeneyes]

I can detect it on some drivers, but not all. But you run into issues with the PC suddenly getting a spike in CPU usage and it throwing your values way off to compensate. Then you end up with much worse video and sound for a longer period of time. A manual value would correct itself as soon as the CPU spike went away.

1.) Detecting crackling means detecting buffer underruns, right?
2.) Couldn't you weigh values on, I don't know, a gaussian scale? The current average would be weighed most heavily, and a spike wouldn't get much weight at all.
(I dunno if the numbering looks obnoxious, was just trying to indicate the questions aren't particularly related )

[henke37]

Plain hysterics also should work, just ignore changes that is too minor.

[blargg]

This is what byuu's response often feels like when I try to simplify things without eliminating anything essential:

Maybe it's because I'm overstepping things and stepping outside my "territory". Still, it's discouraging because it almost seems like a simpler (while still sufficient) model is not wanted. Or maybe it's just me.

[byuu]

Not following your point. How is my model more complex than yours?

Neither method makes it easy to get the correct number. Both methods have one slider that you move. Both can be described in a single sentence.

As I said, I spent ~5 hours trying out your model. I'm not simply ignoring good advice.

[blargg]

Oh well, I didn't communicate clearly, sorry. Nevermind.

[I.S.T.]

>.>

<.<







Sorry, blargg, but it was too tempting.

[blargg]

Nope, there's some kind of protocol mismatch and I don't know how to solve it. Result: lost opportunities. Oh well.

[franpa]

Since the other Vsync thread is now locked, I will ask this here. Why does the functioning of Vsync in BSNES determine where it is placed in the GUI instead of the end effect achieved by the Vsync setting?

pretty much every other emulator in existence that has a Vsync option would have it placed in Video Mode and not Emulation Speed because it alleviates any Tearing that occurs in the current video mode! >.>


EDIT: removed the Video Filter part.

[FitzRoy]

They put it in the filter settings? I'd like to see that

Syncing emulation speed to a video or audio rate is exactly that. Back in the old days, sync to audio wasn't even there, it was implied because that's all bsnes could do. So the emulation speed settings were the equivalent of syncing to audio at that rate and "Uncapped" was the equivalent of desyncing from audio and letting the emulation speed run free.

[DancemasterGlenn]

They put it in the filter settings? I'd like to see that :lol:

[franpa]

I am aware of what it does, I am just saying that the position of it in the GUI should be based on the end effect rather then the actual changes it makes.

[FitzRoy]

I am aware of what it does, I am just saying that the position of it in the GUI should be based on the end effect rather then the actual changes it makes.

The point is that bsnes is not other emulators. Moving the syncs from that section would be confusing, not only for eliminating tearing, but for the emulation speed options which are based off of them and only work when they're enabled. There's no way a user would figure that out if they were taken out of that section. Most other emulators don't have those options or a way to disable audio sync, so they can get away with sticking vsync in a general video section. Filter is just plain illogical, because, well, it isn't one.

[franpa]

There's no way a user would figure that out if they were taken out of that section.

I'm pretty sure the vast majority of users wouldn't be able to figure out what it actually does no matter where it is placed. It only concerns SNES development and anyone working in that field shold be smart enough to read a readme file that will most likely explain the options in detail.

[FitzRoy]

I'm pretty sure the vast majority of users wouldn't be able to figure out what it actually does no matter where it is placed.

Why not? No one's ever reported confusion with the emulation speed section as long as the syncs have stayed in there. I think that will change if they're moved out and split up. By the way, they're not for development, they're for cheating.

[franpa]

It concerns SNES ddevelopment in that SNES developers would want to DISABLE them if they want to accurately test there game and determine if it will work on a real console.

[Verdauga Greeneyes]

It concerns SNES ddevelopment in that SNES developers would want to DISABLE them if they want to accurately test there game and determine if it will work on a real console.

No, it doesn't effect emulation at all. It only effects the end user experience. That said, the developer might want to ensure that the experience is the same as on the SNES, but I doubt messing around with the synchronisation options will help much with that