bsnes v0.036 released

Archived bsnes development news, feature requests and bug reports. Forum is now located at http://board.byuu.org/
Locked
byuu

bsnes v0.036 released

Post by byuu »

byuu.org wrote:2008-09-14 - bsnes v0.036 released

This release fixes a somewhat serious bug introduced in v035, and also vastly improves Windows support for non-ANSI filenames.

The bug was triggered when HDMA would occur during DMA. If the DMA were long enough, subsequent HDMA transfers would be blocked. This caused graphical glitches in Star Ocean, Super Mario Kart, and possible more games. If you noticed any regressions from v034 to v035, this was almost certainly the cause. Once again, we're operating under the assumption that there are no known bugs currently, so please let us know here if you find any.

I've also rewritten the file handling for the emulator. On Windows, attempting to load a file with non-ANSI characters (eg Russian, Japanese, etc) would cause these characters to be removed. This meant that no version of bsnes thus far could load these files. This problem was exacerbated when I ported the user interface to Unicode (UTF-16), this caused even config and locale file loading to crash the emulator.

The root of the problem is that Windows only accepts non-ANSI strings in UTF-16 format, whereas bsnes' UI wrapper converts strings to UTF-8 interally. When passing these file names to the standard file functions (fopen(), std::ifstream, etc), file loading would fail. To fix this, I replaced all file access functions with a new version that would convert the UTF-8 filenames back to UTF-16, and use appropriate access functions (_wfopen(), _wmkdir(), etc.)

... but there is still one limitation to this: ZIP and GZ support use zlib, and JMA support uses libjma. Neither of these libraries convert UTF-8 strings to UTF-16 before attempting to open files. Due to licensing issues, as well as technical issues, I am unable to correct this at this time. What this means is that loading ZIP, GZ and JMA files; on Windows only; and with Unicode characters in the file name only; will cause the image load to fail. Loading uncompressed images (SMC, SFC, etc) will work with or without Unicode on all platforms.

I tried to be as thorough as possible with this fix: command-line arguments (via CommandLineToArvW + GetCommandLineW), user path (via SHGetFolderPathW), real path (via _wfullpath),folder creation (via _wmkdir) and file access/existence checks (via _wfopen) were updated in all cases. I also updated file loading for ROMs (SMC, SFC, etc), save RAM (SRM), real-time clock save (RTC), cheat files (CHT), UPS patches (UPS) and both configuration files (bsnes.cfg and locale.cfg.) Configuration file loading should work even if your username contains non-ANSI characters, and it should also detect config files put in the same folder as the bsnes executable, even if the path to the executable contains non-ANSI characters.

Still, if you spot any bugs, aside from the ZIP/GZ/JMA loading issue, please let me know via e-mail at setsunakun0; at hotmail.

Lastly, I'd like to apologize for the poor support for non-ANSI filenames in the past. Using an English version of Windows didn't expose the problems to me. I'll be more thorough in the future with this.
mudlord
has wat u liek
Posts: 559
Joined: Tue Sep 11, 2007 2:54 pm
Location: Banland.

Post by mudlord »

Nice job with the hiro fixes. :)
gllt
NO VOWELS >:[
Posts: 753
Joined: Sun Aug 31, 2008 12:59 pm
Location: ALABAMA
Contact:

Post by gllt »

Fantastic!
Just built last version today in Sid, I enjoy how simple it is to build and install.
wertigon
Rookie
Posts: 46
Joined: Sat Aug 07, 2004 7:20 pm

Post by wertigon »

Good job, as always. So, what's next? Time to simulate those peripherals? :)
FitzRoy
Veteran
Posts: 861
Joined: Wed Aug 04, 2004 5:43 pm
Location: Sloop

Post by FitzRoy »

wertigon wrote:Good job, as always. So, what's next? Time to simulate those peripherals? :)
Byuu posted in the FAQ thread explaining why this is not happening.
Verdauga Greeneyes
Regular
Posts: 347
Joined: Tue Mar 07, 2006 10:32 am
Location: The Netherlands

Post by Verdauga Greeneyes »

Time for sPPU? Though I imagine you might want to get those div/mul delays and the joypad polling thing emulated first..
Hatsuyuki
New Member
Posts: 8
Joined: Wed May 21, 2008 2:02 pm
Location: St. Petersburg, Russia

Post by Hatsuyuki »

byuu, big thanks for fixing the problem with loading ROMs under non-English Windows :D
Some things can only be seen when you've lost everything
byuu

Post by byuu »

Half of our state is out of power right now, I actually only just managed to post v036 before it happened. They're saying up to seven days. Have intermittent wi-fi access with the laptop, but I'm dead in the water for anything else. Including posting a Hungarian localization I said I would :(
byuu, big thanks for fixing the problem with loading ROMs under non-English Windows
Sure, thanks for being so patient with me on that.

So, aside from ZIP/JMA support, it works as expected? Good to hear. Was afraid I'd miss something. Tons and tons of file access code all over the emulator.

Also used the file class, which has built-in buffering that's missing on Windows w/fgetc(), etc. I guess bsnes is a good place to stress test that library, heheh.
Though I imagine you might want to get those div/mul delays and the joypad polling thing emulated first..
Yeah. Want to get those two first.
byuu

Post by byuu »

I'll carry this discussion over here.
Nach wrote:I could easily add for Windows to use a wchar_t * / wstring open call which wraps to using _wopen() to get an fd to build upon.

The issue is how to make it work on all kinds of other OSs, and having it wrap around other source classes when not using libstdc++.
Given that both Linux and OS X support UTF-8 filenames, I say we just add #ifdef _WIN32 wrappers around each file open call. For fopen(), use MultiByteToWideChar + _wfopen(), for std::ifstream, use the same as fopen(), and pass the handle to the ifstream constructor.

No reason to explicity support Windows' broken Unicode system. If they need to, they can convert UTF-16 to UTF-8 with WideCharToMultiByte before invoking the libraries.

Of course, the thing is we need to update both zlib and libjma to do this, and I have a feeling updating zlib won't be as fun. To use WCTMB and MBTWC we will have to include the Windows header, which will hit compile time quite a bit.

Maybe I should write a simple convertor for UTF-8<>UTF-16. It doesn't appear to be too complicated ... certainly nothing worth including a big time conversion library over.
tetsuo55
Regular
Posts: 307
Joined: Sat Mar 04, 2006 3:17 pm

Post by tetsuo55 »

Who is wrong here
1)zlib and jmalib using suboptimal UTF stuff
OR
2)Bsnes abusing these libraries

In case 1 the libraries need fixing, as this will help everyone, in case 2 bsnes should have the converter built in.
byuu

Post by byuu »

The answer is, of course,

3) Microsoft was stupid enough to completely duplicate ~80% of their API to take UTF-16 strings, when they could've used UTF-8 with no changes to the underlying API. I know, they were worried about older multi-lingual OSes using non-Unicode codepages. Still a really bad move, NT would've been a good break to force UTF-8.

The best part is that UTF-16 doesn't even allow direct O(1) indexing, as there are extended symbols requiring more than 16-bits to encode.

---

Anyway, the libraries both take const char* filenames, so there's no way I can encode the names for them to work at this time. But it's really not their fault when const char* is good enough for every other OS out there.
_willow_
Hazed
Posts: 51
Joined: Mon Dec 24, 2007 2:03 am
Location: Russia
Contact:

Post by _willow_ »

Well, i do use English, German, Japanese and Russian symbols in filenames. Sometimes they got mixed in to single string, esp multimedia files to note the ones. Those UTF-8 nationalist colored programs always works in twists for me. :(. Most time i'm forced to use pure English and Russian. Really i don't care of bsnes is non unicode program just because most emulators remains non unicode too. It's not comfortable sometimes but i can live with it, np.
[url=http://quake2xp.quakedev.com]quake2xp[/url] audio engineer
byuu

Post by byuu »

Really i don't care of bsnes is non unicode program just because most emulators remains non unicode too. It's not comfortable sometimes but i can live with it, np.
Yeah, noticed that myself when I made the Japanese test profile. Firefox wouldn't start at all, Winamp just kept looping through the setup, but would never actually open or play music. Winrar flat out ignored anything with Unicode in it, and spit out ???s all over. Even notepad2 wouldn't open any text files.

The only things that worked were the ones that came with Windows. Literally nothing else. It was completely unusable like that.
tetsuo55
Regular
Posts: 307
Joined: Sat Mar 04, 2006 3:17 pm

Post by tetsuo55 »

byuu wrote:
Really i don't care of bsnes is non unicode program just because most emulators remains non unicode too. It's not comfortable sometimes but i can live with it, np.
Yeah, noticed that myself when I made the Japanese test profile. Firefox wouldn't start at all, Winamp just kept looping through the setup, but would never actually open or play music. Winrar flat out ignored anything with Unicode in it, and spit out ???s all over. Even notepad2 wouldn't open any text files.

The only things that worked were the ones that came with Windows. Literally nothing else. It was completely unusable like that.
Yep thats how it is basically.
I have about 2000 files named with different pagetables and they all show up as "$%^TYG$"

But when i install chinese windows in vmware mount that same directory and browse the files they all work.
Thristian
Hazed
Posts: 76
Joined: Tue Feb 07, 2006 11:02 am

Post by Thristian »

byuu wrote:The answer is, of course,

3) Microsoft was stupid enough to completely duplicate ~80% of their API to take UTF-16 strings, when they could've used UTF-8 with no changes to the underlying API. I know, they were worried about older multi-lingual OSes using non-Unicode codepages. Still a really bad move, NT would've been a good break to force UTF-8.
NT was their good break to force Unicode instead of having different code-pages everywhere. According to Wikipedia, the first version of Unicode was released in October 1991 and Windows NT 3.1 was released in July 1993. UTF-8 was first publically announced in January 1993, but it wasn't invented by the official Unicode people and wasn't included in the Unicode standard until Unicode 2.0 in 1996.

Microsoft *could* have maybe pushed the release of NT back a year or so if they'd paid attention to the introduction of UTF-8 and realised how much better this third-party encoding would be than any of the official Unicode encodings, but they didn't.
The best part is that UTF-16 doesn't even allow direct O(1) indexing, as there are extended symbols requiring more than 16-bits to encode.
Up until Unicode 3.0 (1999), you *could* fit every Unicode character into a 16-bit integer, so it wasn't such a crazy thing to do in 1991.
tetsuo55
Regular
Posts: 307
Joined: Sat Mar 04, 2006 3:17 pm

Post by tetsuo55 »

So if we had a choice.

Which version of UTF is the best one?
creaothceann
Seen it all
Posts: 2302
Joined: Mon Jan 03, 2005 5:04 pm
Location: Germany
Contact:

Post by creaothceann »

UTF-8.
vSNES | Delphi 10 BPLs
bsnes launcher with recent files list
tetsuo55
Regular
Posts: 307
Joined: Sat Mar 04, 2006 3:17 pm

Post by tetsuo55 »

Does anyone know of a program that can convert "any codepage" to unicode? (So i can fix my thousands of files that display garbage)
byuu

Post by byuu »

Posted a new WIP.

The biggest change was that I rewrote nearly all of the cheat code system, so heavy testing on that would be appreciated.

Someone was mentioning over at Snes9X that it was limited to 300 cheats or something, so someone bumped it to 3,000. Not to be outdone (v036 is limited to 1,024), I vectorized the cheat table, meaning you can have infinite cheats now (limited only to available memory.) Actually cleans up the code quite a bit, too. Removed all the ugly strlcpy() stuff, the limitations on description text length, etc.

Looks like I had a bug with deleting codes, too. I wasn't copying the actual cheat codes. That would corrupt the descriptions on every code after the one you deleted, I think. Strange nobody caught that.

I also cleaned up the OBC-1 code, and added a "Fullscreen" checkbox after "Correct Aspect Ratio". Sorry for the delay with that, FitzRoy. Hopefully the checkbox is good enough for now, as I can't change the text to "Switch to ..." just yet.
tetsuo55
Regular
Posts: 307
Joined: Sat Mar 04, 2006 3:17 pm

Post by tetsuo55 »

Does the cleanup of the OBC-1 code have any effect on its speed or accuracy?

Cool work on the cheat-engine. I never use cheats so i wouldn't have run into that problem
FitzRoy
Veteran
Posts: 861
Joined: Wed Aug 04, 2004 5:43 pm
Location: Sloop

Post by FitzRoy »

byuu wrote:Sorry for the delay with that, FitzRoy. Hopefully the checkbox is good enough for now, as I can't change the text to "Switch to ..." just yet.
Thanks, however, I think modes should operate as a checkmark choice with both listed a la PAL/NTSC. I don't like menu actions that don't take checkmarks, because they look exactly like an unchecked item (I consider this an oversight in modern gui standards). And to have to dynamically change what it says based on the mode... seems unnecessary.

By the way, do you think you could make the emulation speed settings universal rather than mode-specific? I'm slightly annoyed at having to enable/disable them in both modes as they're not the kind of things I would want in one but not the other.
FitzRoy
Veteran
Posts: 861
Joined: Wed Aug 04, 2004 5:43 pm
Location: Sloop

Post by FitzRoy »

Thinking about this some more, it doesn't really have anything to do with video settings, it's just the way the program is framing it. This looks like an appropriate time to add a new category:

Display Mode >
[x] Windowed
[ ] Fullscreen
---------------
Video Resolution >
[ ] Scale 1x
...

Everything under Video Mode deals with resolution and how it is stretched, so I think renaming that to avoid two modes would be nicer, too.
King Of Chaos
Trooper
Posts: 394
Joined: Mon Feb 20, 2006 3:11 am
Location: Space

Post by King Of Chaos »

So far so good cheat wise. Testing several games and no problems whatsoever. :)
[url=http://www.eidolons-inn.net/tiki-index.php?page=Kega]Kega Fusion Supporter[/url] | [url=http://byuu.cinnamonpirate.com/]bsnes Supporter[/url] | [url=http://aamirm.hacking-cult.org/]Regen Supporter[/url]
ssbastos

Post by ssbastos »

Hello everyone,
Just thought I should mention that the video frameskip option in the windows port does not save when we set it. So that we have to set it every time we start the emu.
tetsuo55
Regular
Posts: 307
Joined: Sat Mar 04, 2006 3:17 pm

Post by tetsuo55 »

ssbastos wrote:Hello everyone,
Just thought I should mention that the video frameskip option in the windows port does not save when we set it. So that we have to set it every time we start the emu.
Yeah, i didnt report it because it thought it might be on purpose
Locked