NYYRIKKI's MSX ROM loader for SVI

NYYRIKKI's MSX ROM loader for SVI

by NYYRIKKI on 01-06-2014, 15:23
Topic: MSX Related
Languages:

Before the MSX standard was established Spectravideo made a few computer models (SVI-318 and SVI-328) that were used as a base for the MSX standard. They practically had all same components that MSX had, but they were connected to each other in a different way.

Spectravideo probably thought that MSX standard would end up being much closer to their computers, so they were very proud to tell the world that they were on the cutting edge of defining a new world standard and soon the world would be using similar computers. What I learned only lately was that they actually openly claimed that the SVI-328 would be a MSX compatible computer. However the truth in the end was quite different. SVI had different memory handling, a different BIOS, a different version of BASIC and so on. SVI not only used different I/O ports, but it also used different I/O ports for reading and writing, while MSX used less number of ports. Also practically all connectors were different, including the tape drive, the joystick ports, the printer port and the cartridge port.

The big mouth of Spectravideo's marketing department probably made very many customers angry as they had spent a lot of money for computer that ended up being totally incompatible with the MSX-standard. Later Spectravideo tried to fix things by releasing the SVI-606 MSX Game adapter. This was however more like a joke. You needed MSX-joysticks and a cassette drive to use it and it had its own keyboard that looked like one from the Casio wrist watch. Letters were physically placed in alphabetical order and it didn't even work with too much software.

Hobbyists ported some commercial MSX games to SVI and they even ported MSX-BASIC. But in the end users could still only enjoy an extremely limited number of MSX software products.

This story made me feel bad, so I thought what could I do to improve these betrayed SVI users' lives? First I ported Chuck Yeager's Advanced Flight Trainer and Red Zone to SVI, but very soon after starting I realized that this is boring and quite a robotic job. This approach would make me die of old age before I could get anything remarkable done.

Therefore I ended up to write "MSX ROM loader for SVI". In the hart of this project there is very custom version of MSX C-BIOS that is adapted to run on SVI and to translate SVI hardware for MSX programs. The other important module of this program is what I tend to call "Self aware, self modifying artificial intelligence." or simply just NYYRIKKI emulator :) This part of the code is responsible for separating program code from other type of data (such as music or graphics) and doing the programming effort to port the code from MSX to SVI. Most of the patching is done before game is started and rest of the routines are patched during game time.

With this Windows program most of the MSX 8/16/32KB ROM games are now playable on SVI. The program does not however have MSX-BASIC included, so for BASIC games you still need the old MSX-BASIC emulator.

Now that the ground work is done, maybe in the future we can see our favorite MSX-games running on other platforms as well... you never know...

Note that this tool only runs on Windows at this time.

Related link: NYYRIKKI's MSX ROM loader for SVI

Comments (33)

By Prodatron

Paragon (1857)

Prodatron's picture

01-06-2014, 18:13

Great that you implemented and finished this crazy project! I am still so impressed by this "self aware, self modifying artificial intelligence", which separates program code from data automatically. Hard to believe, that it is possible!

By ToriHino

Paladin (925)

ToriHino's picture

01-06-2014, 18:39

Who would have thought that we ever would get a NYYRIKKI emulator Smile But impressive work, and finally some light for this forgotten generation of early SVI adaptors...

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

01-06-2014, 18:56

Few notes:
- I just updated it to version 1.1 This fixes at least sound issues in games like Deep Dungeon and La Corona Encantada. (Download link is same)

- It is optional to use the included Windows script to create CAS-files. In readme.txt there are instructions how to use the ROM loader without it.

- At least Activision games (like Ghostbusters or Pitfall2) have noice behind the game when running on BlueMSX... This is caused by incorrect emulation of PPI. Should work ok on real hardware.

- This program does not try to go around copy protections, so for example Konami games need to be cracked before they can be used with this loader... Finding cracked versions should not be hard though Smile

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

01-06-2014, 18:59

Thank you for your kind words... Indeed I'm also quite impressed, how well this program actually works. Most of the games don't have any problems! Seems that virtual me is doing a good work, LOL Smile

By Jorito

Mr. Ambassadors (1803)

Jorito's picture

01-06-2014, 19:26

Thanks for the updated v1.1; I just added the new version to the downloads database (the newpost points to our downloads database in stead of your site, NYYRIKKI).

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

01-06-2014, 21:19

Ah, I see... I thought the downloads database was for MSX programs only. Smile

By hbarcellos

Hero (649)

hbarcellos's picture

01-06-2014, 22:20

Hey, amazing!
What do you use to separate program code from data? Do you try to check with some kind of matrix near bytes to see if it's something that would make any sense for coding? in case of packed stuff (if any) does it work?

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

01-06-2014, 23:15

Yes... The routine works so that you point a byte with HL and call the detection routine. It will simply return "Yes, this is likely Z80-code" or "No, I don't think I should execute this" in a C-flag.

The actual detection is combination of tests... As we all know you can't look a byte and say what it is meant for, but we need to take a look at a big picture and what context the byte is. For humans this is quite easy, so I just tried to copy my own thinking process in to the code.

To evaluate the code I count "likely events" and "unlikely events" and in the end number of these events define the result. First I take a few dozen bytes back from the location and step trough the code until I'm back in the address. If I accidentally jump over the command byte that I was expecting to patch then this is unlikely event... As you might have noticed programs tend to start stepping correctly very fast even if you start from wrong byte because Z80-commands have different lengths...

After this I actually read what the code does if it is executed from the interesting byte forward... If I can see for example command series like: LD L,D / XOR A / LD HL,#6512 then I can say that this is very unlikely to be Z80-code as L-register is overwritten before it is used to anything. -> Unlikely event. If I see call to address range #200-#3FFF -> Unlikely event. If I see program trying to write it's own ROM or BIOS -> Unlikely event. If I see command like LD E,E -> Unlikely event. If I see call to address where previous command in memory is RET or JP -> Likely event.

I think you get the idea... If the code is packed then there is no much that I can do... How ever this is quite unlikely as most ROM's expect MSX to have only 16KB RAM and it is usually used to store variables. If something is packed in ROM it is more likely graphics. For sure there is always the one program that proves that I'm wrong (hello Beepertron) but mostly my logic seems to work. From the start I understood that this kind of program can't be 100% but I focused to things that are most likely/unlikely and in the end putting these things together made the end result surprisingly rock solid.

By hbarcellos

Hero (649)

hbarcellos's picture

02-06-2014, 02:24

really really cool.
Congratulations Nyyrikki
I've been to several of those "thinking process" while I was analyzing the COLECO ROMS trying to figure out if those we're code or data and in a similar way, after a while converting roms become a "boring and robotic job"! Smile

KUDOS!

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

02-06-2014, 05:36

Thank you. The good thing in this project is that there was no boring moments while working on it. Although the code separation was very interesting problem the actual patching was just as interesting...

I give you an example from "Gommy medieval defender"... The music player in the game looks like this:

	XOR A
	LD C,#A0
	LD HL,#D615
LOOP:	OUT (C),A
	INC C
	OUTI
	DEC C
	INC A
	CP #D
	JR NZ,LOOP

After my program has done patching the routine it looks like:

	XOR A
	LD C,#8B
	LD HL,#D615
LOOP:	OUT (#88),A
	INC C
	OUTI
	DEC C
	INC A
	CP #D
	JR NZ,LOOP

Making that to happen automatically also requires quite a fuzzy logic. Smile

By ARTRAG

Enlighted (6977)

ARTRAG's picture

02-06-2014, 09:27

Awesome ! My greetings, very impressive resut!
Ps
Why not using the NYYRIKKI emulator with hbarcellos sn76489 simulator to port all coleco rom to msx?

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

02-06-2014, 11:41

Hmm... That is not a bad idea...

By PingPong

Enlighted (4155)

PingPong's picture

02-06-2014, 13:14

NYYRIKKI wrote:

Yes... The routine works so that you point a byte with HL and call the detection routine. It will simply return "Yes, this is likely Z80-code" or "No, I don't think I should execute this" in a C-flag.

an heuristic approach i see

By hbarcellos

Hero (649)

hbarcellos's picture

02-06-2014, 13:27

NYYRIKKI wrote:

Hmm... That is not a bad idea...

Not at all, and quite possible, but we would need to think on some changes for Interrupts and sound.
On sound, for instance, as the chip is not the same, instead of an out, we need to call a routine. As that uses more bytes, most of the time I need to "transplant" part of the original code to some specific calls.
Sometimes, I can use an RST, but some games use them all. Maybe with RST 0...
About BIOS, I'm more flexible than the majority of other users, so, the original copyrighted BIOS is included into the ROM file. There's no C-BIOS for Coleco as far as I know.

Last, but not least, maybe it's a good idea to improve the sound emulator. I already "open-sourced" (posted full source code) of the latest one here. It's like a proof of concept that I did in some hours to test noise emulation.

here: http://www.msx.org/forum/msx-talk/software/sn76489-simulator...

Maybe, you two guys would like to take a look.

The original one is way faster than that, but lacks noise completely.

By hbarcellos

Hero (649)

hbarcellos's picture

02-06-2014, 13:32

...NYYRIKKI, you've got mail.

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

02-06-2014, 15:15

hbarcellos wrote:

Not at all, and quite possible, but we would need to think on some changes for Interrupts and sound.
On sound, for instance, as the chip is not the same, instead of an out, we need to call a routine. As that uses more bytes, most of the time I need to "transplant" part of the original code to some specific calls.

That is not a very big issue... The AI has enough information of Z80 internal working that it can make enough room without breaking the code as long as we just have enough RAM space to move stuff into. I think I only need to add some special logic in case OUT is followed by JR / DJNZ

Quote:

Sometimes, I can use an RST, but some games use them all. Maybe with RST 0...

Yes, this is the approach I also used for MSX ROM loader... As C-BIOS does not have BASIC I used RST #0, #8 and #18 for better purposes Smile (In, Out & other I/O) This way I didn't need to start moving things around. I think RST #0 would be enough to get all functionality in, but it would require quite a much translating things back and forth with tables... Maybe in this case "transplant" is better idea anyway...

Quote:

About BIOS, I'm more flexible than the majority of other users, so, the original copyrighted BIOS is included into the ROM file. There's no C-BIOS for Coleco as far as I know.

No, but luckily I found disassembled version of the ROM that I think we could use.

Quote:

Last, but not least, maybe it's a good idea to improve the sound emulator. I already "open-sourced" (posted full source code) of the latest one here. It's like a proof of concept that I did in some hours to test noise emulation.

here: http://www.msx.org/forum/msx-talk/software/sn76489-simulator...

Maybe, you two guys would like to take a look.

The original one is way faster than that, but lacks noise completely.

My knowledge of Colecovision and it's sound chip at the moment is close to 0, so don't expect much help from me. How ever this is interesting stuff and maybe together we could make something out of this... Too bad that the time I have for these MSX (and related) projects is so limited... Please don't expect me jumping right in to this new project. I have also lots of other things WIP.

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

02-06-2014, 19:20

NYYRIKKI wrote:

No, but luckily I found disassembled version of the ROM that I think we could use.

Hmm... Looking now the memory map, it seems that memory area #2000-#5FFF could be safely used for all kind of support routines... Therefore it seems more vice that I just don't touch the BIOS at all but let the AI handle also that... Or are there many games that require 24KB RAM? I would estimate that the support routines need about 4KB storage space...

By hbarcellos

Hero (649)

hbarcellos's picture

02-06-2014, 19:44

actually I already have the BIOS fully converted and it goes from 0 to 1FFF. You can check that on the files I've sent you.
Most of the "extras" I had to add, like joystick emulator and sound chip emulator, are already placed after $2000.
I boot @ 4000 with a small routine that copies (sometimes depack) the bios and extras to 0 and then I jump to that area to fix the game on the right position 8000 and beyond. Coleco RAM is just 1k, at page 1 (don't remember exactly where) but it's mirrored somewhere else.

Have you got the email?

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

02-06-2014, 20:20

Yes, I got your mail, I just didn't actually check the sources yet at all... Now I feel a bit stupid... Sounds like you have already done most of the hard work in a way that it is reusable. Now I'll go and learn...

By hbarcellos

Hero (649)

hbarcellos's picture

03-06-2014, 03:12

c'mon! Smile
On a small community like ours, if we don't join forces, we'll soon disappear...
Finnish parties like these are still happening?
Amazing Skis BTW!

By Maggoo

Paragon (1218)

Maggoo's picture

03-06-2014, 08:19

Brilliant work Nyyrikkii. I'll dust of my 328 and give it a try!

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

03-06-2014, 23:00

@hbarcellos I've looked what you have done and I must say that you have already made almost general Colecovision emulator... Seems that practically only the automatic patching is missing... This project should be walk in a park. -> Just plug in AI Smile

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

04-06-2014, 22:28

Link to list of games that has been tested so far... Feel free to fill it, if you like...

By anonymous

incognito ergo sum (116)

anonymous's picture

04-06-2014, 23:37

@Nyyrikki: please consider adding support for the Franky card also. It just needs another address for the VDP, but the vdp also has the compatible coleco soundchip inside. And thanks for your code, however Dennis is currently to busy to patch the sega romloader. I hope he'll find time later.

By Sander

Founder (1872)

Sander's picture

04-06-2014, 23:38

BTW how does this emulator compare to the msx program loader for SVI from CUC? I got it on tape somewhere.

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

05-06-2014, 18:13

Sander wrote:

@Nyyrikki: please consider adding support for the Franky card also. It just needs another address for the VDP, but the vdp also has the compatible coleco soundchip inside. And thanks for your code, however Dennis is currently to busy to patch the sega romloader. I hope he'll find time later.

Ah, so you do read that mail address Smile In case you get to patch the romloader, please contact me first... The AI is still under development and it has improved quite a bit since I wrote to you...

Sander wrote:

BTW how does this emulator compare to the msx program loader for SVI from CUC? I got it on tape somewhere.

I haven't even heard about that, so I can't really tell... I know only the MSX emulation program by H. Klopper and P.Zevenhoven (Adaption Electronics Nederland) that mostly run only BASIC-programs.

By sd_snatcher

Prophet (3675)

sd_snatcher's picture

24-09-2014, 18:02

Speaking of the SVI-606, has anyone ever happened to open one to see what it has inside?

By encore

Supporter (12)

encore's picture

04-06-2017, 23:08

Thanks for making this NYYRIKKI. Smile I recently acquired myself a Spectravideo SV328 (I had one for about a year when I was 6-7 years old) and it's great to not just be limited to SV-specific software (which there isn't terrible much of).

- I understand that it converts an MSX-ROM to a SVI-ROM (using lots of magic). I might be able to try running the converted files on a real PCB in my SV-328 in the future and I can experiment it with already now in openMSX, but is it somehow possible to add a cassette loader and be able to load them this way as well? I think I saw this mentioned somewhere, but wasn't sure if there's an automated way to do that.

- When I googled on this I saw that you posted a new version (MSX_ROM-load_for_SVI_v1.3.zip) on a Facebook group. I don't have Facebook anymore however and I couldn't get the link to download, could you maybe post it in here as well?

(My first post here, so hi!)

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

06-06-2017, 00:09

Hi!

encore wrote:

- I understand that it converts an MSX-ROM to a SVI-ROM (using lots of magic). I might be able to try running the converted files on a real PCB in my SV-328 in the future and I can experiment it with already now in openMSX, but is it somehow possible to add a cassette loader and be able to load them this way as well? I think I saw this mentioned somewhere, but wasn't sure if there's an automated way to do that.

Yes, the packet comes with "svitools" that can be used to transfer the resulting CAS-file in to audio file that can be recorded to cassette or played back from phone, PC or other similar device in order to load the games on real SVI-328. In emulation world openMSX can load natively the converted audio file, but AFAIK latest development version comes with a TCL script that allows loading fast directly from CAS-file as well, but this feature needs to be separately activated... BlueMSX can only load the CAS-file version.

Latest version or the converter comes also with alternative Windows script that creates a ROM-file for SVI-328... It is a bit more limited as bigger games may not fit in to the available ROM space, but it makes it possible to create a real cartridge games if someone wants to. Naturally the resulting ROM-files work on both emulators as well.

Quote:

- When I googled on this I saw that you posted a new version (MSX_ROM-load_for_SVI_v1.3.zip) on a Facebook group. I don't have Facebook anymore however and I couldn't get the link to download, could you maybe post it in here as well?

Fair enough... I uploaded it now here.

By encore

Supporter (12)

encore's picture

22-08-2021, 10:42

Hello NYYRIKKI. While searching online a while ago I think I saw a mention of a newer version of this utility, now I can't find where I saw it. Could you perhaps post it here and describe what's has been updated? Thanks in advance. Smile

By NYYRIKKI

Enlighted (6092)

NYYRIKKI's picture

22-08-2021, 22:43

Sure... Current version is 1.5.

Here are the changes since 1.3:
- Fixed a crash that may trigger when loading upper half of packed 32kB ROM
- Fixed a bug that kept cassette motor spinning (caused by PPI reset)
- Improved the automatic conversion script to support wildcards, long filenames and output directory
- Combined CAS and ROM conversion to same end user script
- There is 2-button support in case SEGA Master System, Kemston or Atari ST controller is attached to SVI joystick port. In case Sega Megadrive/Genesis joypad is used joypad button B is MSX joystick button A and joypad button C is MSX joystick button B. Normal SVI joystick will support only A-button. (Do not connect MSX-joystick, you may damage your machine!)
- Fixed internal keyboard input conflict by replacing RST# 18 with RST #28
- Unsafe MSX PSG usage is now fixed automatically
- Minor tweaks to make nonstandard stuff less likely to cause critical crash
- TOSVIROM.ASM fixed to support 48kB ROMs (out of 32kB MSX ROMs) when needed.
- Some error messages make more sense now

By encore

Supporter (12)

encore's picture

23-08-2021, 23:27

Thanks! Will give it a spin when I find the time Smile

By thalin

Resident (42)

thalin's picture

12-02-2023, 15:03

Tried this on my own Penguin Run game but it did not work. It seems to crash at the point when it starts the game. Is there anything I try to make it work? The same thing happens in both OpenMSX and BlueMSX.