I fully agree that it is easier to analyse a TSX file than a wav file. But it doesn't mean that emulators would have to support the format. You could use the tools that can create TSX files to analyse all you like. You don't need an emulator to be able to run TSX files in order to analyse them. (This is one of the points in the discussion thread: the analysis tools to create TSX files can be useful to analyse the software.)
Hi, first I wanted to introduce myself a bit to put myself in context, I am the author of the openMSX_TSXadv fork,
which is based on Natalia's fork and Openmsx version 16, I have also helped add support to the PlayZX app
and right now I have almost ready the support for MAME; I also help in the dumps of "tsx.eslamejor", with which
I know the format quite well and I am open to help with its implementation.
Make things clear in case
there would be doubts about it, a TSX is nothing more than a TZX with a
block that supports Kansas Standard, which includes the bytes of the MSX block
and information to recreate the original wave.
It has been commented that when it is transformed from WAV to TSX information is lost,
and unless the block contains some type of protection that is not representable in TSX,
I do not see what information is lost with what I would beg you to clarify this point and thus get me out of my mistake.
As a guideline, of more than 500 tapes that we have preserved working (without patching the
load), there are less than 10 tapes that for now we have not been able to convert due to carrying blocks
special type protection or audio, I consider that it is a very small percentage not to consider
TSX as a faithful preservation format, in CAS it would have been necessary to patch a fairly large percentage.
I consider that the wav generated by a TSX is more faithful to the original master (as long as it does not carry
music or similar), than a wav extracted from a tape, because the size of the pulses depends on the
state of the media and the player, in fact sometimes I have needed at least 5 dumps to
get a valid wav.
Regarding the issue of other formats, if it is for use in emulator or real machines with MFR, logically for speed I would use
CAS because it loads directly into memory, if it exists, because as already mentioned, to
for special loads to work, the original program has to be patched; and the TSX for use in real machines
that can only load by cassette port.
Also an important issue is that in a TSX you can use blocks to indicate where each game is on multigame tapes, or where the games begin.
phases of a multi-load, to go directly without having to go through the previous ones, neither in WAV nor in CAS that information can be saved; on
TSXduino is implementing the use of these blocks, and in my openMSX_TSXadv as well.
In short, in my opinion, I consider the TSX format valid for preservation and I would like you to give me contrary arguments based on
real tapes to see the possibility of their preservation, and I think that OpenMSX should give it official support (the code is available, you just have to adapt it to how you like it)
since it is a format with more than 500 tapes ("Space Harrier II" and "The Race" only available in this format), and that combines the comfort of CAS work and the fidelity of WAV.
And that said, if anyone needs help to implement the format do not hesitate to contact me at isramula@gmail.com
I fully agree that it is easier to analyse a TSX file than a wav file. But it doesn't mean that emulators would have to support the format. You could use the tools that can create TSX files to analyse all you like. You don't need an emulator to be able to run TSX files in order to analyse them. (This is one of the points in the discussion thread: the analysis tools to create TSX files can be useful to analyse the software.)
Sometimes it is necessary to have both, the encoded data and the code at run time, for example, to decode "much faster" data encoded using a proprietary format.
Those tools that you mentioned will only convert from WAV to TSX, including proprietary data blocks, sure, but such blocks can be encoded. If a software analyst is interested in, for example, understanding how the data is encoded, it will be necessary to run the non-proprietary code as well. In that case, the analysis will be more agile if the code is also embedded within the same TSX file that can be run on an emulator.
I am going to present a real case scenario from my own experience. Let's say that as part of a block we have (in a TSX file) the following bytes that are encoded as 47h, 19h, 65h. Then, we want to find out how the real data is encoded as those bytes. However, the software is protected, it modifies itself, and in addition it uses hooks to complicate the analysis at run time, so that when we debug a part of the code, a different one is suddenly executed. Good luck to find and understand the routine that decodes the data. Not really impossible, but it could be quite complex.
However, one out-of-the-box solution is to apply data comparison if we have a TSX file. For example, then we find a memory buffer containing A3h, 8Ch, B2h. Initially, it seems that there is not much in common between, 47h, 19h, 65h and A3h, 8Ch, B2h in hexadecimal format. But if we convert this into binary data, we have:
01000111, 00011001, 01100101 (Raw data in TSX file)
10100011, 10001100, 10110010 (Raw data in buffer)
Now, by data comparison, we can quickly realise that the data is encoded using a Rotate Left Circular (RLC). From that point, we know that we need to find the code that applies the inverse function (RRC) to be on the right track.
I did what I have described above in three different ways:
1) WAV file.
2) WAV file for dynamic code analysis and external tools for extraction of encoded data blocks.
3) Code and encoded data blocks in TSX file.
By far, the third option was the most agile and fastest way for software analysis. Therefore, I can see the benefits if TSX could be run on an emulator.
Last but not least, I have nothing to lose if TSX is never supported, and I have nothing to win if TSX is ever supported, as I am not who defined the format. What is more, I was once even against TSX, so my position is totally neutral in terms of sentimental or emotional attachment.
I agree that my example is punctual, but it was indeed a real case scenario that departs from preservation, which is one of the most common and controversial arguments around TSX. So, I thought that it would be valuable to have a word and share my experience using a different example, because I really saw in it the potential of TSX. Otherwise, I would not say anything.
It has been commented that when it is transformed from WAV to TSX information is lost,
and unless the block contains some type of protection that is not representable in TSX,
I do not see what information is lost with what I would beg you to clarify this point and thus get me out of my mistake.
The tool that converts the WAV to TSX has to make some interpretation of the waveform. This is a non-trivial process. It is possible that during this conversion mistakes are introduced. Such mistakes may sometimes only be discovered weeks or months later (e.g. it only goes wrong when you reach level N in the game).
When you only have the TSX file available you can only guess what the correct information should have been. If you do still have the original WAV, then you can (manually) look at the part of the waveform that was interpreted wrongly and then make a much more educated decision for how to correct the mistake. For example: maybe the low pass filter needs to be more or less aggressive, ...
I believe this is what 'bore' meant in his earlier post. @bore: correct?
There's also other information that's lost. For example TSX assumes the waveform is 'ideal':
- The timing: all pulse-durations are an exact multiple of some base-duration.
- The shape of the waveform: perfect square waves.
An actual tape can have all kinds of deviations from this 'ideal' representation. Some examples:
- The actual waveform is not a perfect square wave (in some cases this difference between 'square' and more 'rounded' waveform can even be audible).
- It's not uncommon that there are (very) slight timing difference between the start and the end of the tape. In principle a data-block in TSX can encode a non-standard timing, but it does assume this same timing for the whole duration of that block.
For some people these differences may be totally irrelevant. For other people it might be an important aspect for 100% preservation.
So in summary:
- If you only care about the digital-information represented by the TSX file (and assuming the TSX generation was error-free), then sure: TSX hasn't 'lost' that information.
- But there's is other information that does get lost. In particular the ability to verify the conversion.
Tell you that although the tool has not caused a problem in the conversion, it is never taken for granted
good each block until it has been compared at the CRC level with the other side, another copy or if
there is nothing else, with the existing CAS; if all the blocks match, the TSX is validated, if
some differ, the differences are contrasted and the wave is reviewed to try to see the value
matches the other copy. Besides, the Spectrum blocks have their own checksum that
the tool checks. If it is not possible to ensure that the data is correct, it is not published,
waiting for another copy.
The conversion is not only the automatic process, it takes after a revision work, verification
, test, and even repair the wave if it is damaged.
The tool warns when it finds an anomaly during the conversion and that is when it enters
the human eye to see what is happening, for example the "hobbit" has non-standard parts within itself
block, which have had to be recreated using pulses and special blocks in a traditional way.
Regarding the waveform, I think it depends on the player and the state of the tape
When the tape is good, my bitcorder reproduces a perfect square wave.
Logically, the WAVS obtained are not deleted in case they are necessary in the future, but a
a normal user will not store them with the size they are.
I'm sure you take great care when creating TSX file. I just wanted to give some examples of what is meant with "information is lost".
I believe this is what 'bore' meant in his earlier post. @bore: correct?
Correct.
I don't really care much about the average emulator user in this case, for them TSX might be the best choice.
What bugged me was using data preservation as an excuse for supporting the format, it just doesn't make sense to me.
TSX might be good for some cases of data analysis but will completely prevent others.
When talking about data preservation I'm thinking of a raw capture so that it is possible to not only look at the intended data but also partially overwritten data before and after the intended data that may or may not be for MSX.
Even a partial audio recording could put an upper boundary for the date of the software.
There is also the possibility that the MSX data is weighted from what was previously written on the tape.
Then there is of course your more common example where the tape is noisy and manual evaluation needs to be done.
A tool to clean up WAV files can output a heatmap that indicates how certain it is that each bit was correct to simplify manual analysis.
Even better would be to do multiple recordings with different head adjustments to get a 2D map of the tape content.
Storage could be done in PNG, multi channel WAV or whatever format you fancy.
I think what bugs me the most is that I just don't get the arguments people are making for TSX.
Is there some other popular MSX emulator that exclusively uses TSX or why do people even have MSX software in TSX format?
If I were to make tape captures it wouldn't be my go-to format unless I had a specific reason to.
I think what bugs me the most is that I just don't get the arguments people are making for TSX.
Is there some other popular MSX emulator that exclusively uses TSX or why do people even have MSX software in TSX format?
I think rather than some other popular MSX emulator, it is some other popular system. Because the Spectrum had a lot of tape turbo loaders and copy protections and what not, their TAP format was incompatible with a lot of games and TZX was an important development for that system’s community. (I’m not part of the Spectrum community and don’t know their history but this is what I surmise.)
But for MSX we have CAS which covers the majority of the software, and we can use WAV if it is not sufficient. Because there are relatively fewer games for which CAS is insufficient, the need for another format is less.
So I think that while TZX has a lot of value attached to it in the Spectrum community, this enthusiasm did not automatically carry over to the MSX community. Which is why the reception is lukewarm and the format is looked at critically, and not adopted as a heaven-sent that we must have no matter what.
If I were Marty, I'd use a disk version of the game and cheat that. Or use the emulator to poke the lives variable.
Well, Marty only has a license for his tape copy, and he feels that it's wrong to look for a disk version. He only knows what bytes to change, but not how to use a debugger to do a search. That's too complex, much more than a hex editor, which is pretty much like a normal text editor.
I do think you overestimate Marty's capabilities if you expect him to patch the TSX file.
Why? It's just a search and replace, he knows how to do that with a text editor and he knows how to do that with a hex editor. They're not that different after all.
Did people ever patch TSX files, in the non-Kansas format part?
I don't know if anyone ever did before me, but here's a patch that does exactly that. The patch affects only the last block, which is in Spectrum standard format (block type 10; the KCS block type is 4B).
It's an IPS patch to get immunity in Avenger. It applies to the TSX file with CRC32 0005a67d, MD5 c085e2699a61ad1a20c0312affa1250d. I can't link to the TSX for obvious reasons, but I can certainly link to the IPS patch:
http://www.formauri.es/personal/pgimeno/temp/Avenger_Immunit...
Just for fun, I generated an IPS that applies to the conversion of the TSX to WAV. It's 326,732 bytes, compared to the 20 bytes that the above IPS takes.
I do think you overestimate Marty's capabilities if you expect him to patch the TSX file.
Why? It's just a search and replace, he knows how to do that with a text editor and he knows how to do that with a hex editor. They're not that different after all.
Because in your example he started out with an original tape.
To get to the point where he would want to apply a TSX specific patch he would have to record the tape (most likely to WAV) and then convert it to TSX.
At that point he has already passed all the hurdles that makes WAV so unfeasible.
If hypothetical Marty manages to get that far, converting the TSX back to WAV after applying the patch is an insignificant step compared to the things he already went through.