Understanding the HOTDOG files on DVD of California electronics

[bigboss97]

All my experiments were done on a DVD-RW. That's the "beauty" of our player

woid wrote

Have you tired if the player accepts DVD-RWs or DVD+RWs? Time to make my first experiment soon...

[woid]

I have done some updates to Dvd2Midi. The main changes is the format of the outputfiles.

Dvd2Midi.zip

The program now outputs the following 6 files:
.dat: this is a raw dump of the whole song entry (header+lyrics+midi).
.lpd: Lyrics Part Dat. This is a data file that contains the lyrics part of the song entry (first 2 letters of title + Xor mask +  header + lyrics. header = title etc, lyrics = lyrics with timecode and 0/0a byte. Characters have been decoded to ASCII)
.txt: Text file with the ASCII extracted from the .lpd file.
.mpd: Midi Part Dat. This is a raw dump of the midi part of the song entry.
.mid: This is midi part converted into midi file format.
.d2: This file is identical to the .dat file. The .lpd and .mpd files have been converted back to .dat format. The idea is to do this in 2 steps later: First extract the .lpd and .mpd files, then manipulate the files, then convert them back to .dat format and then insert the new .dat file into the HOTDOG00.dat file or a DVD image file.

[bigboss97]

http://www.nabble.com/Re%3A-DVD2MIDI-(fixed-!!!)-p11691536.html

[bigboss97]

I put a cross reference under Facts & Summary to make it easier to be found.
Any update of the tool will go to the new branch.

[bigboss97]

I did a quick change in my mapLyric() for dual-byte language (Chinese):
    if ((i % 4) == 0 || (i % 4) == 1)
    {
      ch = (ch & 0x00FF);
      if (ch==0x0d)
        fprintf(outFile,"\n");
      else if (ch)
        fprintf(outFile,"%c", ch);
    }

It's interesting to see that the first 2 letters (4 bytes) are actually swapped (song 0x03a8):
<D9 D4XD8 BB>30 39 D8 B9 94 62 9E D5 BE B1 AA C9
 B5 B1 0B 14 E6 2B 98 21 DB 64 57 E1 EA C9 F4 E5
 98 56 0B 14 D7 DD B0 96 48 23 58 98 32 3E 59 92
 63 E8 4B 6D 77 50 58 98 01 BD 40 7E 40 00 00 00
<D4 D9>00 00<BB D8>00 00 CA D7 00 00 0D 0A 00 00

Which means if the end of the mask is determined by the first 2 letters then we need to swap them before search.

[bigboss97]

dvd2Midi.zip
An updated version for dual-byte lyrics.

I've added mapLyric2b() and updated the 2nd mask for the 2nd byte, in ReadLyricDat().
I also fixed one in the first byte.
The 2nd bytes in the 2nd mask still look very incomplete. I've only checked one song in Chinese. More work to be done.

[bigboss97]

dvd2midi.zip

Now we got it! The first version of DVD2MIDI (with lyrics).
I'm so happy that I can't wait until I've done more tests with other songs. I only tried it on MORE THAN I CAN SAY. It looks perfect.

Outstanding: It crashes on some Chinese songs (that was before the lyric part was added). It seems that it fails at searching mask.

[woid]

Did you try your changes of key2 on english songs? When I run your version on english songs the lyrics is not correct. So I added a command line flag to main (-a for ASCII mode and -d for dual byte mode) and changed back key2 for ASCII mode. I also changed the 2 versions of key2 to global arrays since it is needed in both ReadLyricDat() and writeDat().
Are you sure about the 0x0a in the second bytes of key2? For english songs the second byte is always 0 or 0x0a. By having 0x0a in the mask, 0 will turn into 0x0a and 0x0a will turn into 0. I removed the 0x0a from the ASCII version of key2 for now.

Here are my updates of Dvd2Midi:
Dvd2Midi.zip

Ps. Why did you change thread? I think its better to keep all updates of the code in one thread and use "Facts and Summary" only when we find out something new.

[bigboss97]

As I mentioned, I only tried MORE THAN I CAN SAY. It seems to be okay with that change. When I changed the 1st byte in key2 I wasn't sure. That's the reason I left the original line in the codes. Certainly, it's good to have a switch for dual-byte.

But I'm quite sure with the 2nd byte because when the mask is set it fixed many spots in the same song. I've also checked a 2nd Chinese song all words were correct.

I diverted the thread because I thought the converter is now usable. I'm always assuming that there could be people finding us anytime and I want to allow them to access the tool without going through all the conversations. Basically, we've got everything. Only the fine tuning is left. The more people use the tool the better it will be. From now on, our discussion will slow down anyway.

I'm going to try more Chinese songs and let you know whether the 2nd bytes in key2 are okay. Maybe key2 is really language dependent??

woid wrote

Did you try your changes of key2 on english songs? When I run your version on english songs the lyrics is not correct. So I added a command line flag to main (-a for ASCII mode and -d for dual byte mode) and changed back key2 for ASCII mode. I also changed the 2 versions of key2 to global arrays since it is needed in both ReadLyricDat() and writeDat().
Are you sure about the 0x0a in the second bytes of key2? For english songs the second byte is always 0 or 0x0a. By having 0x0a in the mask, 0 will turn into 0x0a and 0x0a will turn into 0. I removed the 0x0a from the ASCII version of key2 for now.

Here are my updates of Dvd2Midi:
Dvd2Midi.zip

Ps. Why did you change thread? I think its better to keep all updates of the code in one thread and use "Facts and Summary" only when we find out something new.

[woid]

I have added some more tools for modifing songs and updating image file:

Dvd2Midi.zip

Here is an example of how to use the tools:

Extract song 3df4 from Hotdog file:
$ ./dvd2Midi -d ../../HOTDOG00.DAT 3df4


Add time offset to all timecodes in the lyrics:
$ ./adjustTiming
Usage: ./adjustTiming  <lyrics file (.lpd)> <time offset>

$ ./adjustTiming 3d_f4.lpd -337
offset: -337


Convert lyrics file and midi file back to song file (.dat format):
$ ./makeDat
Usage: ./makeDat [-a|d] <infile.lpd> <infile.mpd> <outfile.dat>

$ ./makeDat -a 3d_f4.lpd 3d_f4.mpd 3d_f4.d3


Update the DVD image file with the new songfile:
WARNING1: the imagefile will be overwritten and changed.
WARNING2: It takes a lot of time to search the whole imagefile, more than 35 minutes on my computer...

$ ./updateDvd
Usage: ./updateDvd  <hotdog00/image file> <song file (.dat)>

$ ./updateDvd ../../MidiDvdVol8.iso 3d_f4.d3
Pattern found at address: b52a93e9, length 100
One unique match found. Pattern length: 100


Unfortunately the experiment was a failure, but I still think the tools may be useful.
The plan was this: Song 3df4 (Moon shadow) the lyrics is delayed about 5 seconds. The music starts immediatly, but the lyrics starts about 5 seconds later. By substracting the delay of the first character from all timecodes the lyrics should also start immediatly and in sync with the music.
Unfortunately the result was same as before. The lyrics is still delayed. Maybe this have to do with the countdown in the beginning...

[bigboss97]

Just a suggestion...
When I was doing my experiments I always use a 2nd copy of the image, i.e. the start of the HD00 file will be always the same for me. Maybe you should allow user entering that address then you calculate from there onward. Then it will be faster. The only thing the user has to do is writing down the start address of HD00 of his own image.

[woid]

I have noticed that in many songs the lyrics and the music is out of sync. So I made a small tool "adjustTiming" to adjust the timing of the lyrics. The lyrics contains timecodes for all letters and the "adjustTiming" tool adds a offset to all timecodes (except timecodes with value 0 which seems to have a special meaning, new line? line feed?).

I have made some experiments with "adjustTiming" on song 3df4 ("Moon shadow") and the result is very confusing. The lyrics on "Moon shadow" is delayed about 5 seconds (timecode of first character: 0x152=338). The music starts immediatly, but the lyrics starts about 5 seconds later.

The first attempt was to substract the delay (timecode) of the first character from all timecodes in the lyrics. My hope was that the lyrics would also start immediatly and in sync with the music.
Unfortunately the result was same as before. The lyrics was still delayed about 5 seconds.
$ ./adjustTiming 3d_f4.lpd -337

Then I made a second experiment. I tired to add the delay of the first character to the timecodes. My theory was that this would double the delay to about 10 seconds. When I did this something strange happend. The tempo of the music was changed to very very slow. The lyrics continued in the original tempo until the middle of the song where the lyrics stopped. The music continued in the same very slow pace.
$ ./adjustTiming 3d_f4.lpd 338

I am confused about this result. It seems to be some dependacy between the lyrics part and the midi part. Maybe there are some syncronisation events embedded in the midi stream? Have you found any strange events in the midistream that could be syncronisation events?

[bigboss97]

Yes, the new line and space always have the time code 0 which caused some troubles in dvd2midi. I also noticed there's still some works in dvd2midi to do.
I'll have a closer look to that phenomena below after I fixed dvd2midi.

woid wrote

(except timecodes with value 0 which seems to have a special meaning, new line? line feed?).

I have made some experiments with "adjustTiming" on song 3df4 ("Moon shadow") and the result is very confusing. The lyrics on "Moon shadow" is delayed about 5 seconds (timecode of first character: 0x152=338). The music starts immediatly, but the lyrics starts about 5 seconds later.

The first attempt was to substract the delay (timecode) of the first character from all timecodes in the lyrics. My hope was that the lyrics would also start immediatly and in sync with the music.
Unfortunately the result was same as before. The lyrics was still delayed about 5 seconds.
$ ./adjustTiming 3d_f4.lpd -337

Then I made a second experiment. I tired to add the delay of the first character to the timecodes. My theory was that this would double the delay to about 10 seconds. When I did this something strange happend. The tempo of the music was changed to very very slow. The lyrics continued in the original tempo until the middle of the song where the lyrics stopped. The music continued in the same very slow pace.
$ ./adjustTiming 3d_f4.lpd 338

I am confused about this result. It seems to be some dependacy between the lyrics part and the midi part. Maybe there are some syncronisation events embedded in the midi stream? Have you found any strange events in the midistream that could be syncronisation events?

[bigboss97]

dvd2midi.zip
1) Added 0x0a in key2 mask for ASCII mode to filter out unexpected 0x0a in English songs.
2) Adjusted tempo.
3) Fixed the order of the characters in lyric.
4) Convert Chinese GBK to Unicode. Haven't got a unicode player to try. So, it could be totally wrong.

[bigboss97]

Based on our latest interpretation, are those 10,000 songs still missing?
Was that only a marketing trick and they believe that no one would count the songs?

woid wrote

I also noticed that I could only find about 20000 song entries in HOTDOG00.DAT. HOTDOG20.DAT suggested that there would be about 30000 songs so I worried that my scripts was not able to find all song entries in HOTDOG00.DAT and that would be why I could not find any english lyrics. So I made another script to see if there where any gaps where the missing 10000 songs could be located. I found that there where no gaps and in fact there are only 20000 song entries in HOTDOG00.DAT.

[bigboss97]

Dvd2Midi.zip
Just noticed that ReadLyricDat() is called without the mode parameter which makes the key2 mask no effect and cause incorrect characters in Chinese.

[woid]

Yes, I still think that they are "missing". I havent tired to find out which songs are missing, but it should be relatively easy to do that now that we know how to connect HOTDOG20.DAT with HOTDOG00.DAT.

[woid]

Have you had time to look at this closer? Have you found any sync events in the midi part?

I tired to look at your code in hd00Midi.cpp...
You seem to use the .lpd file and add one event for each letter. Does the timecodes in the .lpd file and the midi part have the same range and unit?

[bigboss97]

Not yet, I'm still working on the unicode issue, see another post (later).

Yes, I'm using your lpd-file because it's decoded already   I took it as it is. Until now, the songs I have tried seem to be correct. But they aren't sync'ed as we know from other kar-files, i.e. each text event matches exactly the note in melody. I was trying to do so at the beginning and then found out that it wasn't necessarily.

Regardless unsync in melody, are they appear the same in hard- and soft-players? As you can see, the 3rd and 4th bytes are zero in the mask2. So, the timecodes are "unfiltered". Maybe we need to do something.

woid wrote

Have you had time to look at this closer? Have you found any sync events in the midi part?

I tired to look at your code in hd00Midi.cpp...
You seem to use the .lpd file and add one event for each letter. Does the timecodes in the .lpd file and the midi part have the same range and unit?

[bigboss97]

After working a little deeper in the player, now I can see the actual problematic in unicode. As I mentioned before, the player is helpless if you don't tell him that it's dealing with a unicode file.

Nothing is easier than displaying unicode lyrics. BUT... when you see scrambled lyrics on the screen that means the text has been read incorrectly already. The better the player, the worse it could be. Why?? The reason is, there's no strict definition for KAR-file and a good player will try to cater as many files as possible. Some files (from internet) use '/' or '\' for new line and some have spaces at the end. The player will try to process all possible information and this is the problem. Some of the unicode information will be lost or converted unexpectedly while the file is being read.

Solution:
We need an indicator for unicode in the MIDI file. As far as I know Tune 1000 uses some keywords (with a leadin @), e.g. for title, language etc.. Wondering whether a keyword has already defined. If I use the same keyword the file can be played by more players. Otherwise I will suggest my own one:
@CODE=UNICODE
anyother suggestion?

Anyone is familiar with keywords in MIDI files? What are the most common ones, e.g. for title and singer etc?