Understanding the HOTDOG files on DVD of California electronics

classic Classic list List threaded Threaded
135 messages Options
1234 ... 7
Reply | Threaded
Open this post in threaded view
|

Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
This topic is initiated by the "FAQ": How to Create Midi Disc To Play On Karaoke Player?

The discussion came to rolling by the post:
http://board.midibuddy.net/showpost.php?p=533722&postcount=31

Then followed by:
http://board.midibuddy.net/showpost.php?p=535795&postcount=35

Since the topic is getting too technical and might brand to few more other topics, I decided to create a seperate forum.

**** Recap ****
HOTDOG20 is holding a kind of TOC information.
0x38501
0E 12 32 0E 2 BECOME 1
0B 12 35 28 2 TIMES
0D 12 39 25 500 MILES

0x4bf30
0A 12 3A EA ZOMBIE
07 14 46 F7 123

The first byte is the length. The second byte seems to be a language identifier.


Phuoc Can HUA
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
When I looked at:
0E 12 31 1F DON'T STOP

and followed 0x311f in HOTDOG10. I found there's a 0x00 byte at the potential start of a data block:
00 16 30 00 00 00 00 00 00 00 00 00 00 00 00 00

because the file also starts with 2 bytes value as well:
08 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00

To confirm that they are data blocks I simply went to next data block visually where another pair of values is located (0x31a5). Then I found:
1F 12 31 A5 I'LL NEVER BREAK YOUR HEART

I went to next block and so on and found:
19 12 32 2B DON'T ANSWER THE DOOR
12 12 32 5C MAYBE TOMORROW

Conclusion, the 2 bytes are pointing to data blocks in hotdog10, as I suspected in a previous post.
Other languages should be pointing to other files.
Somewhere in the data block must be pointing to the main file, but where?
The song number should also be found in that piece of information.

That's for now.

bigboss97 wrote
0x38501
0E 12 32 0E 2 BECOME 1
0B 12 35 28 2 TIMES
0D 12 39 25 500 MILES

0x4bf30
0A 12 3A EA ZOMBIE
07 14 46 F7 123
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

woid
In reply to this post by bigboss97
hAnalyse1.c

Here is my script for analysing the TOC in the HOTDOG20.DAT file.

It is very simple and you have to manualy edit the 2 defines below before (re-)compiling:
#define TOC_OFFSET 0x3a444
#define TOC_FILE "/tmp/kar/HOTDOG20.DAT"

(TOC_OFFSET is the offset from the beginning of the file where the script starts to read the TOC)
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

woid
In reply to this post by bigboss97
Can you explain again the steps you made? I didnt follow.
Did you manage find out something about the structure in HOTDOG10.dat?

I dont think the 0x311F should be used as an offset, because there is also a 0x3120 and a 0x3121 entry and so on:

10 : 12 31 20 :  I FEEL LUCKY
14 : 12 31 21 :  I'LL HAVE TO SAY

Possible it could be used as an offset in a formula like (0x311F*x)+y but I think it will be difficult to find the value of x and y.
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
You are absolutely right. I know what was my mistake. I'll always find an entry for any position in hotdog10 because those numbers are so closed to each other (as you pointed you out)

So, they can't be offset address, even something like (0x311F*x)+y, because that means fixed size block, then you don't need an offset for each entry.
Could it be the song number, the one printed in the song list? or something related to that?
But then we don't have any link to the main file  

woid wrote
I dont think the 0x311F should be used as an offset, because there is also a 0x3120 and a 0x3121 entry and so on:

10 : 12 31 20 :  I FEEL LUCKY
14 : 12 31 21 :  I'LL HAVE TO SAY

Possible it could be used as an offset in a formula like (0x311F*x)+y but I think it will be difficult to find the value of x and y.
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

woid
I have found out some thing new about the format of HODOG20.DAT.

The first 4 bytes of the file is the number of songs on the DVD (in my case 30926)

In my HODOG20.DAT, The table of contents starts at address 0x2640.
In my file it is exacly 30926 entrys (starting at address 0x2640) of the format:

xx yy yy yy [Songtitle]

where:

xx is the Length of the entry
yy some kind of language code
zz zz is a number I yet haven't discovered the meaning of
And the rest of the entry contains a string (ASCII for english) with the songtitle.

In an other version of HOTDOG20.dat from another DVD I find:
+ 30534 songs (first 4 bytes of HOTDOG20.dat)
+ Exactly 30534 song entrys starting from adress 0x25E0.


Remaining questions:
+ How to interpret zz zz.
+ How to interpret the data from 5th byte until start of TOC.
+ How to interpret the data after the TOC
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
In reply to this post by bigboss97
They are actually song ID related. Following number are in the song list:
712815 2 BECOME 1
713609 2 TIMES
714630 500 MILES
715083 ZOMBIE

and these are in hotdog20 (zz zz):
32 0E = 12814
35 28 = 13608
39 25 = 14629
3A EA = 15082

Don't they look very similar to the song ID's?
Now it's the question, are the song ID and the address?
In order to find out about that we might need an experiment. We have to patch the zz zz bytes and feed the player. If I take "35 28" (2nd song) and patch it into 1st and 3rd song (in hotdog20 & 30). If I enter 713609 which song will it play?

P.S. I can't do that until next weekend because the player is currently not in house.


bigboss97 wrote
Could it be the song number, the one printed in the song list? or something related to that?
But then we don't have any link to the main file  
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

woid
This post was updated on .
In reply to this post by woid
The HOTDOG30.DAT have a simular format as HOTDOG20.DAT.
HOTDOG30.DAT contains a subset (english and vietnamese) of the songs in HOTDOG30.DAT, sorted by songwriter.

Format in HOTDOG30.DAT:

First 4 bytes contains the number of songs listed in this file.

Song entries (same format as discribed previously for HOTDOG20.DAT) starts at 0x0FD0 or 0x0F60 or somting like that. The file contains the number of enties as indicated by the first 4 bytes of the file. After the last entry the file ends.

Raimains to find out how to find the the beginning of the song entries and how to interpret the information before the song entries.

Maybe it is possible to compare the beginning of HOTDOG20.DAT and HOTDOG30.DAT and see if there is a pattern.
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

woid
HOTDOG10.DAT, HOTDOG11.DAT and HOTDOG12.DAT are identical in both versions of the DVD. I assume they contain something that does not change with the song on the disc, like e.g. the font or something like that.
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

woid
In reply to this post by woid
hAnalyse2.c

Revised version...
Reply | Threaded
Open this post in threaded view
|

Facts & Summary: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
In reply to this post by bigboss97
I want to gather some of the results here.

Let's call the DVD with 30534 songs DVDa and the one with 30926 songs DVDb.
DVDa has following contents:
15/08/2006  04:40 PM    <DIR>          VIDEO_TS
15/08/2006  04:40 PM    <DIR>          AUDIO_TS
15/08/2006  01:02 AM       536,517,424 HOTDOG00.DAT
17/06/2005  09:51 PM         3,143,680 HOTDOG10.DAT
19/06/2005  12:06 AM         1,177,600 HOTDOG11.DAT
31/03/2006  10:45 AM           690,176 HOTDOG12.DAT
15/08/2006  01:02 AM           473,724 HOTDOG20.DAT
20/06/2006  01:27 PM           230,751 HOTDOG30.DAT


1) HOTDOG10, 11 and 12 are exactly the same on DVDa & b.
2) HOTDOG20 & 30 contain potential TOC information.
3) The first 4 bytes in HOTDOG20 & 30 indicate the number of songs in the list of (2).
4) Each entry in list of (2) consists of 4 information bytes xx yy zz zz.
  a) xx is the length of the following string.
  b) yy is probably a language indicator.
  c) zz zz is a number related to the song ID (which is used for song choice).
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
In reply to this post by bigboss97
I'm thinking of returning the analysis back to HOTDOG00.

woid,
Based on your latest finding there is a difference of 392 songs between two versions. Let's assume there are only songs added (we can find out this by comparing the song list in HOTDOG20). We should be able to see some common data blocks in two versions. If that's the case then we can find out which songs and which data blocks are added.
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
If you put 32 bytes in one line and compare the two versions these are the unmatched lines (line number starts at 3):
3      00000000
...
21     000001F8
------------------------------------------------------------------------
------------------------------------------------------------------------
37     000003B8
...
76     000007FC
------------------------------------------------------------------------
------------------------------------------------------------------------
196921 005421E8
...
196976 005427EC
------------------------------------------------------------------------
------------------------------------------------------------------------
373799 009FB3F0
...
373836 009FB7FC
------------------------------------------------------------------------
------------------------------------------------------------------------
374494 009FFFF4
------------------------------------------------------------------------

Speculation:
The song data doesn't start until line 76.
In line 196921, it might be the first different song between two versions.
Since the first 76 lines are totally different they might be offset related (system information).
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
Sorry, here is the right data (it's line 66 instead of 76):
3      00000000
...
18     000001E0
------------------------------------------------------------------------
------------------------------------------------------------------------
32     000003A0
...
66     000007E0
------------------------------------------------------------------------
------------------------------------------------------------------------
172307 00542200
...
172354 005427E0
------------------------------------------------------------------------
------------------------------------------------------------------------
327075 009FB400
...
327106 009FB7E0
------------------------------------------------------------------------
------------------------------------------------------------------------
327682 009FFFE0
------------------------------------------------------------------------
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

woid
In reply to this post by bigboss97
I have found out something about the HOTDOG00.DAT file:

Starting from address 0x0200 in the file there seems to be a table containing addresses to another tables of song entries (4 byte addresses). Each song entry table may be for a diffrent language or something like that.

Each song entry table contains addresses to song entries (4 byte addresses).

Each song entry starts with the length of the entry (4 bytes). If you add the start address with the length, you end up with the next adress in the song entry table. The next 4 bytes in the song entry, I guess, is the length of the lyrics part.

e.g:

(Addresses to song entry tables)
00000200: 0000 0800 0054 2800 009f b800 00f1 a800

(First song entry table: Addresses to song entries)
00000800: 0000 1000 0000 f4a0 0001 5a20 0001 95f0
00000810: 0002 1eb0 0002 6270 0002 9800 0002 efb0
...
(First song entry in first song entry table)
00001000: 0000 e4a0 0000 052c d5b3 e9c7 3aaa c25c
...
(Second song entry in first song entry table)
0000f4a0: 0000 6580 0000 04f4 eec9 eec9 e15c 1510
...
(Third of first...)
00015a20: 0000 3bd0 0000 047c f1bd b9d2 4600 d397
...
(Forth of first..)
000195f0: 0000 88c0 0000 07cc ebc0 aabf 22ae 0a00
...
(Second song entry table: Addresses to song entries)
00542800: 0054 3000 0054 6c90 0054 ae40 0055 1a30
00542810: 0055 7450 0055 bba0 0055 de20 0056 56a0
...
(First song entry in second song entry table)
00543000: 0000 3c90 0000 05ec e3c5 e3c4 5968 9626
...

I havent had time to look closer at each song entry yet...
Time to make a new script for analysing HOTDOG00.DAT...  
Reply | Threaded
Open this post in threaded view
|

Re: Facts & Summary: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
In reply to this post by bigboss97
There are mainly Vietnamese and 0x05 songs added in DVDb.
diff.zip
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
In reply to this post by woid
Well done!  

I think this is the break through.
The first 4 bytes of each data block is indicating a value (~15-50k) which could be the length.
I checked some of the second 4 bytes. They seems to be the song ID. In your example, I found them with the language code 0x03 (HOTDOG20).

woid wrote
(Addresses to song entry tables)
00000200: 0000 0800 0054 2800 009f b800 00f1 a800

(First song entry table: Addresses to song entries)
00000800: 0000 1000 0000 f4a0 0001 5a20 0001 95f0
00000810: 0002 1eb0 0002 6270 0002 9800 0002 efb0
...
(First song entry in first song entry table)
00001000: 0000 e4a0 0000 052c d5b3 e9c7 3aaa c25c
...
(Second song entry in first song entry table)
0000f4a0: 0000 6580 0000 04f4 eec9 eec9 e15c 1510
...
(Third of first...)
00015a20: 0000 3bd0 0000 047c f1bd b9d2 4600 d397
...
(Forth of first..)
000195f0: 0000 88c0 0000 07cc ebc0 aabf 22ae 0a00
...
(Second song entry table: Addresses to song entries)
00542800: 0054 3000 0054 6c90 0054 ae40 0055 1a30
00542810: 0055 7450 0055 bba0 0055 de20 0056 56a0
...
(First song entry in second song entry table)
00543000: 0000 3c90 0000 05ec e3c5 e3c4 5968 9626
...
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

woid
I dont think the the second 4 bytes is the song ID, because some song entries have the same second 4 bytes (byte 4-7).

More probably it is the offset to the beginning of the midi part. I tried to add the offset (second 4 bytes) to the beginning of the song entry and seach for the magic midi string 00 FF C. For most enties I found the midi string after 15 bytes (byte 14 and 15 where always 00).

I have made a small c-program for analysing the song entries (Warning: I only tried it on a truncated version of HOTDOG00.DAT yet):

h00Analyse3.c
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

bigboss97
Administrator
I agree with you (after having a closer look).
I'll take over the MIDI part because I've internsively worked with MIDI before (have to refresh my mind). But it won't happen today.

I noticed that the entry tables seem all start at 0x*800 and another kind of table starts at 0x*c00.

We still need to find a link to the song title in order to interpret the lyrics or we have to wait until we get the MIDI right and listen to them until we find a familiar song

woid wrote
More probably it is the offset to the beginning of the midi part. I tried to add the offset (second 4 bytes) to the beginning of the song entry and seach for the magic midi string 00 FF C. For most enties I found the midi string after 15 bytes (byte 14 and 15 where always 00).
Reply | Threaded
Open this post in threaded view
|

Re: Understanding the HOTDOG files on DVD of California electronics

woid
Speculation:
I would guess that the song ID from HOTDOG20.DAT and HOTDOG30.DAT is used to index the tables in HOTDOG00.DAT.
The song IDs from HOTDOG20.DAT and HOTDOG30.DAT had the format:
yy xx xx
Where yy was some kind of country code and xx xx related to the numbers you punch into the remote control.

I would guess that the 8 most significant bits from xx xx is used to index the table at address 0x0200. From this we get a new table were the 8 lsb could used as index. Or something simular. Again this is just speculation.

I have not given up in finding the lyrics in each song entry. I hope that the lyrics for the english songs are encoded in ascii. Then it should be possible to write a script that find song entries containing english words. I will try something this evening.    
1234 ... 7