Issue with non ISO-8859-1 characters

[notsemisu]

There is an issue where further artist/composer/genre tags (TPE1/TCOM/TCON) after the first artist/composer/genre are cut/ignored if any of them use characters that are not part of ISO-8859-1 (in my case: JIS). Characters from those standards do, however, get displayed.

Edit: Another, rather unrelated, issue: when all artists are displayed, they get divided by a comma instead of a semicolon, as it's done in "song info", in all tabs but "Artist". There, for some reason, it displays only the currently selected artist. Would be nice if it were the same everywhere.

[GoneMAD]

your tags need to be UTF8/Unicode is they arent part of ISO-8859-1. Other text encodings are not supported by java/kotlin, gmmp's tag reader, nor the id3v2 tagging standard:

id3.org/id3v2.3.0#ID3v2_frame_overview
"If nothing else is said a string is represented as ISO-8859-1 characters in the range $20 - $FF. Such strings are represented as <text string>, or <full text string> if newlines are allowed, in the frame descriptions. All Unicode strings use 16-bit unicode 2.0 (ISO/IEC 10646-1:1993, UCS-2). Unicode strings must begin with the Unicode BOM ($FF FE or $FE FF) to identify the byte order."

"Edit: Another, rather unrelated, issue: when all artists are displayed, they get divided by a comma instead of a semicolon, as it's done in "song info", in all tabs but "Artist". There, for some reason, it displays only the currently selected artist. Would be nice if it were the same everywhere."

Song Info reads straight from the tags which are comma delimited, other views are reading from the database. GMMP supports multiple artists / genres / etc and splits up the tags when storing in the database. When retrieving from the database, it is not possible to specify the delimiter used, so that is why a comma shows. This is a limitation of sqlite3 (the database)

[GoneMAD]

en.wikipedia.org/wiki/ID3

Textual frames are marked with an encoding byte.[14]

$00 – ISO-8859-1 (LATIN-1, Identical to ASCII for values smaller than 0x80).
$01 – UCS-2 encoded Unicode with BOM, in ID3v2.2 and ID3v2.3.
$02 – UTF-16BE encoded Unicode without BOM, in ID3v2.4.
$03 – UTF-8 encoded Unicode, in ID3v2.4.

[notsemisu]

Jun 21, 2022 20:09:13 GMT -5 GoneMAD said:

your tags need to be UTF8/Unicode is they arent part of ISO-8859-1. Other text encodings are not supported by java/kotlin, gmmp's tag reader, nor the id3v2 tagging standard

Well, the thing is, they are Unicode.

[GoneMAD]

email in some sample files that have the issue

[e-mail]

[notsemisu]

Done.

[GoneMAD]

thanks. might take some time for me to look into it, but i'll post back (and/or reply to your email when i do)

[GoneMAD]

i replied via email, but in summary these files do not follow the id3 spec. GMMP shows up to where the text/string is terminated with a 00 (in hex)

The id3v2 spec states:
" If the textstring is followed by a termination ($00 (00)) all the following information should be ignored and not be displayed. "

[notsemisu]

I did read and reply to your mail. Suffice to say, you didn't read that file as Unicode.

[GoneMAD]

Jun 28, 2022 0:26:25 GMT -5 notsemisu said:

I did read and reply to your mail. Suffice to say, you didn't read that file as Unicode.

I 100% read it as unicode. I kept my original reply simple since I didnt know how technical you were (reason i just said 00)

00 00 = null code which ends the string. GMMP displays all the characters up until 00 00 which is what the id3v2 spec states
" If the textstring is followed by a termination ($00 (00)) all the following information should be ignored and not be displayed. "

id3.org/id3v2.3.0#Text_information_frames

This is not a bug. I added a feature request on trello to look into supporting null separators, but tags that do not follow spec are not considered bugs. Regardless, for multi artist / genre / etc in gmmp, either multiple tags need to be used or a single tag that is semi-colon delimited

[notsemisu]

Yeah, after you sent me the link to taglib I figured out where it breaks. Rather unfortunate but at least it can be avoided.

If anyone cares: taglib understands $00 as delimiter but not $00 00. taglib does however read ID3v2.4 tags, and there the standard delimiter is $00 (Unicode too). So if you want all artists/genre/etc. to be displayed using any Unicode character, use 2.4 tags.

[MotleyGord]

Jun 28, 2022 10:24:38 GMT -5 notsemisu said:

If anyone cares: taglib understands $00 as delimiter but not $00 00. taglib does however read ID3v2.4 tags, and there the standard delimiter is $00 (Unicode too). So if you want all artists/genre/etc. to be displayed using any Unicode character, use 2.4 tags.

However before you go and update all of your v2.3 library metadata to ID3v2.4, be sure to check what is supported by all of your players and library manager software. The most common support is still ID3v2.3 for the majority. If you move to v2.4 you may have issues with any programs that aren't there (yet). On the other hand, the multi-tag or semicolon delimiter approach has support already even with ID3v2.3 and may be the more stable solution for most.

[GoneMAD]

yeah 2.4 support doesnt seem very widespread, im guessing because its an "informal standard" according to the doc which is from 2000 lol id3.org/id3v2.4.0-structure