Encoding error for captions with special characters

[jswafford]

I'm helping college students in a French class make Omeka exhibits, and we noticed that captions for videos brought in as items by the YouTube import plugin have encoding errors. For example, the word "vidéo" shows up as "vidéo" (see below). This doesn't happen for captions for images in exhibits. In case it matters, this happens on Chrome and Safari, and we're using the Big Picture theme.
Any idea what's happening?
Thanks so much!

[NedHenry]

Thanks for reporting this! It sounds like an issue with UTF. Unfortunately we don't have a lot of development resources to look into this for a while. If anyone wants to take a look into the code and try to see what's going on, we would really appreciate the help!

…

On Fri, May 4, 2018 at 11:48 AM, jswafford ***@***.***> wrote: I'm helping college students in a French class make Omeka exhibits, and we noticed that captions for videos brought in as items by the YouTube import plugin have encoding errors. For example, the word "vidéo" shows up as "vidéo" (see below). This doesn't happen for captions for images in exhibits. In case it matters, this happens on Chrome and Safari, and we're using the Big Picture theme. Any idea what's happening? Thanks so much! [image: screen shot 2018-05-04 at 2 47 16 pm] <https://user-images.githubusercontent.com/1047458/39646642-1735ff48-4faa-11e8-8d08-da41cd182fee.png> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#28>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGCJ-NBVoTBUgllQpxKerX5AnRDz0HJxks5tvKJugaJpZM4TzGzG> .

[benbwu]

Hi,
I had this problem with videos that have the weird UTF-8 symbols in the text that accompanies the video.
It turns out MySQL does not support UTF-8 characters with more than 3 bytes.
After much searching I found this:
https://stackoverflow.com/questions/8491431/how-to-replace-remove-4-byte-characters-from-a-utf-8-string-in-php

I put the cleaning line into YouTubeImport\helpers\import.php:
$elementText = preg_replace('/[\x{10000}-\x{10FFFF}]/u', "\xEF\xBF\xBD", $elementText);
I had to clean the incoming element text and tags.
Look in the code for $elementText and $tags and clean these.
When I have more time I'd like to look into this in more detail and maybe submit it as a potential commit.
In the meantime hopefully this helps someone else.
Ben

[NedHenry]

Many thanks Ben, that's super helpful

…

On Tue, Aug 27, 2019, 6:19 AM benbwu ***@***.***> wrote: Hi, I had this problem with videos that have the weird UTF-8 symbols in the text that accompanies the video. It turns out MySQL does not support UTF-8 characters with more than 3 bytes. After much searching I found this: https://stackoverflow.com/questions/8491431/how-to-replace-remove-4-byte-characters-from-a-utf-8-string-in-php I put the cleaning line into YouTubeImport\helpers\import.php: $elementText = preg_replace('/[\x{10000}-\x{10FFFF}]/u', "\xEF\xBF\xBD", $elementText); I had to clean the incoming element text and tags. Look in the code for $elementText and $tags and clean these. When I have more time I'd like to look into this in more detail and maybe submit it as a potential commit. In the meantime hopefully this helps someone else. Ben — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#28?email_source=notifications&email_token=ABQIT6DOGFANGBAMQLCHL33QGUSXHA5CNFSM4E6MNTDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5HWOFY#issuecomment-525297431>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQIT6H4DAPZFKZWHKCFOETQGUSXHANCNFSM4E6MNTDA> .

[ecbettinger]

@benbwu Yes!!! Thank you!! Your solution helped me solve a problem in which we had not been able to import videos that included emojis in the Description field. Now we can.