Cyrillic filenames in 10.0.2 broken?


#1

I used OC from version 6.x. After migration 9.1.6 - 10.0.2 some files with cyrillic names was deleted by local win-client. 8-( ) Yes, “i do backup before”.

My test: Freebsd 11.1, php56-5.6.31, mysql 5.6, apache24.

  1. Create new database, create new OC-10.0.2 instans, create user.
    Login to browser, create file “testИ.txt” ( И is unicode “D0 98” symbol), edit, save. Change directory to any, return - wow! Blank screen and rotating sector.
    Desktop client do not sync file: “File cant use ? in name”.

LANG=ru_RU.UTF-8, command ls displays cyrillic name “testИ.txt”

  1. File with name “АБВГДЕЁЖЗЙКЛ…ЭЮЯ.txt” (all 32 cyrillic symbols without И) created and synced to both directions.

  2. OC version 9.1.6 work fine witn any names. Checked with clear DB, new instanse.

Any ideas?


#2

I just tried on my 10.0.2 instance, created a file "testИ.txt" in the web UI and input some text.
Then let the desktop client sync it.

It works fine.
My server has openSUSE Leap 42.2.

You might want to try setting the server's local to "en_US.UTF-8" for the Apache/PHP process


#3

Thanks for idea.

Apache run as "www" user/group. /etc/login.conf does not contain user www, "default" section without any LANG-settings. I think, environment of apache "as is". php.ini consist default_charset = "UTF-8", all mbsting.* commented.

ru_RU.UTF-8 inside my shell.

I install nextcloud-12 (offtopic? :sunglasses: ), directory with filename with letter "И" say "Directory is not available, please check the message log or contact the administrator" and did not open. "loglevel=>0", restarted apache - no errors in $data/nextcloud.log.

After opening txt-file for editing, apache log has "GET /index.php/apps/files_texteditor/ajax/loadfile?filename=" and my long 32-cyrillic-symbols filename %D0%90%D0%91, next string "PUT /index.php/apps/files_texteditor/ajax/savefile"
When open dir with "И-file" - only "GET /ocs/v2.php/apps/notifications/api/v2/notifications", no error.

php-mbstring? iconv? May be rewrite sh-code "for s in А..Я; do echo php_cool_iconv($s, bla-bla-utf8); done" to php-code and run?

9.1.6 work fine! mv owncloud owncloud-10.0.2; mv owncloud-9.1.6 owncloud, press F5 in browser - all OK.

PS: FTP did not like russian "я" (code 255 in cp1251), PHP "И". No dropbox, OC-9.1.x forever! :sunglasses:


#4

Hmmm, the only change between 9.1 and 10.0 is the introduction of support for 4 bytes UTF-8 strings (emojis) with MySQL. Not sure what could be causing the problem in your environment.

As I cannot reproduce this, it will be impossible to debug and find out what is causing this.

@vicdeo can you confirm that Cyrillic also works for you with 10.0.2 ?


#5

Created dir "test1", open it, moved to dir file "testИ.txt", changed to "/", open "test1". On screen rotating circle. Logs:

apache access.log
1 - - [15/Aug/2017:09:09:36 +0300] "GET /index.php/apps/files/ajax/getstoragestats.php?dir=%2F HTTP/1.1" 200 275 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:09:48 +0300] "GET /ocs/v2.php/apps/notifications/api/v1/notifications?format=json HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:09:50 +0300] "MKCOL /remote.php/webdav/test1 HTTP/1.1" 201 - "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:09:50 +0300] "PROPFIND /remote.php/webdav/test1 HTTP/1.1" 207 801 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:09:53 +0300] "PROPFIND /remote.php/webdav/test1 HTTP/1.1" 207 801 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:09:53 +0300] "GET /index.php/avatar/az10/28 HTTP/1.1" 200 31 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:09:53 +0300] "GET /index.php/apps/files/ajax/getstoragestats.php?dir=%2Ftest1 HTTP/1.1" 200 275 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:10:18 +0300] "GET /ocs/v2.php/apps/notifications/api/v1/notifications?format=json HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:11:40 +0300] "PUT /remote.php/webdav/test1/test%D0%98.txt HTTP/1.1" 201 - "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:11:40 +0300] "PROPFIND /remote.php/webdav/test1/test%D0%98.txt HTTP/1.1" 207 768 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:11:41 +0300] "GET /index.php/apps/files/ajax/getstoragestats.php?dir=%2Ftest1 HTTP/1.1" 200 275 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
1 - - [15/Aug/2017:09:11:40 +0300] "GET /index.php/core/preview.png?file=%2Ftest1%2Ftest%D0%98.txt&c=7a4f16c64a0d610ce3db1133f5521834&x=32&y=32&forceIcon=0 HTTP/1.1" 200 944 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0
"
1 - - [15/Aug/2017:09:11:48 +0300] "GET /ocs/v2.php/apps/notifications/api/v1/notifications?format=json HTTP/1.1" 200 74 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
...repeat notifications...

====
apache error.log
empty

=====
owncloud.log ( loglevel>=0 )
{"reqId":"*","level":0,"time":"2017-08-15T06:11:41+00:00","remoteAddr":"1","user":"az10","app":"core","method":"GET","url":"\/index.php\/core\/preview.png?file=%2Ftest1%2Ftest%D0%98.txt&c=7a4f16c64a0d610ce3db1133f5521834&x=32&y=32&forceIcon=0","message":"Generating preview for \"\/test1\/test\u0418.txt\" with \"OC\Preview\TXT\""}

Tested Firefox54, IExplorer-11.0.41. Tested file "testИ.bin" - no logged in owncloud.log, dir not open.

Any other 32 testА, testБ, ...testЯ - all OK. Fantastic code \u0418 inside my FAMP.


#6

Are you saying that some unicode characters work fine but not others ?

Can you try with this one: "test福.txt"

Also check the "oc_filecache" table for these files and see if the special character is missing there as well.


#7

All cyrillic unicode work fine. One symbol: И = \u0418 broken.

I create file "testБ.txt" (unicode \u0411), inside oc_filecache string: "files/test3/testБ.txt"

File 'testИ.txt" in oc_filecahe has incorrect string: "files/test1/test�?.txt".

Output from table as is and hard encoded:

[root@hosting-j ~/owncloud]# mysql -u own-10 -p"*" < sel-table.sql
test�?�?.txt
test�?.txt
testБ.txt

hard encoded :
[root@hosting-j ~/owncloud]# mysql -u own-10 -p"*" < sel-table.sql | hd
00000000 6e 61 6d 65 0a 74 65 73 74 d0 3f d0 3f 2e 74 78 |name.test.?.?.tx|
00000010 74 0a 74 65 73 74 d0 3f 2e 74 78 74 0a 74 65 73 |t.test.?.txt.tes|
00000020 74 d0 91 2e 74 78 74 0a |t...txt.|
00000028

sel-table.sql=
use own-10;
select name from oc_filecache where name LIKE "test%txt";
quit

I can't type china alphabet... May be AltGr+numpad numbers?


#8

I copied filename from your post, rename file "testБ.txt" inside browser. All work fine with china-japan(?)-unicode. Dir test3 opening, i can edit file.

Data from table oc_cache as text and hex:

[root@hosting-j ~/owncloud]# mysql -u own-10 -p"*" < sel-table.sql
name
test�?�?.txt
test�?.txt
test福.txt

[root@hosting-j ~/owncloud]# mysql -u own-10 -p"" < sel-table.sql | hd
00000000  6e 61 6d 65 0a 74 65 73  74 d0 3f d0 3f 2e 74 78  |name.test.?.?.tx|
00000010  74 0a 74 65 73 74 d0 3f  2e 74 78 74 0a 74 65 73  |t.test.?.txt.tes|
00000020  74 e7 a6 8f 2e 74 78 74  0a                       |t....txt.|
00000029

What function convert name from string to table oc_cache? I will check for all chars: \u0410-042F only one func. Hmm... One function and one symbol \u0418. It's my FAMP or OC-code?


#9

there shouldn't be any conversion at all.

Now that you mention it, there is some normalizatiom here: https://github.com/owncloud/core/blob/master/lib/private/Files/Filesystem.php#L864


#10

Total disaster! Same problem on my installation after manual upgrade from 9.1 to 10.0.2! Polish ą,ć,ę,ł,ń,ó,ś,ź,ż also make problem! On the server side everything stays intact. Client changes the name of files (on computers) to strange characters and stops syncing! Web access is broken - cannot administer users (if they contain those chars in their name), cannot view/modify folders with files through web interface! I forced every user to login and change their name to contain only a,c,e,l,n,o,s,z,z - that allowed me to regain control over user accounts. Wow! What a mess! How can I contribute? I run OC 10.0.2 on shared webserver with no control over processes.


#11

Solved.

Very simlpe problem. Stupid problem. my.cnf contains:
default-character-set=cp1251
character-set-server=cp1251
init-connect="SET NAMES cp1251"

after changing to utf8, all work fine.

Why OC-9.1.6 (and early ver) worked? hmm... All tables inside has utf8-code.

PVince81! Thanks for patience and translation my English to normal. :sunglasses:

inox: check my.cnf, may be we from one kinderscool...