Webdav problem / Special Characters

Hans-Jurgen_Tappe · October 5, 2010, 9:13pm

Hi!

Is it a known problem (I know you did a lot of testing on that) that
file names which I upload through Windows / NetDrive have the file name
stripped after the first special character?

E.g. take a file name “GrößereDatei.txt” and upload it through NetDrive.
The file name is saved as “Gr” in the file manager.

This problem does not exist if I upload the file through the web interface.
Then, the Umlaut displays wrongly in latin-1 / UTF-8 display within
NetDrive.
I can then rename the file, until I change / insert an Umlaut in
netdrive, which again results in the stripped name.

If this is a new problem, I could help investigating on that. There is
nothinǵ related in the error log here. The same problem applies for 1.8
and trunk.

Best regards,
Hans-Jürgen

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Hans-Jurgen_Tappe · October 6, 2010, 5:58am

Hi!

The problem is even worse if I try to up upload a file starting with an
umlaut, e.g. “Öffentlichkeitsarbeit.txt”:
The whole directory seems empty then in Netdrive,
but in the web frontend, a file “/” or “” has been created.
If you delete this invalid file in the web frontend, all files in that
directory are removed (!), also cleaning the invalid file.

We should definitely take a look after this, I’d suppose, either
prohibiz such files to be uploaded if it is a technical problem between
netdrive and webdav.php or fix the behavior…

Best regards,
Hans-Jürgen

P.S.: This is the message in error.log, when deleting the invalid file
from /home/user/Test-Dateien (an existing directory), which deletes all
directory contents:

sqlfs_stream_wrapper::dir_opendir(‘sqlfs://default/home/user/Test-Dateien’,4)
sqlfs://default/home/user/Test-Dateien is no directory!, referer:
http://localhost/egroupware/index.php?menuaction=filemanager.filemanager_ui.index&path=%2Fhome%2Fuser%2FTest-Dateien

vfs_stream_wrapper::dir_opendir( vfs://default/home/user/Test-Dateien,4)
opendir(sqlfs://default/home/user/Test-Dateien) failed!, referer:
http://localhost/egroupware/index.php?menuaction=filemanager.filemanager_ui.index&path=%2Fhome%2Fuser%2FTest-Dateien

Is it a known problem (I know you did a lot of testing on that) that
file names which I upload through Windows / NetDrive have the file name
stripped after the first special character?

E.g. take a file name “GrößereDatei.txt” and upload it through NetDrive.
The file name is saved as “Gr” in the file manager.

This problem does not exist if I upload the file through the web interface.
Then, the Umlaut displays wrongly in latin-1 / UTF-8 display within
NetDrive.
I can then rename the file, until I change / insert an Umlaut in
netdrive, which again results in the stripped name.

If this is a new problem, I could help investigating on that. There is
nothinǵ related in the error log here. The same problem applies for 1.8
and trunk.

Best regards,
Hans-Jürgen

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Ralf_Becker_6 · October 6, 2010, 1:36pm

Hi Hans Jürgen,

You have to tell Netdrive the charset to use. You can not detect client charset from the URL. Server charset is easy to detect, as propfind answers contain it.

Ralf

Ralf Becker
Stylite GmbH
[ open style of IT ]

Morschheimer Strasse 15
D - 67292 Kirchheimbolanden

fon +49 (6352) 70629-0
fax +49 (6352) 70629-30
mobil +49 (170) 3147833
email rb@stylite.de

www.egroupware-support.eu
www.stylite.de

Geschäftsführer: Andre Keller,
Gudrun Müller, Ralf Becker
Registergericht Kaiserslautern, HRB 30575

Hi!

Is it a known problem (I know you did a lot of testing on that) that
file names which I upload through Windows / NetDrive have the file name
stripped after the first special character?

E.g. take a file name “GrößereDatei.txt” and upload it through NetDrive.
The file name is saved as “Gr” in the file manager.

This problem does not exist if I upload the file through the web interface.
Then, the Umlaut displays wrongly in latin-1 / UTF-8 display within
NetDrive.
I can then rename the file, until I change / insert an Umlaut in
netdrive, which again results in the stripped name.

If this is a new problem, I could help investigating on that. There is
nothinǵ related in the error log here. The same problem applies for 1.8
and trunk.

Best regards,
Hans-Jürgen

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Leithoff_Klaus · October 6, 2010, 2:11pm

As far as I have seen, NetDrive 1.0.8.15 Higher Versions - I dont know. does not support the charset option.

kl@stylite.de
Stylite GmbH
[ open style of IT ]
Morschheimer Strasse 15
67292 Kirchheimbolanden

fon 06352 . 70629-0
fax 06352 . 70629-30
www.stylite.de

Geschäftsführer: Andre Keller, Gudrun Müller, Ralf Becker
Handelsregister Kaiserslautern HRB 30575
USt-ID: DE214280951

Hans-Jurgen_Tappe · October 6, 2010, 9:56pm

Hi Ralf!

You have to tell Netdrive the charset to use. You can not detect client
charset from the URL. Server charset is easy to detect, as propfind
answers contain it.

NetDrive seems to provide the encoding on connect:

PROPFIND /egroupware/webdav.php/ HTTP/1.1
Host: localhost
Keep-Alive:
Connection: TE, Keep-Alive
TE: trailers
Depth: 1
Content-Length: 86
Content-Type: application/xml[4 bytes missing in capture file]<?xml
version=“1.0” encoding=“utf-8”?>

In wireshark (follow stream); I also see:

MOVE /egroupware/webdav.php/home/user/Test/Neues%20Textdokument.txt
HTTP/1.1
Host: localhost
Connection: TE
TE: trailers
Destination:
http://localhost/egroupware/webdav.php/home/user/Test/Gr��ereDatei.txt

I wonder what the problem is - even with an incorrect charset and with
eGroupware server being utf-8, the %f6 should be translateable into “ö”,
regardless the server encoding. urldecode should be capable of this if I
understand correctly.

Even if the character would result in a “strange” character in UTF-8,
this does not explain why the string is cut at this character (instead
of saving any character)?

Maybe, you can shed some light into this?
It should be of most interest, which part of the code cuts instead of
converting. The %20 is finally decoded correctly…

Thanks a lot,
Hans-Jürgen

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Leithoff_Klaus · October 7, 2010, 8:46am

cutting of strings is probably caused by iconv converting from or to not matching charset.
Depending on the options provided with the to parameter the conversion gets:
If you append the string //TRANSLIT to out_charset transliteration is activated. This means that when a character can’t be represented in the target charset, it can be approximated through one or several similarly looking characters. If you append the string //IGNORE, characters that cannot be represented in the target charset are silently discarded. Otherwise, str is cut from the first illegal character

So that may happen here.

–
kl@stylite.de
Stylite GmbH
[ open style of IT ]
Morschheimer Strasse 15
67292 Kirchheimbolanden

fon 06352 . 70629-0
fax 06352 . 70629-30
www.stylite.de

Geschäftsführer: Andre Keller, Gudrun Müller, Ralf Becker
Handelsregister Kaiserslautern HRB 30575
USt-ID: DE214280951

Hans-Jurgen_Tappe · October 7, 2010, 4:04pm

Hello Klaus!

cutting of strings is probably caused by iconv converting from or to not matching charset.
Depending on the options provided with the to parameter the conversion gets:
If you append the string //TRANSLIT to out_charset transliteration is activated. This means that when a character can’t be represented in the target charset, it can be approximated through one or several similarly looking characters. If you append the string //IGNORE, characters that cannot be represented in the target charset are silently discarded. Otherwise, str is cut from the first illegal character
So that may happen here.

Thanks for the hint. I am currently quite busy, but if I find time, I
will take your hint and start hunting that effect (if not someone else
is faster this time).

Best regards,
Hans-Jürgen

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Hans-Jurgen_Tappe · October 8, 2010, 7:36am

Hello Ralf!

You have to tell Netdrive the charset to use. You can not detect
client charset from the URL. Server charset is easy to detect, as
propfind answers contain it.

I finally found the “Advanced” settings for each netdrive site and the
select to set UTF-8. This actually solved the problem for this netdrive
installation and site.

The problem is that it is not obvious and can delete files if used
improperly.

The least, I would like to do is to prevent eGroupware to accept files
with an empty name or the name “/” (these are the files which cause most
problems).

The second thing (which would also cover the first) is: I would like to
reject files which have a non-convertible encoding and add an error_log
if that happens, as a hint to the administrator (or is there another way
to inform the user about the need to select UTF-8 in the client program?).

Would you agree on that approach?

Best regards,
Hans-Jürgen

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Ralf_Becker_6 · October 8, 2010, 6:37pm

I agree in principle, thought I have no idea how to implement that. Currently we do nothing with a filename from a PUT URL, it’s assumed to be UTF-8 and stored as is.
Maybe Php has a method to check if a string is valid utf-8 …

Ralf

Ralf Becker
Stylite GmbH
[ open style of IT ]

Morschheimer Strasse 15
D - 67292 Kirchheimbolanden

fon +49 (6352) 70629-0
fax +49 (6352) 70629-30
mobil +49 (170) 3147833
email rb@stylite.de

www.egroupware-support.eu
www.stylite.de

Geschäftsführer: Andre Keller,
Gudrun Müller, Ralf Becker
Registergericht Kaiserslautern, HRB 30575

Hello Ralf!

You have to tell Netdrive the charset to use. You can not detect
client charset from the URL. Server charset is easy to detect, as
propfind answers contain it.

I finally found the “Advanced” settings for each netdrive site and the
select to set UTF-8. This actually solved the problem for this netdrive
installation and site.

The problem is that it is not obvious and can delete files if used
improperly.

The least, I would like to do is to prevent eGroupware to accept files
with an empty name or the name “/” (these are the files which cause most
problems).

The second thing (which would also cover the first) is: I would like to
reject files which have a non-convertible encoding and add an error_log
if that happens, as a hint to the administrator (or is there another way
to inform the user about the need to select UTF-8 in the client program?).

Would you agree on that approach?

Best regards,
Hans-Jürgen

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Leithoff_Klaus · October 11, 2010, 7:21am

I found some stuff, that I use in felamimail now, if there is no charset given I try to detect the charset by testing the mail against several charsets by comparing the md5 of the original text withe the md5 of the text iconved with the charset set to the same charset for source and target.

kl@stylite.de
Stylite GmbH
[ open style of IT ]
Morschheimer Strasse 15
67292 Kirchheimbolanden

fon 06352 . 70629-0
fax 06352 . 70629-30
www.stylite.de

Geschäftsführer: Andre Keller, Gudrun Müller, Ralf Becker
Handelsregister Kaiserslautern HRB 30575
USt-ID: DE214280951

Hans-Jurgen_Tappe · October 11, 2010, 8:35am

Hi Klaus!

I found some stuff, that I use in felamimail now, if there is no charset given I try to detect the charset by testing the mail against several charsets by comparing the md5 of the original text withe the md5 of the text iconved with the charset set to the same charset for source and target.

I did not yet dig into the code, but did some research. I found the
following snipped on the web, which might help:

function file_get_contents_utf8($fn) {
$content = file_get_contents($fn);
return mb_convert_encoding($content, ‘UTF-8’,
mb_detect_encoding($content, ‘UTF-8, ISO-8859-15’, true));

So there is a function mb_detect_encoding(), which could be used for a
best-guess, but I am not sure if this can be used in the eGroupware
environment, also because I did not yet find the time to look for the
probable place where this webdav file name conversion takes place.

Best regards,
Hans-Jürgen

Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers