OX6 message encoding improvement

[johnye]

Dear community,

I have tested OX6 for some days. It's excellent software.

The big issue that prevents it from being widely used is the language encoding.

For a big percentage of messages in Chinese or some other eastern languages sent by other mail clients, OX6 can’t display them correctly.

The reason is:
Many mail clients (such as outlook express) have a "default message encoding", they have not explicitly rfc-2047 encoded subject and body, some messages don't have full header like:
Content-Type: text/html; charset=XXXX
Content-Transfer-Encoding: XXXX

For this kind of email, OX6 can't display them correctly.
It's no problem for messages sent by OX6 webgui, because 0X6 generates a fully RFC compatible mail.

If OX6 can have different default encoding for different user, the problem will be resolved. If a user found the message is bad character, he can switch to a good encoding to view it.

Maybe OX6 have this function, but I don't know.

Anyway, I think, OX6 should NOT take it for granted that all mail clients are fully compatible to RFC and must generate fully rfc compatible messages.

Thanks.

John Ye

[Martin Heiland]

Hi,

i also think that we are not the ones to blame for bad e-mails on a technical side(!), but to the user, it does not matter if the e-mail has been sent broken or if we have a problem displaying it - he just wants to read his E-Mail. Many E-Mail clients implemented a bunch of workarounds to display E-Mails that are not rfc compliant, so we have done. Could it be possible to attach such a "broken" E-Mail to this thread? Sadly i don't have a chinese Outlook Express at hand. With that E-Mail we could investigate the problem and probably deliver a fix for it.

[johnye]

Hi Braun,

I have a test server 218.247.5.189. I can mail you the OX account and password so you can logon OX6 to see yourself, if I have your email address. My email is johnye@webizmail.com.

I put some 700+ messages in that mailbox, if viewed by OUTLOOK, OUTLOOK EXPRESS, or by squirrelwebmail, the messages are shown OK.

Technically, it's NOT OX6 to blame for the mess. But we and other potential users just want it to work --- they don't care the technical details, they want to choose a tool to work for them.

I think OX is great groupware server software, it must have more and more potential users who will choose OX rather than MS exchange or who will quit existing MS products.

I may be doing some work for the Chinese encoding things in OX, but I am not JAVA expert (I am an expert on C and email system, anti spam things). So far, tried many times but failed to compile OX on my suse11.1 server. Now I am using suse RPMs.

Thanks.

John Ye

[johnye]

Hi Braun,

I got your email and have sent to you all the logon information, but the mail has not available to you yet, because your mail server has a grey listing that temporarily rejects the message.

I use 2 servers to send, the same greylisting is there.

John Ye

[Martin Heiland]

Hi,

yes i got both of your E-Mail, the greylisting blocks the first mail but typically your smtp server will try to send it again after a while and then it will pass through. Most spam relays only send once, for performance causes.

i already took a look at you installation and it seems you're using some kind of filtering or preprocessing software that adds something to the header:
Subject: (PAT)Amazon.cn����3�۵�����嵥���ع⣬����1�ۿ�ϲ�9��
=?GB2312?B?x+W1pbTzxti54qOsuPzT0DHV27/xz7LHwLm6u+62rw==?=

in order to decode this subject correctly, the charset GB2312 (chinese) is used but only for a part of the subject line. The encoding starts at =? and ends at =?=, all stuff at the line above will not be encoded with GB2312 and therefor looks broken.
Your E-Mail "TEST???TEST?????" is encoded with iso-8859-1 standard (western europe) and this charset does not allow chinese characters at all.

Could you please check if there is something that modifies the Subject when delivering E-Mail?

Greetings

[johnye]

Hi Braun,

Yes. The subject is just using a user-default encoding, not using =?GB2312?B? to tell the decoder the encoding.

It just does the same as you use a console telnet x.x.x.x 25 to manually send an email.

For other messages, I can use Simplified Chinese GB18030 to send msg in OE, and it's encoded with ISO8859-1 so ox can't correctly see the body.

My thought on this issue:

1) OX had better to have a per user default encoding. User can logon to specify his own default encoding. For a system that is used by multi-lingual users at the same time (in an international company, this is the case), the system should work this way. OE, MSOUTLOOK, others, webmails can do so.

If the mail text (including subject and body) is tagged with =?ENCODE?b?,
then the system decode it with the ENCODE. If the text is no encode tag, the user default encoding is used to display the text.

2) For html body, OX should directly display the body content text (as more as possible) rather than showing an html attach name to let user click it to display. We can see OE MSO and other Linux email client, they are doing better.

John Ye

[Martin Heiland]

Hi,

i assume nobody is sending E-Mail via telnet in 2009, but anyway

1)
The subject lacks an encoding, only a part of the text is given with a proper encoding, so the first (temporarily) way to fix it would be finding the application that adds the non-encoded string to the subject. If Outlook Express is using iso-8859-1 even if you have configured GB18030 then it's simply a bug in Outlook Express, they actually add a wrong charset and we and other clients are decoding this broken charset information - there are thousands of charsets in the world, if the E-Mail contains a charset we assume it is the correct one, guessing other charsets even if a valid charset is given is not an option here.

Your idea with letting a user choosing the default compose/read charset sounds nice - but we're already using UTF-8 as the default charset and this charset does contain most other charsets, no need for sending a E-Mail in another charset imho. When it comes to a default charset for reading E-Mail, well surely this could be done, but how should a user know what charset has been originally used? Trying out a dozen charset won't satisfy the user. Plus, the E-Mail charset is set by the sender, not by the reader so the user will have no indicator what he wants to use.

2)
Enable "Allow html formatted E-Mails?" at Configuration->E-Mail

[johnye]

Hi Braun,

You are right. OX is OK using utf-8 to generate message that is fully compatible to rfc, so it should be viewed correctly by other mail client.

Technically, OX has no problem both sending and viewing incomings. The problem is that there are many kind of mail client out there and they are not fully compatible to rfc, sometimes.
And also, not all mails are sent manually by people with mail clients, some are generated by machine, they are often not stick to rfc, but using the "local font encoding"---just local text without =?XXXX?b? tag.

I used OE, OUTLOOK, and webmail to view the messages in the test account, they all are OK with the no-encoding subjects. OE, OUTLOOK seem to use OS(winxp)'s encoding to decode the text, while squirrelwebmail has a personal language encoding to select. If I select english, I can't view them correctly.

My thoughts on this issue is only preliminary. The basic idea is to compare OX to other popular MCs like OUTLOOK. If they can do, OX had better to do.

This encoding is really not an "issue" for westen languages. It becomes some kind of issue for double bytes languages like Chinese or Japanese, etc.

Thanks for your testing and answering.

John Ye

[johnye]

Hi Braun,

Is there any way that I can hide the menu box at the up-right area of the screen, or shrink it into a menu bar ( one row )?

I can click the button to change its size but can't shrink to one row.

Some testers feel the menu box takes too much precious space that can be used by email and others. For new user, the box is necessary, but for old users, the box can become a bar with small icons.

Thanks.

John Ye

[Martin Heiland]

Hi,

i've added a enhancement nomination to allow the user to set his prefered decoding for E-Mail parts that don't have a encoding, i think that's useful, thanks for that suggestion now it's up to product management to decide if or when we're going to implement it. If there is market demand (customer that has this problem) it'll be a accelerator though.

The panel bar can be collapsed to 3-rows at minimum, making it 1-row high is not possible at the moment, but i think this is an interesting idea for some modules. Since we don't have icons for every button, and there are not many useful buttons for "V-Split" for example, we'll have to check in detail what such a feature would look like and how it affects the user. Whats the background of this request? Is it just because a user don't need it or is is a screen resolution based problem?

Greetings

[johnye]

Braun,

Many thanks for your consideration.

I am hoping that OX can get more and more successful and gain more and more users so I tested it and put questions and suggestions which may not fully reasonable.

John Ye

[johnye]

Braun,

In terms of the menu box issue,
"Whats the background of this request? Is it just because a user don't need it or is is a screen resolution based problem?
"

I feel the box is a waste of space. For example having a Sony notebook 15# monitor 1024x768 resolutions, I feel not enough space for email list display.

When I shrink it to 3 rows, the actual rows it occupies are 7:
Category (New, E-Mails, ...) takes 1 row
3 rows body
pull down extension menu takes 1 row
Bank bar half or 1 row
Then, the main body's title (Mails, Contact, ...) takes one row.
If enabling quick info display, 2 more rows are needed.

It's much better to free these rows to give the main body contents (mail list, contact, ... etc) to use. This will give user a better look and make him feel more comfortable with the working area.

BTW, I feel that the quick info had better to be moved down into the bottom bar with only 1 row space. If not enough, use a button to left-right scroll.

John Ye