Feature #7037

CSV export encoding and excel. UTF-8 and BOM

Added by John Yani about 12 years ago. Updated over 7 years ago.

Status:ClosedStart date:2010-12-03
Priority:NormalDue date:
Assignee:Jean-Philippe Lang% Done:

0%

Category:Issues
Target version:3.1.0
Resolution:Fixed

Description

Redmine doesn't add BOM to the UTF-8 exported csv file, so I have to manually enforce csv encoding to windows-1251. In other way Excel will not display it correctly.

Maybe there is some setting to enforce BOM while doing csv export?

References:

7037-csv-bom.diff Magnifier (4.53 KB) Toshi MARUYAMA, 2015-04-02 10:37

7037-csv-bom-1.diff Magnifier (4.53 KB) Toshi MARUYAMA, 2015-04-02 10:42

7037-csv-bom-2.diff Magnifier (4.54 KB) Toshi MARUYAMA, 2015-04-02 11:02

excel2007.png (38.4 KB) Jean-Philippe Lang, 2015-05-10 11:29

libreoffice.png (46.7 KB) Toshi MARUYAMA, 2015-05-10 12:48

Office2013Pro.png (41.6 KB) Toshi MARUYAMA, 2015-05-10 13:01

Related issues

Related to Redmine - Defect #7461: Special chars / umlauts broken in CSV / PDF export Closed
Related to Redmine - Defect #7039: CSV export. Line breaks. Excel Closed 2010-12-03
Related to Redmine - Defect #19523: problem in exporting Arabic text to csv Closed
Related to Redmine - Defect #11116: CSV export encoding problem when issues and interface lan... Closed
Related to Redmine - Patch #17902: CSV encoding should be UTF-8 in French locale Closed
Duplicated by Redmine - Feature #11301: Add "byte order mark" to the export CSV file Closed

Associated revisions

Revision 14302
Added by Jean-Philippe Lang over 7 years ago

Adds a class for handling CSV generation (#7037).

Revision 14303
Added by Jean-Philippe Lang over 7 years ago

Add BOM to UTF-8 encoded CSV (#7037).

History

#1 Updated by Terence Mill about 12 years ago

Is that fixed?

#2 Updated by John Yani about 12 years ago

Well, the fix exists. But AFAIK it is not merged yet.
The correct way would be to add some setting to enable/disable BOM instead of just hard-coding it.

#3 Updated by Alexander Kriventsov over 11 years ago

John Yani wrote:

Redmine doesn't add BOM to the UTF-8 exported csv file, so I have to manually enforce csv encoding to windows-1251. In other way Excel will not display it correctly.

How did you do this?

#4 Updated by John Yani over 11 years ago

Alexander Kriventsov wrote:

John Yani wrote:

Redmine doesn't add BOM to the UTF-8 exported csv file, so I have to manually enforce csv encoding to windows-1251. In other way Excel will not display it correctly.

How did you do this?

I've edited the "general_csv_encoding" value in corresponding "/config/locales/*.yml" file.

#5 Updated by Etienne Massip over 11 years ago

  • Category set to Issues

#6 Updated by Nick Tan almost 10 years ago

hi,

i have the exactly same issue, with the plugin https://github.com/sprosin/redmine_csv_with_bom now the csv export works as expected.

but i think this patch should be merge into the redmine core, not in form of a plugin with monkey patching style

#7 Updated by Yuri Rumega about 8 years ago

The problem is still actual in 2.6. Had to adapt "redmine_csv_with_bom" plugin.

#8 Updated by Toshi MARUYAMA almost 8 years ago

  • Tracker changed from Defect to Feature

#9 Updated by Toshi MARUYAMA almost 8 years ago

  • Related to Defect #19523: problem in exporting Arabic text to csv added

#10 Updated by Toshi MARUYAMA almost 8 years ago

This is patch for trunk r14155.

#11 Updated by Toshi MARUYAMA almost 8 years ago

Removed extra white space from previous patch.

#12 Updated by Toshi MARUYAMA almost 8 years ago

Previous patch has mistake.
This patch fixed.

-+    encoding = Setting.csv_utf8_bom ? "UTF-8" : l(:general_csv_encoding)
++    encoding = Setting.csv_utf8_bom? ? "UTF-8" : l(:general_csv_encoding)

#13 Updated by Toshi MARUYAMA almost 8 years ago

  • Related to Defect #11116: CSV export encoding problem when issues and interface languages are not equal added

#14 Updated by Jean-Philippe Lang over 7 years ago

Doesn't work for me when opening the CSV with Excel 2007 (which is supposed to support UTF-8 with BOM), tested with 7037-csv-bom-2.diff applied :

#15 Updated by Toshi MARUYAMA over 7 years ago

Could you check "csv_utf8_bom" is on in setting?

#16 Updated by Jean-Philippe Lang over 7 years ago

Yes, it was checked. The csv file is recognized as UTF-8 with BOM by my text editor but not by Excel 2007.

#17 Updated by Toshi MARUYAMA over 7 years ago

$ hexdump -C issues.csv 
00000000  ef bb bf 23 3b 54 72 61  63 6b 65 72 3b 53 74 61  |...#;Tracker;Sta|
00000010  74 75 74 3b 50 72 69 6f  72 69 74 c3 a9 3b 53 75  |tut;Priorit..;Su|
00000020  6a 65 74 3b 41 73 73 69  67 6e c3 a9 20 c3 a0 3b  |jet;Assign.. ..;|
00000030  4d 69 73 2d c3 a0 2d 6a  6f 75 72 0a 38 37 3b 42  |Mis-..-jour.87;B|
00000040  75 67 3b 4e 65 77 3b 4e  6f 72 6d 61 6c 3b 4e 6f  |ug;New;Normal;No|
00000050  6e 2d 41 53 43 49 49 20  c3 a7 c3 a9 c3 a0 c3 b9  |n-ASCII ........|
00000060  3b 22 22 3b 31 30 2f 30  35 2f 32 30 31 35 20 31  |;"";10/05/2015 1|
00000070  39 3a 32 35 0a                                    |9:25.|
00000075

$ hexdump -c issues.csv 
0000000 357 273 277   #   ;   T   r   a   c   k   e   r   ;   S   t   a
0000010   t   u   t   ;   P   r   i   o   r   i   t 303 251   ;   S   u
0000020   j   e   t   ;   A   s   s   i   g   n 303 251     303 240   ;
0000030   M   i   s   - 303 240   -   j   o   u   r  \n   8   7   ;   B
0000040   u   g   ;   N   e   w   ;   N   o   r   m   a   l   ;   N   o
0000050   n   -   A   S   C   I   I     303 247 303 251 303 240 303 271
0000060   ;   "   "   ;   1   0   /   0   5   /   2   0   1   5       1
0000070   9   :   2   5  \n                                            
0000075

$ libreoffice --version
LibreOffice 4.0.4.2

$ libreoffice issues.csv

#18 Updated by Toshi MARUYAMA over 7 years ago

I can open on my Japanese MS Office 2013 Professional with general_csv_separator change.

diff --git a/config/locales/fr.yml b/config/locales/fr.yml
--- a/config/locales/fr.yml
+++ b/config/locales/fr.yml
@@ -158,7 +158,8 @@ fr:
   general_text_no: 'non'
   general_text_yes: 'oui'
   general_lang_name: 'French (Français)'
-  general_csv_separator: ';'
+  # general_csv_separator: ';'
+  general_csv_separator: ','
   general_csv_decimal_separator: ','
   general_csv_encoding: ISO-8859-1
   general_pdf_fontname: freesans

#19 Updated by Toshi MARUYAMA over 7 years ago

#7039#note-15:

German wikipedia

The delimiter semicolon is expected when file is in ANSI.
The delimiter TAB is expected when file is in Unicode.

I don't know about French.

#20 Updated by Jean-Philippe Lang over 7 years ago

Is there any situation where we want to use UTF-8 without BOM?
Because I'm not sure that a new setting is required, we could add the BOM when CSV is encoded in UTF-8 just like the redmine_csv_with_bom plugin does.

#21 Updated by Jean-Philippe Lang over 7 years ago

  • Status changed from New to Closed
  • Assignee set to Jean-Philippe Lang
  • Resolution set to Fixed

#22 Updated by Go MAEDA over 7 years ago

  • Duplicated by Feature #11301: Add "byte order mark" to the export CSV file added

#23 Updated by Toshi MARUYAMA about 7 years ago

  • Related to Patch #17902: CSV encoding should be UTF-8 in French locale added

Also available in: Atom PDF