Feature #7037: CSV export encoding and excel. UTF-8 and BOM - Redmine

Custom queries

Documentation issues
Open defects
Open features
Patch queue
Plugin issues
Translation patches

Actions

Copy link

Feature #7037

closed

CSV export encoding and excel. UTF-8 and BOM

Added by John Yani over 13 years ago. Updated almost 9 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Jean-Philippe Lang

Category:

Issues

Target version:

3.1.0

Start date:

2010-12-03

Due date:

% Done:

Estimated time:

Resolution:

Fixed

Description

Redmine doesn't add BOM to the UTF-8 exported csv file, so I have to manually enforce csv encoding to windows-1251. In other way Excel will not display it correctly.

Maybe there is some setting to enforce BOM while doing csv export?

References:

Stackoverflow Q&A
Fix for this

Files

Download all files

7037-csv-bom.diff (4.53 KB) 7037-csv-bom.diff		Toshi MARUYAMA, 2015-04-02 10:37
7037-csv-bom-1.diff (4.53 KB) 7037-csv-bom-1.diff		Toshi MARUYAMA, 2015-04-02 10:42
7037-csv-bom-2.diff (4.54 KB) 7037-csv-bom-2.diff		Toshi MARUYAMA, 2015-04-02 11:02
excel2007.png (38.4 KB) excel2007.png		Jean-Philippe Lang, 2015-05-10 11:29
libreoffice.png (46.7 KB) libreoffice.png		Toshi MARUYAMA, 2015-05-10 12:48
Office2013Pro.png (41.6 KB) Office2013Pro.png		Toshi MARUYAMA, 2015-05-10 13:01

Related issues

Related to Redmine - Defect #7461: Special chars / umlauts broken in CSV / PDF export

Closed

Actions

Related to Redmine - Defect #7039: CSV export. Line breaks. Excel

Closed

2010-12-03

Actions

Related to Redmine - Defect #19523: problem in exporting Arabic text to csv

Closed

Actions

Related to Redmine - Defect #11116: CSV export encoding problem when issues and interface languages are not equal

Closed

Actions

Related to Redmine - Patch #17902: CSV encoding should be UTF-8 in French locale

Closed

Jean-Philippe Lang

Actions

Has duplicate Redmine - Feature #11301: Add "byte order mark" to the export CSV file

Closed

Actions

Issue # Delay: days Cancel

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by Terence Mill over 13 years ago

Is that fixed?

Actions

Copy link

Updated by John Yani over 13 years ago

Well, the fix exists. But AFAIK it is not merged yet.
The correct way would be to add some setting to enable/disable BOM instead of just hard-coding it.

Actions

Copy link

Updated by Alexander Kriventsov almost 13 years ago

John Yani wrote:

Redmine doesn't add BOM to the UTF-8 exported csv file, so I have to manually enforce csv encoding to windows-1251. In other way Excel will not display it correctly.

How did you do this?

Actions

Copy link

Updated by John Yani almost 13 years ago

Alexander Kriventsov wrote:

John Yani wrote:

Redmine doesn't add BOM to the UTF-8 exported csv file, so I have to manually enforce csv encoding to windows-1251. In other way Excel will not display it correctly.

How did you do this?

I've edited the "general_csv_encoding" value in corresponding "/config/locales/*.yml" file.

Actions

Copy link

Updated by Etienne Massip almost 13 years ago

Category set to Issues

Actions

Copy link

Updated by Nick Tan about 11 years ago

hi,

i have the exactly same issue, with the plugin https://github.com/sprosin/redmine_csv_with_bom now the csv export works as expected.

but i think this patch should be merge into the redmine core, not in form of a plugin with monkey patching style

Actions

Copy link

Updated by Yuri Rumega about 9 years ago

The problem is still actual in 2.6. Had to adapt "redmine_csv_with_bom" plugin.

Actions

Copy link

Updated by Toshi MARUYAMA about 9 years ago

Tracker changed from Defect to Feature

Actions

Copy link

Updated by Toshi MARUYAMA about 9 years ago

Related to Defect #19523: problem in exporting Arabic text to csv added

Actions

Copy link

#10

Updated by Toshi MARUYAMA about 9 years ago

File 7037-csv-bom.diff 7037-csv-bom.diff added
Target version set to 3.1.0

This is patch for trunk r14155.

Actions

Copy link

#11

Updated by Toshi MARUYAMA about 9 years ago

File 7037-csv-bom-1.diff 7037-csv-bom-1.diff added

Removed extra white space from previous patch.

Actions

Copy link

#12

Updated by Toshi MARUYAMA about 9 years ago

File 7037-csv-bom-2.diff 7037-csv-bom-2.diff added

Previous patch has mistake.
This patch fixed.

-+    encoding = Setting.csv_utf8_bom ? "UTF-8" : l(:general_csv_encoding)
++    encoding = Setting.csv_utf8_bom? ? "UTF-8" : l(:general_csv_encoding)

Actions

Copy link

#13

Updated by Toshi MARUYAMA about 9 years ago

Related to Defect #11116: CSV export encoding problem when issues and interface languages are not equal added

Actions

Copy link

#14

Updated by Jean-Philippe Lang almost 9 years ago

File excel2007.png excel2007.png added

Doesn't work for me when opening the CSV with Excel 2007 (which is supposed to support UTF-8 with BOM), tested with 7037-csv-bom-2.diff applied :

Actions

Copy link

#15

Updated by Toshi MARUYAMA almost 9 years ago

Could you check "csv_utf8_bom" is on in setting?

Actions

Copy link

#16

Updated by Jean-Philippe Lang almost 9 years ago

Yes, it was checked. The csv file is recognized as UTF-8 with BOM by my text editor but not by Excel 2007.

Actions

Copy link

#17

Updated by Toshi MARUYAMA almost 9 years ago

File libreoffice.png libreoffice.png added

$ hexdump -C issues.csv 
00000000  ef bb bf 23 3b 54 72 61  63 6b 65 72 3b 53 74 61  |...#;Tracker;Sta|
00000010  74 75 74 3b 50 72 69 6f  72 69 74 c3 a9 3b 53 75  |tut;Priorit..;Su|
00000020  6a 65 74 3b 41 73 73 69  67 6e c3 a9 20 c3 a0 3b  |jet;Assign.. ..;|
00000030  4d 69 73 2d c3 a0 2d 6a  6f 75 72 0a 38 37 3b 42  |Mis-..-jour.87;B|
00000040  75 67 3b 4e 65 77 3b 4e  6f 72 6d 61 6c 3b 4e 6f  |ug;New;Normal;No|
00000050  6e 2d 41 53 43 49 49 20  c3 a7 c3 a9 c3 a0 c3 b9  |n-ASCII ........|
00000060  3b 22 22 3b 31 30 2f 30  35 2f 32 30 31 35 20 31  |;"";10/05/2015 1|
00000070  39 3a 32 35 0a                                    |9:25.|
00000075

$ hexdump -c issues.csv 
0000000 357 273 277   #   ;   T   r   a   c   k   e   r   ;   S   t   a
0000010   t   u   t   ;   P   r   i   o   r   i   t 303 251   ;   S   u
0000020   j   e   t   ;   A   s   s   i   g   n 303 251     303 240   ;
0000030   M   i   s   - 303 240   -   j   o   u   r  \n   8   7   ;   B
0000040   u   g   ;   N   e   w   ;   N   o   r   m   a   l   ;   N   o
0000050   n   -   A   S   C   I   I     303 247 303 251 303 240 303 271
0000060   ;   "   "   ;   1   0   /   0   5   /   2   0   1   5       1
0000070   9   :   2   5  \n                                            
0000075

$ libreoffice --version
LibreOffice 4.0.4.2

$ libreoffice issues.csv

Actions

Copy link

#18

Updated by Toshi MARUYAMA almost 9 years ago

File Office2013Pro.png Office2013Pro.png added

I can open on my Japanese MS Office 2013 Professional with general_csv_separator change.

diff --git a/config/locales/fr.yml b/config/locales/fr.yml
--- a/config/locales/fr.yml
+++ b/config/locales/fr.yml
@@ -158,7 +158,8 @@ fr:
   general_text_no: 'non'
   general_text_yes: 'oui'
   general_lang_name: 'French (Français)'
-  general_csv_separator: ';'
+  # general_csv_separator: ';'
+  general_csv_separator: ','
   general_csv_decimal_separator: ','
   general_csv_encoding: ISO-8859-1
   general_pdf_fontname: freesans

Actions

Copy link

#19

Updated by Toshi MARUYAMA almost 9 years ago

#7039#note-15:

German wikipedia

The delimiter semicolon is expected when file is in ANSI.
The delimiter TAB is expected when file is in Unicode.

I don't know about French.

Actions

Copy link

#20

Updated by Jean-Philippe Lang almost 9 years ago

Is there any situation where we want to use UTF-8 without BOM?
Because I'm not sure that a new setting is required, we could add the BOM when CSV is encoded in UTF-8 just like the redmine_csv_with_bom plugin does.

Actions

Copy link

#21

Updated by Jean-Philippe Lang almost 9 years ago

Status changed from New to Closed
Assignee set to Jean-Philippe Lang
Resolution set to Fixed

Actions

Copy link

#22

Updated by Go MAEDA over 8 years ago

Has duplicate Feature #11301: Add "byte order mark" to the export CSV file added

Actions

Copy link

#23

Updated by Toshi MARUYAMA over 8 years ago

Related to Patch #17902: CSV encoding should be UTF-8 in French locale added

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Redmine

Custom queries

Feature #7037

CSV export encoding and excel. UTF-8 and BOM

Updated by Terence Mill over 13 years ago

Updated by John Yani over 13 years ago

Updated by Alexander Kriventsov almost 13 years ago

Updated by John Yani almost 13 years ago

Updated by Etienne Massip almost 13 years ago

Updated by Nick Tan about 11 years ago

Updated by Yuri Rumega about 9 years ago

Updated by Toshi MARUYAMA about 9 years ago

Updated by Toshi MARUYAMA about 9 years ago

Updated by Toshi MARUYAMA about 9 years ago

Updated by Toshi MARUYAMA about 9 years ago

Updated by Toshi MARUYAMA about 9 years ago

Updated by Toshi MARUYAMA about 9 years ago

Updated by Jean-Philippe Lang almost 9 years ago

Updated by Toshi MARUYAMA almost 9 years ago

Updated by Jean-Philippe Lang almost 9 years ago

Updated by Toshi MARUYAMA almost 9 years ago

Updated by Toshi MARUYAMA almost 9 years ago

Updated by Toshi MARUYAMA almost 9 years ago

Updated by Jean-Philippe Lang almost 9 years ago

Updated by Jean-Philippe Lang almost 9 years ago

Updated by Go MAEDA over 8 years ago

Updated by Toshi MARUYAMA over 8 years ago