Project

General

Profile

Actions

Defect #19550

closed

Issue query: crash when processing saved query(filters) with cyrillic after upgrade to 3.0 from 2.*

Added by Pavel Liavonau almost 9 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Issues
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Resolution:
Wont fix
Affected version:

Description

After updrading from 2 to 3 version I've found a bug.

Some if saved custom queries have become broken. Those that have enumeration of values in filter with cyrillic symbols. I've attached some information from log.
When I create new custom query at 3.0 version in works properly.


Files

query_with_ciryllic.txt (1.59 KB) query_with_ciryllic.txt Pavel Liavonau, 2015-04-02 14:18
rails.log (1.2 KB) rails.log Pavel Liavonau, 2015-04-02 14:18
about_env.txt (584 Bytes) about_env.txt Pavel Liavonau, 2015-04-02 15:19
mysql_encoding_info.txt (3.32 KB) mysql_encoding_info.txt Pavel Liavonau, 2015-04-03 10:57
queries.filters_new_without_error.txt (607 Bytes) queries.filters_new_without_error.txt do not works Pavel Liavonau, 2015-06-23 13:21
queries.filters_old_with_error.txt (1.02 KB) queries.filters_old_with_error.txt works Pavel Liavonau, 2015-06-23 13:21
Actions #1

Updated by Toshi MARUYAMA almost 9 years ago

  • Status changed from New to Needs feedback
Actions #2

Updated by Pavel Liavonau almost 9 years ago

Added output of about script. What else do you need?

Actions #3

Updated by Toshi MARUYAMA almost 9 years ago

Actions #4

Updated by Pavel Liavonau almost 9 years ago

Attached output of mysql character set information. What can you advice me to do?

Actions #5

Updated by Toshi MARUYAMA almost 9 years ago

  • Status changed from Needs feedback to Closed
  • Resolution set to Invalid

You need to set "UTF-8" for all MySQL setting.
http://www.redmine.org/boards/1/topics/36573?r=36629#message-36629
And you need to convert all MySQL data from latin1 to UTF-8.
http://www.redmine.org/boards/2/topics/35979?r=36082#message-36082

Actions #6

Updated by Pavel Liavonau over 8 years ago

Your advice has not helped me.
My all settings are now only for UTF-8, but error still occures.
It is the bug. Admit it.
Conversion of dump don't help because the problem is at queries.filters column.
I've added 2 files with content of this column for not old not working query and new created with the same functionality.
Look: old format is "binary" but new is not.
So, fix it, or propose me script that converts queries.filters field properly.

Thanks.

Actions #7

Updated by Toshi MARUYAMA over 8 years ago

  • Status changed from Reopened to Closed
  • Resolution changed from Invalid to Wont fix

Pavel Liavonau wrote:

Look: old format is "binary" but new is not.

Old format is Ruby 1.8 yaml format.
http://stackoverflow.com/questions/7576500/ruby-1-8-7-why-to-yaml-converts-some-strings-to-non-readable-bytes

It is the bug. Admit it.

It is not bug.
I don't know your old yaml data.
I think it is saved in non UTF-8.

Redmine treat yaml data as UTF-8 nevertheless it is Ruby 1.8 yaml data or not.
source:tags/3.0.3/app/models/custom_field.rb#L121

We cannot support in your case such as database encoding was not UTF-8.
You need to convert yaml data manually.

Actions #8

Updated by Toshi MARUYAMA over 8 years ago

Toshi MARUYAMA wrote:

I don't know your old yaml data.
I think it is saved in non UTF-8.

It seems your yaml is UTF-8.
But I don't know your all Ruby 1.8 yaml is UTF-8.

$ RAILS_ENV=development rails console
Loading development environment (Rails 4.2.2)
1.9.3-p551 :001 > x = YAML.load('
1.9.3-p551 :002'>   - !binary |
1.9.3-p551 :003'>
1.9.3-p551 :004'>     0JDQl9CY0JzQo9Ci
1.9.3-p551 :005'>
1.9.3-p551 :006'>   - !binary |
1.9.3-p551 :007'>
1.9.3-p551 :008'>     0JDQstGC0L7QutC+0LzQv9C+0LfQuNGC
1.9.3-p551 :009'>
1.9.3-p551 :010'>   - !binary |
1.9.3-p551 :011'>
1.9.3-p551 :012'>     0JHQodCX
1.9.3-p551 :013'>
1.9.3-p551 :014'>   - !binary |
1.9.3-p551 :015'>
1.9.3-p551 :016'>     0JHQldCb0JrQkNCg0JQ=
1.9.3-p551 :017'>
1.9.3-p551 :018'>   - !binary |
1.9.3-p551 :019'>
1.9.3-p551 :020'>     0JHQvtCx0YDQkNCz0YDQvtC80LDRiA==
1.9.3-p551 :021'>
1.9.3-p551 :022'>   - !binary |
1.9.3-p551 :023'>
1.9.3-p551 :024'>     0JLQmNCi0K/Ql9Cs
1.9.3-p551 :025'>
1.9.3-p551 :026'>   - !binary |
1.9.3-p551 :027'>
1.9.3-p551 :028'>     0JPQldCk0JXQodCi
1.9.3-p551 :029'>
1.9.3-p551 :030'>   - !binary |
1.9.3-p551 :031'>
1.9.3-p551 :032'>     0JfQnNCX
1.9.3-p551 :033'>
1.9.3-p551 :034'>   - !binary |
1.9.3-p551 :035'>
1.9.3-p551 :036'>     0JrQnNCX
1.9.3-p551 :037'>
1.9.3-p551 :038'>   - !binary |
1.9.3-p551 :039'>
1.9.3-p551 :040'>     0JrQntCg0J/Qo9Ch
1.9.3-p551 :041'>
1.9.3-p551 :042'>   - !binary |
1.9.3-p551 :043'>
1.9.3-p551 :044'>     0JrQo9Cf0J7Qmw==
1.9.3-p551 :045'>
1.9.3-p551 :046'>   - !binary |
1.9.3-p551 :047'>
1.9.3-p551 :048'>     0JvQtdCx0LXQtNGP0L3RjA==
1.9.3-p551 :049'>
1.9.3-p551 :050'>   - !binary |
1.9.3-p551 :051'>
1.9.3-p551 :052'>     0JzQkNCX
1.9.3-p551 :053'>
1.9.3-p551 :054'>   - !binary |
1.9.3-p551 :055'>
1.9.3-p551 :056'>     0JzQotCX
1.9.3-p551 :057'>
1.9.3-p551 :058'>   - !binary |
1.9.3-p551 :059'>
1.9.3-p551 :060'>     0J3QldCk0JDQlw==
1.9.3-p551 :061'>
1.9.3-p551 :062'>   - !binary |
1.9.3-p551 :063'>
1.9.3-p551 :064'>     0J/Qu9Cw0L3QsNGA
1.9.3-p551 :065'>
1.9.3-p551 :066'>   - !binary |
1.9.3-p551 :067'>
1.9.3-p551 :068'>     0KDQtdGB0L/QuNGA0LDRgtC+0YA=
1.9.3-p551 :069'>
1.9.3-p551 :070'>   - !binary |
1.9.3-p551 :071'>
1.9.3-p551 :072'>     0KDQo9CR0JjQnQ==
1.9.3-p551 :073'>
1.9.3-p551 :074'>   - !binary |
1.9.3-p551 :075'>
1.9.3-p551 :076'>     0KHQkNCX
1.9.3-p551 :077'>
1.9.3-p551 :078'>   - !binary |
1.9.3-p551 :079'>
1.9.3-p551 :080'>     0KHQsNC70Y7Rgg==
1.9.3-p551 :081'>
1.9.3-p551 :082'>   - !binary |
1.9.3-p551 :083'>
1.9.3-p551 :084'>     0KHQrdCT0Jc=
1.9.3-p551 :085'>
1.9.3-p551 :086'>   - !binary |
1.9.3-p551 :087'>
1.9.3-p551 :088'>     0KHRgtCw0L3QutC+0LDQs9GA0LXQs9Cw0YI=
1.9.3-p551 :089'>
1.9.3-p551 :090'>   - !binary |
1.9.3-p551 :091'>
1.9.3-p551 :092'>     0KLRgNC+0LjRhtC6
1.9.3-p551 :093'>
1.9.3-p551 :094'>   - !binary |
1.9.3-p551 :095'>
1.9.3-p551 :096'>     0KfQotCX
1.9.3-p551 :097'>
1.9.3-p551 :098'> ')
 => ["\xD0\x90\xD0\x97\xD0\x98\xD0\x9C\xD0\xA3\xD0\xA2", "\xD0\x90\xD0\xB2\xD1\x82\xD0\xBE\xD0\xBA\xD0\xBE\xD0\xBC\xD0\xBF\xD0\xBE\xD0\xB7\xD0\xB8\xD1\x82", "\xD0\x91\xD0\xA1\xD0\x97", "\xD0\x91\xD0\x95\xD0\x9B\xD0\x9A\xD0\x90\xD0\xA0\xD0\x94", "\xD0\x91\xD0\xBE\xD0\xB1\xD1\x80\xD0\x90\xD0\xB3\xD1\x80\xD0\xBE\xD0\xBC\xD0\xB0\xD1\x88", "\xD0\x92\xD0\x98\xD0\xA2\xD0\xAF\xD0\x97\xD0\xAC", "\xD0\x93\xD0\x95\xD0\xA4\xD0\x95\xD0\xA1\xD0\xA2", "\xD0\x97\xD0\x9C\xD0\x97", "\xD0\x9A\xD0\x9C\xD0\x97", "\xD0\x9A\xD0\x9E\xD0\xA0\xD0\x9F\xD0\xA3\xD0\xA1", "\xD0\x9A\xD0\xA3\xD0\x9F\xD0\x9E\xD0\x9B", "\xD0\x9B\xD0\xB5\xD0\xB1\xD0\xB5\xD0\xB4\xD1\x8F\xD0\xBD\xD1\x8C", "\xD0\x9C\xD0\x90\xD0\x97", "\xD0\x9C\xD0\xA2\xD0\x97", "\xD0\x9D\xD0\x95\xD0\xA4\xD0\x90\xD0\x97", "\xD0\x9F\xD0\xBB\xD0\xB0\xD0\xBD\xD0\xB0\xD1\x80", "\xD0\xA0\xD0\xB5\xD1\x81\xD0\xBF\xD0\xB8\xD1\x80\xD0\xB0\xD1\x82\xD0\xBE\xD1\x80", "\xD0\xA0\xD0\xA3\xD0\x91\xD0\x98\xD0\x9D", "\xD0\xA1\xD0\x90\xD0\x97", "\xD0\xA1\xD0\xB0\xD0\xBB\xD1\x8E\xD1\x82", "\xD0\xA1\xD0\xAD\xD0\x93\xD0\x97", "\xD0\xA1\xD1\x82\xD0\xB0\xD0\xBD\xD0\xBA\xD0\xBE\xD0\xB0\xD0\xB3\xD1\x80\xD0\xB5\xD0\xB3\xD0\xB0\xD1\x82", "\xD0\xA2\xD1\x80\xD0\xBE\xD0\xB8\xD1\x86\xD0\xBA", "\xD0\xA7\xD0\xA2\xD0\x97"]
1.9.3-p551 :099 > x.each { |i| puts i }
АЗИМУТ
Автокомпозит
БСЗ
БЕЛКАРД
БобрАгромаш
ВИТЯЗЬ
ГЕФЕСТ
ЗМЗ
КМЗ
КОРПУС
КУПОЛ
Лебедянь
МАЗ
МТЗ
НЕФАЗ
Планар
Респиратор
РУБИН
САЗ
Салют
СЭГЗ
Станкоагрегат
Троицк
ЧТЗ
 => ["\xD0\x90\xD0\x97\xD0\x98\xD0\x9C\xD0\xA3\xD0\xA2", "\xD0\x90\xD0\xB2\xD1\x82\xD0\xBE\xD0\xBA\xD0\xBE\xD0\xBC\xD0\xBF\xD0\xBE\xD0\xB7\xD0\xB8\xD1\x82", "\xD0\x91\xD0\xA1\xD0\x97", "\xD0\x91\xD0\x95\xD0\x9B\xD0\x9A\xD0\x90\xD0\xA0\xD0\x94", "\xD0\x91\xD0\xBE\xD0\xB1\xD1\x80\xD0\x90\xD0\xB3\xD1\x80\xD0\xBE\xD0\xBC\xD0\xB0\xD1\x88", "\xD0\x92\xD0\x98\xD0\xA2\xD0\xAF\xD0\x97\xD0\xAC", "\xD0\x93\xD0\x95\xD0\xA4\xD0\x95\xD0\xA1\xD0\xA2", "\xD0\x97\xD0\x9C\xD0\x97", "\xD0\x9A\xD0\x9C\xD0\x97", "\xD0\x9A\xD0\x9E\xD0\xA0\xD0\x9F\xD0\xA3\xD0\xA1", "\xD0\x9A\xD0\xA3\xD0\x9F\xD0\x9E\xD0\x9B", "\xD0\x9B\xD0\xB5\xD0\xB1\xD0\xB5\xD0\xB4\xD1\x8F\xD0\xBD\xD1\x8C", "\xD0\x9C\xD0\x90\xD0\x97", "\xD0\x9C\xD0\xA2\xD0\x97", "\xD0\x9D\xD0\x95\xD0\xA4\xD0\x90\xD0\x97", "\xD0\x9F\xD0\xBB\xD0\xB0\xD0\xBD\xD0\xB0\xD1\x80", "\xD0\xA0\xD0\xB5\xD1\x81\xD0\xBF\xD0\xB8\xD1\x80\xD0\xB0\xD1\x82\xD0\xBE\xD1\x80", "\xD0\xA0\xD0\xA3\xD0\x91\xD0\x98\xD0\x9D", "\xD0\xA1\xD0\x90\xD0\x97", "\xD0\xA1\xD0\xB0\xD0\xBB\xD1\x8E\xD1\x82", "\xD0\xA1\xD0\xAD\xD0\x93\xD0\x97", "\xD0\xA1\xD1\x82\xD0\xB0\xD0\xBD\xD0\xBA\xD0\xBE\xD0\xB0\xD0\xB3\xD1\x80\xD0\xB5\xD0\xB3\xD0\xB0\xD1\x82", "\xD0\xA2\xD1\x80\xD0\xBE\xD0\xB8\xD1\x86\xD0\xBA", "\xD0\xA7\xD0\xA2\xD0\x97"]

Actions #9

Updated by Pavel Liavonau over 8 years ago

Toshi MARUYAMA wrote:

It seems your yaml is UTF-8.
But I don't know your all Ruby 1.8 yaml is UTF-8.

what do you mean?

If it is UTF-8 already, what can I do now? I am not so strong in Ruby to write conversion script by myself.

I use ruby ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux].

Actions #10

Updated by Toshi MARUYAMA over 8 years ago

Pavel Liavonau wrote:

Toshi MARUYAMA wrote:

It seems your yaml is UTF-8.
But I don't know your all Ruby 1.8 yaml is UTF-8.

what do you mean?

$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "日本語" 
>>> s.decode("UTF-8").encode("EUC-JP")
'\xc6\xfc\xcb\xdc\xb8\xec'

$ irb
1.8.7-head :001 > require 'yaml'
 => true
1.8.7-head :002 > "\xc6\xfc\xcb\xdc\xb8\xec".to_yaml
 => "--- !binary |\nxvzL3Ljs\n\n" 
1.8.7-head :003 > "日本語".to_yaml
 => "--- !binary |\n5pel5pys6Kqe\n\n" 

Actions #11

Updated by Pavel Liavonau over 8 years ago

Ок. I see.
It is hard to create universal script for migration from ruby 1.8 to new versions.
This queries are not important for me and I've just deleted them with a sql query.

delete * from queries where filters like '%binary%';

Thank you very much for answering.

Actions

Also available in: Atom PDF