Defect #10590

WikiContent::Version#text return string with #<Encoding:ASCII-8BIT> when uncompressed

Added by Alexander Oryol over 5 years ago. Updated over 5 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Wiki
Target version:1.4.0
Resolution:Fixed Affected version:

Description

  • ruby 1.9.3p125
  • postgresql 8.4.11
  • rails 2.3.14
  • redmine trunk

Wiki history compression: none

$ script/console 
Loading development environment (Rails 2.3.14)
/home/asor/.rvm/gems/ruby-1.9.3-p125-fast/gems/activesupport-2.3.14/lib/active_support/inflector.rb:3:in `<top (required)>': iconv will be deprecated in the future, use String#encode instead.
NOTE: Gem.source_index is deprecated, use Specification. It will be removed on or after 2011-11-01.
Gem.source_index called from /home/asor/.rvm/gems/ruby-1.9.3-p125-fast/gems/rails-2.3.14/lib/rails/gem_dependency.rb:21.
>> WikiPage.find(:first).content.text.encoding
=> #<Encoding:UTF-8>
>> WikiPage.find(:first).content.versions.first.text.encoding
=> #<Encoding:ASCII-8BIT>

We should always force_encoding to UTF-8, if data stored as binary type.
The patch attached.

When applied:

>> WikiPage.find(:first).content.text.encoding
=> #<Encoding:UTF-8>
>> WikiPage.find(:first).content.versions.first.text.encoding
=> #<Encoding:UTF-8>

force_encoding_always.patch Magnifier (1.06 KB) Alexander Oryol, 2012-04-03 10:16

Associated revisions

Revision 9329
Added by Jean-Philippe Lang over 5 years ago

WikiContent::Version#text may return ascii-8bit encoded string (#10590).

History

#1 Updated by Toshi MARUYAMA over 5 years ago

  • Assignee deleted (Toshi MARUYAMA)

What database do you use?
mysql1 has problems on Ruby1.9.
You need to use mysql2 instead of mysql1.
source:trunk/Gemfile@9244#L44

#2 Updated by Toshi MARUYAMA over 5 years ago

Sorry, in description "postgresql 8.4.11".

#3 Updated by Alexander Oryol over 5 years ago

Patch is (in attachment):

commit b0ec4111730809bc2f4bafd071ba2271e290e9e1
Author: Alex Eagle <eagle.alex@gmail.com>
Date:   Tue Apr 3 11:58:05 2012 +0400

    Force encoding string from data binary always (not only compressed)

diff --git a/app/models/wiki_content.rb b/app/models/wiki_content.rb
index 430e9a5..d5a3870 100644
--- a/app/models/wiki_content.rb
+++ b/app/models/wiki_content.rb
@@ -91,14 +91,16 @@ class WikiContent < ActiveRecord::Base
     end

     def text
-      @text ||= case compression
-      when 'gzip'
-        str = Zlib::Inflate.inflate(data)
+      @text ||= begin
+        str = case compression
+              when 'gzip'
+                Zlib::Inflate.inflate(data)
+              else
+                # uncompressed data
+                data
+              end
         str.force_encoding("UTF-8") if str.respond_to?(:force_encoding)
         str
-      else
-        # uncompressed data
-        data
       end
     end

#4 Updated by Toshi MARUYAMA over 5 years ago

It seems acts_as_versioned or PostgreSQL problem.

#5 Updated by Alexander Oryol over 5 years ago

Not PostgreSQL exactly.
DB return byte-array. Rails interpretate column (t.column :data, :binary) in string context as ASCII-8BIT unless otherwise.
Соde for transformation field 'data' into text defined in WikiContent::Version#text.
IMHO, this is model Version problem.

#6 Updated by Etienne Massip over 5 years ago

Might be tied to #6941; I just saw I had this old issue back and that upgrading pg gem to latest version in Gemfile (+bundle update) solved it.

#7 Updated by Alexander Oryol over 5 years ago

Updated pg gem to pg (0.13.2) do not solve it.

$ cat Gemfile|grep pg
    gem "pg"#, "~> 0.9.0" 
$ cat Gemfile.lock|grep pg
    pg (0.13.2)
  pg

Result:

>> WikiPage.first.content.versions.last.data.encoding
=> #<Encoding:ASCII-8BIT>

#8 Updated by Etienne Massip over 5 years ago

  • Tracker changed from Patch to Defect
  • Target version set to 1.4.0
  • Affected version (unused) set to devel

Potential release blocker.

Aside note: as explained above, required pg gem version should be raised to prevent #6941.

#9 Updated by Etienne Massip over 5 years ago

  • Status changed from New to Confirmed

#10 Updated by Jean-Philippe Lang over 5 years ago

  • Status changed from Confirmed to Closed
  • Resolution set to Fixed

Patch applied in r9329.

Also available in: Atom PDF