https://www.redmine.org/https://www.redmine.org/favicon.ico?16793021292012-12-04T13:10:29ZRedmineRedmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=431982012-12-04T13:10:29ZStanislav German-Evtushenko
<ul></ul><p>I have checked the database and found out that old gzip records and new gzip records of "wiki_content_versions" are different. New one looks like a text with X' at the beginning: <pre>X'HEXHEXHEXHEX'</pre> but old one is more like a binary <pre>'x?u?OJ?@.????a??t?z?.......etc'</pre><br />How can I fix that?</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=431992012-12-04T15:07:15ZEtienne Massip
<ul></ul><p>PostgreSQL v9?</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432022012-12-04T16:54:39ZStanislav German-Evtushenko
<ul></ul><p>SQLite</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432082012-12-04T18:58:18ZStanislav German-Evtushenko
<ul></ul><p>I have just tried to proceed as clean migration as possible. I made "sqlite3 .dump" on the old system and created new production db with that dump on the new system. Then I made "rake db:migrate". And now I get the following message for old gzipped history:<br /><pre>
Started GET "/projects/project1/wiki/Start?version=57" for 10.10.10.10 at 2012-12-04 22:47:20 +0400
Processing by WikiController#show as HTML
Parameters: {"version"=>"57", "project_id"=>"project1", "id"=>"Start"}
Current user: anonymous
Rendered wiki/_content.html.erb (1.1ms)
Rendered wiki/show.html.erb within layouts/base (7.6ms)
Completed 500 Internal Server Error in 21ms
ActionView::Template::Error (invalid code -- missing end-of-block):
1: <div class="wiki wiki-page">
2: <%= textilizable content, :text, :attachments => content.page.attachments,
3: :edit_section_links => (@sections_editable && {:controller => 'wiki', :action => 'edit', :project_id => @page.project, :id => @page.title}) %>
4: </div>
app/models/wiki_content.rb:97:in `inflate'
app/models/wiki_content.rb:97:in `text'
app/helpers/application_helper.rb:522:in `textilizable'
app/views/wiki/_content.html.erb:2:in `_app_views_wiki__content_html_erb__46270150915704852_59849880'
app/views/wiki/show.html.erb:37:in `_app_views_wiki_show_html_erb__3888669060056873954_57312980'
app/controllers/wiki_controller.rb:92:in `show'
</pre></p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432092012-12-04T19:00:24ZStanislav German-Evtushenko
<ul></ul><p>Any ideas?</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432222012-12-05T07:06:27ZStanislav German-Evtushenko
<ul></ul><p>Please help! I'm not sure if I have to go back to Redmine 1.2 so far and drop any changes had been made since upgrade (two days of changes for now) or I can still work with the new Redmine 2.1 and will be able to move history from the old database in an appropriate way later.</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432242012-12-05T09:09:29ZStanislav German-Evtushenko
<ul></ul><p>I have just checked Redmine 1.4-stable. It uses same gzip format as Redmine 1.2-stable does: <pre>INSERT INTO "wiki_content_versions" VALUES(1162,30,30,4,'xڕSMk<DC>@^L<BD>/<EC>...etc','gzip','This change is aimed to test gzip in SQLite database','2012-12-05 09:59:35',40);</pre></p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432422012-12-05T14:52:59ZStanislav German-Evtushenko
<ul></ul>Can this change be the cause of the problem?
<ul>
<li>RAILS 2:
<ul>
<li><a class="external" href="http://apidock.com/rails/v2.1.0/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class">http://apidock.com/rails/v2.1.0/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class</a></li>
<li><a class="external" href="http://apidock.com/rails/v2.3.8/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class">http://apidock.com/rails/v2.3.8/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class</a></li>
</ul>
</li>
<li>RAILS 3:
<ul>
<li><a class="external" href="http://apidock.com/rails/v3.2.8/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class">http://apidock.com/rails/v3.2.8/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class</a></li>
</ul></li>
</ul> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432462012-12-05T17:29:38ZStanislav German-Evtushenko
<ul></ul>I've done some more investigation. Please have a look.
<ul>
<li>How old DB stores blob data: <pre>
$ sqlite3 production-test.db "update wiki_content_versions set data='test' where id=2294"
$ sqlite3 production-test.db "select data from wiki_content_versions where id=2294"
test
$ sqlite3 production-test.db "select QUOTE(data) from wiki_content_versions where id=2294"
'test'
</pre></li>
<li>How new DB stores blob data: <pre>
$ echo -n 'test' | hexdump -ve '1/1 "%.2x"'
74657374
$ sqlite3 production-test.db "update wiki_content_versions set data=x'74657374' where id=2294"
$ sqlite3 production-test.db "select data from wiki_content_versions where id=2294"
test
$ sqlite3 production-test.db "select QUOTE(data) from wiki_content_versions where id=2294"
X'74657374'
</pre></li>
</ul>
<ul>
<li>Old database (Redmine 1.2) contains blobs in binary representation and new database (Redmine 2.1) contains blobs in X'HEXHEXHEX' representation.
<ul>
<li>So if you try to get data using <strong>'<code>select data ...</code>'</strong> query both databases give same results but if you try to get data using <strong>'<code>select QOUTE(data) ...</code>'</strong> you get binary in the first case and X'HEXHEXHEX' in the second one.</li>
</ul>
</li>
<li>Problem is that old database style (with binary representation) doesn't work with the new Redmine version.</li>
</ul> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432472012-12-05T17:31:06ZStanislav German-Evtushenko
<ul></ul><p>Does it look like a Rails bug?</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432482012-12-05T17:46:02ZEtienne Massip
<ul></ul><p>Did you try to activate debug and trace SQL queries on your 2.x instance?</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432502012-12-05T18:48:48ZStanislav German-Evtushenko
<ul></ul><p>No idea how to debug. Could you give a hint?</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432522012-12-05T21:49:47ZEtienne Massip
<ul></ul><p>Use <a class="source" href="https://www.redmine.org/projects/redmine/repository/svn/entry/trunk/config/additional_environment.rb.example">source:/trunk/config/additional_environment.rb.example</a>.</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432552012-12-06T06:18:48ZStanislav German-Evtushenko
<ul></ul><p>Don't see anything that can help:<br /><pre>
Started GET "/projects/it-tasks/wiki/Page1?version=11" for 10.10.10.10 at 2012-12-06 10:11:21 +0400
Processing by WikiController#show as HTML
Parameters: {"version"=>"11", "project_id"=>"it-tasks", "id"=>"Page1"}
(0.2ms) SELECT MAX("settings"."updated_on") AS max_id FROM "settings"
User Load (0.1ms) SELECT "users".* FROM "users" WHERE "users"."type" IN ('User', 'AnonymousUser') AND "users"."id" = ? AND (users.status = 1) LIMIT 1 [["id", 3]]
Current user: user1 (id=3)
Project Load (0.1ms) SELECT "projects".* FROM "projects" WHERE "projects"."identifier" = 'it-tasks' LIMIT 1
Wiki Load (0.1ms) SELECT "wikis".* FROM "wikis" WHERE "wikis"."project_id" = 1 LIMIT 1
EnabledModule Load (0.1ms) SELECT name FROM "enabled_modules" WHERE "enabled_modules"."project_id" = 1
SQL (0.3ms) SELECT "members"."id" AS t0_r0, "members"."user_id" AS t0_r1, "members"."project_id" AS t0_r2, "members"."created_on" AS t0_r3, "members"."mail_notification" AS t0_r4, "projects"."id" AS t1_r0, "projects"."name" AS t1_r1, "projects"."description" AS t1_r2, "projects"."homepage" AS t1_r3, "projects"."is_public" AS t1_r4, "projects"."parent_id" AS t1_r5, "projects"."created_on" AS t1_r6, "projects"."updated_on" AS t1_r7, "projects"."identifier" AS t1_r8, "projects"."status" AS t1_r9, "projects"."lft" AS t1_r10, "projects"."rgt" AS t1_r11, "roles"."id" AS t2_r0, "roles"."name" AS t2_r1, "roles"."position" AS t2_r2, "roles"."assignable" AS t2_r3, "roles"."builtin" AS t2_r4, "roles"."permissions" AS t2_r5, "roles"."issues_visibility" AS t2_r6 FROM "members" LEFT OUTER JOIN "projects" ON "projects"."id" = "members"."project_id" LEFT OUTER JOIN "member_roles" ON "member_roles"."member_id" = "members"."id" LEFT OUTER JOIN "roles" ON "roles"."id" = "member_roles"."role_id" WHERE "members"."user_id" = 3 AND (projects.status<>9) ORDER BY projects.name
WikiPage Load (0.3ms) SELECT "wiki_pages".* FROM "wiki_pages" WHERE "wiki_pages"."wiki_id" = 1 AND (LOWER(title) = LOWER('Page1')) ORDER BY title LIMIT 1
WikiContent Load (0.1ms) SELECT "wiki_contents".* FROM "wiki_contents" WHERE "wiki_contents"."page_id" = 100 LIMIT 1
WikiContent::Version Load (0.2ms) SELECT "wiki_content_versions".* FROM "wiki_content_versions" WHERE "wiki_content_versions"."wiki_content_id" = 100 AND "wiki_content_versions"."version" = 11 LIMIT 1
Wiki Load (0.1ms) SELECT "wikis".* FROM "wikis" WHERE "wikis"."id" = 1 LIMIT 1
Project Load (2.1ms) SELECT "projects".* FROM "projects" WHERE "projects"."id" = 1 LIMIT 1
CACHE (0.0ms) SELECT name FROM "enabled_modules" WHERE "enabled_modules"."project_id" = 1
WikiPage Load (0.1ms) SELECT "wiki_pages".* FROM "wiki_pages" WHERE "wiki_pages"."id" = 100 LIMIT 1
CACHE (0.0ms) SELECT "wiki_contents".* FROM "wiki_contents" WHERE "wiki_contents"."page_id" = 100 LIMIT 1
(0.1ms) SELECT "users".id FROM "users" INNER JOIN "watchers" ON "users"."id" = "watchers"."user_id" WHERE "users"."type" IN ('User', 'AnonymousUser') AND "watchers"."watchable_id" = 100 AND "watchers"."watchable_type" = 'WikiPage'
WikiPage Load (0.1ms) SELECT "wiki_pages".* FROM "wiki_pages" WHERE "wiki_pages"."id" = 269 LIMIT 1
WikiPage Load (0.1ms) SELECT "wiki_pages".* FROM "wiki_pages" WHERE "wiki_pages"."id" = 67 LIMIT 1
WikiPage Load (0.1ms) SELECT "wiki_pages".* FROM "wiki_pages" WHERE "wiki_pages"."id" = 1 LIMIT 1
CACHE (0.0ms) SELECT "wikis".* FROM "wikis" WHERE "wikis"."id" = 1 LIMIT 1
CACHE (0.0ms) SELECT "projects".* FROM "projects" WHERE "projects"."id" = 1 LIMIT 1
CACHE (0.0ms) SELECT "wikis".* FROM "wikis" WHERE "wikis"."id" = 1 LIMIT 1
CACHE (0.0ms) SELECT "projects".* FROM "projects" WHERE "projects"."id" = 1 LIMIT 1
CACHE (0.0ms) SELECT "wikis".* FROM "wikis" WHERE "wikis"."id" = 1 LIMIT 1
CACHE (0.0ms) SELECT "projects".* FROM "projects" WHERE "projects"."id" = 1 LIMIT 1
Rendered wiki/_content.html.erb (1.1ms)
Rendered wiki/show.html.erb within layouts/base (13.9ms)
Completed 500 Internal Server Error in 38ms
ActionView::Template::Error (invalid distance too far back):
1: <div class="wiki wiki-page">
2: <%= textilizable content, :text, :attachments => content.page.attachments,
3: :edit_section_links => (@sections_editable && {:controller => 'wiki', :action => 'edit', :project_id => @page.project, :id => @page.title}) %>
4: </div>
app/models/wiki_content.rb:97:in `inflate'
app/models/wiki_content.rb:97:in `text'
app/helpers/application_helper.rb:522:in `textilizable'
app/views/wiki/_content.html.erb:2:in `_app_views_wiki__content_html_erb__1100882616125802453_63646400'
app/views/wiki/show.html.erb:37:in `_app_views_wiki_show_html_erb__2952486958244708562_38946520'
app/controllers/wiki_controller.rb:92:in `show'
</pre></p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432582012-12-06T08:21:12ZStanislav German-Evtushenko
<ul></ul><p>I have written a script and converted all all the records to the X'HEXHEXHEX' format but that didn't help :(<br />Each time I get "ActionView::Template::Error (invalid distance too far back)"</p>
<pre>
#!/bin/bash
db=production-old.db
db_out=production-new.db
id_list=$(echo "select id from wiki_content_versions where compression='gzip';" | sqlite3 $db)
for id in $id_list; do
data=$(echo "select data from wiki_content_versions where id=$id;" | sqlite3 $db)
data_hex=$(echo -n "$data" | hexdump -ve '1/1 "%.2x"')
echo "update wiki_content_versions set data=x'$data_hex' where id=$id;" | sqlite3 $db_out
done
</pre> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432602012-12-06T10:20:57ZStanislav German-Evtushenko
<ul></ul><p>Another idea. Can the moving from x32 to x64 system be the cause?</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432622012-12-06T12:02:39ZStanislav German-Evtushenko
<ul></ul>Here is another test.<br />I have created two pages (one in Redmine 2.1 and another in Redmine 1.2) and made the same changes for them (current page text is "h1. Testpage2").<br />Databases contents:
<ul>
<li>Redmine 2.1 <pre>
00000000 78 da cb 30 d4 53 08 49 2d 2e 29 48 4c 4f 35 02 |x..0.S.I-.)HLO5.|
00000010 0a |.|
00000011
</pre></li>
<li>Redmine 1.2 <pre>
00000000 78 da cb 30 d4 53 08 49 2d 2e 29 48 4c 4f 35 02 |x..0.S.I-.)HLO5.|
00000010 25 30 30 1c b9 04 57 0a |%00...W.|
00000018
</pre></li>
</ul>
<p>Does anybody has an idea how to fix it so far?</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432652012-12-06T14:25:27ZStanislav German-Evtushenko
<ul></ul><p>Stanislav German-Evtushenko wrote:</p>
<blockquote>
Can this change be the cause of the problem?
<ul>
<li>RAILS 2:
<ul>
<li><a class="external" href="http://apidock.com/rails/v2.1.0/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class">http://apidock.com/rails/v2.1.0/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class</a></li>
<li><a class="external" href="http://apidock.com/rails/v2.3.8/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class">http://apidock.com/rails/v2.3.8/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class</a></li>
</ul>
</li>
<li>RAILS 3:
<ul>
<li><a class="external" href="http://apidock.com/rails/v3.2.8/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class">http://apidock.com/rails/v3.2.8/ActiveRecord/ConnectionAdapters/SQLiteColumn/binary_to_string/class</a></li>
</ul></li>
</ul>
</blockquote>
<p>I was right. Problem is exactly there and this is bug of the old rails 2.x version.<br />I have manually replaced "25 30 30" <br /><pre>
00000000 78 da cb 30 d4 53 08 49 2d 2e 29 48 4c 4f 35 02 |x..0.S.I-.)HLO5.|
00000010 25 30 30 1c b9 04 57 0a |%00...W.|
00000018
</pre><br />by "00" <br /><pre>
00000000 78 da cb 30 d4 53 08 49 2d 2e 29 48 4c 4f 35 02 |x..0.S.I-.)HLO5.|
00000010 00 1c b9 04 57 0a |....W.|
00000016
</pre><br />and successfully decompressed the data! And now I have a lot of work with converting data. I have to convert all of "253030" to "00" and all of "253235" to "25". Can anybody suggest a quick solution?</p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432682012-12-06T15:10:22ZStanislav German-Evtushenko
<ul></ul><p>I did it! Hope it can save huge amount time for somebody who gets into the same situation.<br /><pre>
#!/bin/bash
# production-old.db - Redmine 1.2 database
# production-new.db - Redmine 2.1 database (after db:migration)
db=production-old.db
db_out=production-new.db
id_list=$(echo "select id from wiki_content_versions where compression='gzip';" | sqlite3 $db)
for id in $id_list; do
data=$(echo "select data from wiki_content_versions where id=$id;" | sqlite3 $db)
data_hex=$(echo -n "$data" | sed 's/\x25\x30\x30/\x00/g; s/\x25\x32\x35/\x25/g' | hexdump -ve '1/1 "%.2x"')
echo "update wiki_content_versions set data=x'$data_hex' where id=$id;" | sqlite3 $db_out
done
</pre></p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432702012-12-06T15:31:25ZStanislav German-Evtushenko
<ul></ul><p>Would be nice to have this information on the page: <a class="wiki-page" href="https://www.redmine.org/projects/redmine/wiki/RedmineUpgrade#SQLite-database">RedmineUpgrade#SQLite-database</a></p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=432712012-12-06T16:58:20ZStanislav German-Evtushenko
<ul></ul><p>Link to the related changes in Rails on GitHub: <a class="external" href="https://github.com/rails/rails/compare/v3.2.1...v3.2.2#L66L5">https://github.com/rails/rails/compare/v3.2.1...v3.2.2#L66L5</a><br />Pull request with a fix description: <a class="external" href="https://github.com/rails/rails/pull/4809">https://github.com/rails/rails/pull/4809</a></p> Redmine - Defect #12501: Wiki history partially doesn't work after upgradehttps://www.redmine.org/issues/12501?journal_id=433052012-12-08T09:34:10ZJean-Philippe Langjp_lang@yahoo.fr
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li><li><strong>Resolution</strong> set to <i>Wont fix</i></li></ul><p>Thanks Stanislav for your investigation, I'm closing it as "Won't Fix" since its a Rails issue. I've added a note to <a class="wiki-page" href="https://www.redmine.org/projects/redmine/wiki/RedmineUpgrade#Related-Resources">RedmineUpgrade</a>.</p>