Xapian search plugin

Added by Xabier Elkano almost 4 years ago

I wrote a little plugin to allow searches trough redmine attachments. I've developed it on Redmine 1.0.0. I don't know how to extend redmine search to include my code so I've extended my plugin to include redmine search too.
I used xapian search engine to search and omega to index files.

To use xapian, you have to install libxapian-ruby1.8 and xapian-omega packages (on ubuntu). To index some files with omega you may have to install some other packages like xpdf, antiword, ...

From Omega documentation:

  • PDF (.pdf) if pdftotext is available (comes with xpdf)
  • PostScript (.ps, .eps, .ai) if ps2pdf (from ghostscript) and pdftotext (comes with xpdf) are available
  • OpenOffice/StarOffice documents (.sxc, .stc, .sxd, .std, .sxi, .sti, .sxm, .sxw, .sxg, .stw) if unzip is available
  • OpenDocument format documents (.odt, .ods, .odp, .odg, .odc, .odf, .odb, .odi, .odm, .ott, .ots, .otp, .otg, .otc, .otf, .oti, .oth) if unzip is available
  • MS Word documents (.doc, .dot) if antiword is available
  • MS Excel documents (.xls, .xlb, .xlt) if xls2csv is available (comes with catdoc)
  • MS Powerpoint documents (.ppt, .pps) if catppt is available (comes with catdoc)
  • MS Office 2007 documents (.docx, .dotx, .xlsx, .xlst, .pptx, .potx, .ppsx) if unzip is available
  • Wordperfect documents (.wpd) if wpd2text is available (comes with libwpd)
  • MS Works documents (.wps, .wpt) if wps2text is available (comes with libwps)
  • AbiWord documents (.abw)
  • Compressed AbiWord documents (.zabw) if gzip is available
  • Rich Text Format documents (.rtf) if unrtf is available
  • Perl POD documentation (.pl, .pm, .pod) if pod2text is available
  • TeX DVI files (.dvi) if catdvi is available
  • DjVu files (.djv, .djvu) if djvutxt is available
  • XPS files (.xps) if unzip is available

Before use the plugin you have to index redmine files:

omindex -s english --db /var/www/xapian-index/ /var/www/redmine/files/

You can launch this command from crontab one or two times a day.

After indexing you have to set index db path on plugin settings (/var/www/xapian-index/). It takes documents permissions to allow searches to users.
It is my first plugin so code improvements are welcome.

I hope it will be usefull for someone.

Extended search

redmine_xapian.tar.gz - Xapian search plugin (7.64 KB)

extended-search.PNG - Extended search (8.3 KB)

Replies (15)

RE: Xapian search plugin - Added by Eric Davis almost 4 years ago

Xabier Elkano wrote:

I don't know how to extend redmine search to include my code so I've extended my plugin to include redmine search too.

Look at vendor/plugins/acts_as_searchable/

Eric Davis

RE: Xapian search plugin - Added by Xabier Elkano almost 4 years ago

New release, fully integrated with redmine at http://github.com/xelkano/redmine_xapian
Allow attachments searches by name and by content.

thank you Eric

RE: Xapian search plugin - Added by Terence Mill over 3 years ago

I was able to analyze the needed libs and depencies deeper. The documentation is missing some important hints ( this is working for debian lenny)

cd /redmine
apt-get install bxapian-ruby1.8 xapian-omega libxapian-dev xpdf antiword unzip antiword catdoc libwpd8c2a libwps-0.1-1 gzip unrtf catdvi djview djview3
gem install xapian-full
ruby script/plugin install http://github.com/xelkano/redmine_xapian.git (or download and extract tar or zip file.)
rake db:migrate_plugins RAILS_ENV=“production”
Restart web server
rake db:migrate_plugins RAILS_ENV=production --trace

RE: Xapian search plugin - Added by Olaf Ahlers over 3 years ago

Hi there,
Despite this, I cannot seem to make the plugin appear in redmine.
I tried versions 1.1.0, 1.0.5 and 1.0.1. on a Centos 5.5 Vm.
I have been using this Gemfile: {{{
source "http://rubygems.org"
gem "rake", "0.8.3"
gem "rack", "1.0.1"
gem "i18n", "0.4.2"
gem "rubytree", "0.5.2", :require => "tree"
gem "RedCloth", "~>4.2.3", :require => "redcloth" # for CodeRay
gem "mysql"
}}}
Permissions are good and other plugins (such as the knowledgebase plugin) do show up in the Administration->plugins page.
Can anyone provide me with the exact set of Gems that he has been using in a successful installation of the redmine_xapian plugin ?
Or some other clue ?
Many thanks

RE: Xapian search plugin - Added by Jerry Van Baren over 3 years ago

@Olaf, it worked for me under Ubuntu 10.04 following all the hints above and in the readme.

Get the support packages:

sudo apt-get install libxapian-ruby1.8 xapian-omega xpdf antiword  unrtf catdoc catdvi

Get the plugin and install it (restart the web server after doing this):

ruby script/plugin install http://github.com/xelkano/redmine_xapian.git
rake db:migrate_plugins RAILS_ENV=production
rake db:migrate_plugins RAILS_ENV=production --trace

Index the attachment files:

omindex -s english --db /var/tmp/xapian-index/ /var/www/rails-apps/redmine-1.0/files/

Go into the Administration / Plugins page, configure Xapian
  • Set the "Index database location" to /var/tmp/xapian-index/

I changed the ownership of the plugin and made the Xapian index owned by www-data, e.g. chown -R www-data.www-data (web server user in Ubuntu) instead of root. This may be unnecessary, but is a good practice.

RE: Xapian search plugin - Added by Zazvonnyh Alexsey over 2 years ago

did xapian plugin 1.2.3 tested with redmine 1.3.0? sorry in changelog written that it's tested, may be it my problem

RE: Xapian search plugin - Added by Zazvonnyh Alexsey over 2 years ago

Zazvonnyh Alexsey wrote:

did xapian plugin 1.2.3 tested with redmine 1.3.0? sorry, in changelog written that it's tested, may be it my problem

RE: Xapian search plugin - Added by Xabier Elkano over 2 years ago

Zazvonnyh Alexsey wrote:

Zazvonnyh Alexsey wrote:

did xapian plugin 1.2.3 tested with redmine 1.3.0? sorry, in changelog written that it's tested, may be it my problem

Yes, I have it running on redmine 1.3.0, may be you have another plugin and they are not working well together.

RE: Xapian search plugin - Added by Daniel Schaarschmidt over 2 years ago

Has anybody managed to get this working on a Windows-based Redmine? I've been trying to get it to work on the Bitnami Redmine Stack 1.3 for Windows, but I'm not even sure where to start from.

RE: Xapian search plugin - Added by david sharpe over 2 years ago

Hi, any luck getting this to work on windows, my initial research shows the best way would be the xapian windows binaries supplied by flax.co.uk, also does this work in redmine 1.4 ?

RE: Xapian search plugin - Added by Terence Mill over 2 years ago

What happens how does this work if i use jruby on tomcat where only gems are bundled with warble into the war?

RE: Xapian search plugin - Added by david sharpe over 2 years ago

Hey, I started my windows install attempt by installing the plugin without doing any of the installs of xapian itself. Id expect to at least see the plugin listed in plugins, but nothing ! this is a completely bare install of redmine 1.3, no other plugins

Any Ideas ?

RE: Xapian search plugin - Added by david sharpe over 2 years ago

i'm still not sure if anyone has managed to get this workiong in windows env, or even if it can !

RE: Xapian search plugin - Added by Terence Mill almost 2 years ago

I get problem with jruby 1.6.8 - Please help!

opt/jruby/bin/jruby --1.9 /opt/jruby/bin/bundle install --without development test sqlite sqlite3 postgresql pg postgres

/home/user/.rvm/gems/jruby-1.7.0/gems/xapian-full-1.2.3/Rakefile:6:in `system!'                                                                                             
/home/user/.rvm/gems/jruby-1.7.0/gems/xapian-full-1.2.3/Rakefile:35:in `(root)'                                                                                             
org/jruby/RubyDir.java:466:in `chdir'                                                                                                                                              
/home/user/.rvm/gems/jruby-1.7.0/gems/xapian-full-1.2.3/Rakefile:32:in `(root)'                                                                                             
org/jruby/RubyProc.java:270:in `call'                                                                                                                                              
org/jruby/RubyProc.java:220:in `call'                                                                                                                                              
org/jruby/RubyArray.java:1615:in `each'                                                                                                                                            
org/jruby/RubyArray.java:1615:in `each'                                                                                                                                            
Tasks: TOP => default                                                                                                                                                              
(See full trace by running task with --trace)                                                                                                                                      
                                                                                                                                                                          Gem files will remain installed in /home/user/.rvm/gems/jruby-1.7.0/gems/xapian-full-1.2.3 for inspection.                                                                  
Results logged to /home/user/.rvm/gems/jruby-1.7.0/gems/xapian-full-1.2.3/./gem_make.out                                                                                    
An error occurred while installing xapian-full (1.2.3), and Bundler cannot continue.                                                                                               
Make sure that `gem install xapian-full -v '1.2.3'` succeeds before bundling.                         

RE: Xapian search plugin - Added by Terence Mill almost 2 years ago

For Jruby 1.7 still can't install xapian-full
See https://github.com/xelkano/redmine_xapian/issues/29

(1-15/15)